The verifier based vs verifier free test time scaling result is older than people act, and it keeps getting confirmed [D]

The Setlur et al result that scaling test time compute without verification or RL is provably suboptimal keeps showing up in my reading and I think it deserves more weight than the "yet another scaling paper" treatment it got. The core claim is that verifier based methods, RL or search guided by a verifier, dominate verifier free methods like distilling successful traces, given a fixed compute budget, and the gap widens as the test time budget grows.

What I find underappreciated is how cleanly this maps onto what the deployed systems are now converging on. The single agent ReAct loop is the verifier free extreme, you sample a trace and keep it, maybe with some self reflection that is still the same model grading itself. The multi agent setups that actually move numbers split the verifier off into a separate process. Apodex is the most explicit example I have seen, they train the team behavior in and run a verification team, conflict reviewer, fact checker, draft reviewer, that does not share the reasoning trace, and the reported lift is coming from the verifier not from added parameters. Same trained model, heavy duty mode adds double digits on BrowseComp and FrontierScience-Research. That is exactly the regime the theory predicts, the verifier is where the gain lives.

The reason I think this matters beyond benchmark watching is that it reframes where the next chunk of capability comes from. If you believe the VB over VF result, then the path is not just bigger models or longer traces, it is better verifiers that are structurally independent of the generator. The pseudo correctness framing fits here too. The failure mode the verifier has to catch is not the obvious hallucination, it is the answer that passes every self check but is still wrong, and that failure mode is invisible to any verifier that shares context with the generator.

What I want to hear from others is the open questions. My list. How much of the verifier gain is transferable to domains without clean reward signals, since the math proof case is the easy one. Whether the independence has to be architectural, separate agents, or whether a sufficiently disciplined prompt separation on one model gets you most of the way. And whether the VB advantage keeps widening or saturates once the verifier itself becomes the bottleneck.

The practical version of this for anyone building. If your agent loop has the same model reviewing its own work, you are in the VF regime and the theory says you are leaving capability on the table. The cheapest structural change is to make the verifier a different process with denied context, even if it is the same weights.