2026-07-03
There's a widespread assumption in the AI coding world that if you give an agent more — more tools, browser access, elaborate system prompts telling it how to be a good engineer — you'll get better software out. This paper asks the awkward question: is that actually true? The author ran a controlled experiment to find out.
The setup was simple but rigorous. Ninety independent agent runs were pointed at the same detailed specification: build a real-time retrospective board (think Miro-style sticky-note app). Each run's output was graded against a fixed 14-criterion functional rubric worth 42 points, plus a visual quality review. The variables tested included things like whether the agent had browser-based testing tools, whether it received design-focused prompts, and — crucially — how much reasoning effort the underlying model was allowed to spend before writing code.
The headline finding: tool access and fancy prompts barely moved the needle. Reasoning effort did. Runs where the model was allowed to think harder before acting produced dramatically more reliable first-try builds. Runs given browser tooling or aesthetic-focused system prompts, without extra reasoning budget, showed no meaningful improvement in functional correctness.
Why does this matter for how we build with agents?
The observational nature of the study is a caveat: 90 runs on one spec is suggestive, not definitive, and different task types (say, data pipelines vs. UI apps) might weight things differently. But it lands as a useful corrective for teams building agent scaffolding. Before you wire up another tool, ask whether you've actually let the model think.