Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems

2026-08-20

Authors: George Andrikopoulos

ArXiv: 2608.19140v1

PDF: Download PDF

Imagine two marksmen at a shooting range. One puts every shot within a millimeter of the bullseye. The other's shots average out to the bullseye too — but they're scattered wildly around it. On paper, both have the same "average accuracy." In practice, only one of them is someone you'd trust with a real target.

Andrikopoulos argues that the AI industry has been ranking large language models like the second marksman: by where the average shot lands (capability) rather than how tightly the shots cluster (precision). Every big benchmark — MMLU, HumanEval, the leaderboards — reports mean performance. Ask the same model the same question ten times, though, and you'll often get ten meaningfully different answers. That variance is invisible in the headline number, but it's exactly what determines whether you can build a reliable system on top of the model.

The paper's core claim: capability has saturated. Frontier models all land near the target on most tasks. What actually separates a usable system from an unusable one now is reliability under repetition — whether a customer support bot, a code reviewer, or a medical triage assistant gives you the same sensible answer each time a user asks the same thing.

Andrikopoulos borrows a framework from manufacturing and marksmanship called grouping: the tightness of the cluster, independent of where the cluster sits. He proposes reporting AI performance the way you'd report a rifle's precision or a factory's yield:

The practical upshot is that a "worse" model with tight grouping may be more valuable in production than a "better" model that occasionally goes off the rails — because you can engineer around a consistent bias, but you can't engineer around randomness. Two systems with identical benchmark scores can have wildly different failure profiles, and buyers currently have no way to see that from the marketing.

The piece reads as a call to change what the industry measures. If benchmarks started publishing distributions instead of means — the AI equivalent of a shot group instead of a single accuracy number — vendor claims and buyer decisions would look quite different.

Why it matters: Anyone building on top of LLMs already knows that non-determinism is the real production headache; this paper gives that intuition a clean vocabulary and argues the whole benchmarking industry is measuring the wrong axis.

All newsletters