When people ask whether AI coding assistants write fast code — not just correct code — the honest answer has been "we don't really know." A handful of benchmarks (EffiBench, Enamel, EvalPerf, Mercury) try to answer this by timing LLM-generated solutions against reference implementations. The trouble is, these benchmarks have consistently shown that AI-written code runs about as fast as the human-written baseline. That's either good news (the AI is competitive!) or a warning that the benchmarks aren't sensitive enough to tell the difference. This paper argues it's mostly the latter.

The authors took 1,538 tasks across those four popular benchmarks and did something surprisingly rare: they ran each task 30 times and measured the variance, not just the average. Then they scrutinized the whole measurement pipeline — how tasks are timed, what inputs get used, whether the "canonical" reference solutions are actually efficient, and whether the reported speedups are statistically real or just noise.

What they found is a bit of a mess:

• Timing noise swamps the signal. A lot of reported differences between LLM code and reference code fall within normal run-to-run variation. If you flip a coin 30 times you'll get streaks; benchmarks running once mistake those streaks for capability.
• The "gold standard" solutions aren't always golden. Some reference implementations are themselves inefficient, which makes the LLM look artificially competitive — you can't measure a speedup against a slow baseline and call it impressive.
• Test inputs are often too small. Many tasks finish in microseconds, where OS scheduling jitter dominates. You can't distinguish an O(n) from an O(n log n) solution on an input of size 10.
• Warmup, isolation, and hardware variability are handled inconsistently across benchmarks, so results don't compare cleanly across studies.

The paper proposes fixes: bigger inputs, repeated runs with proper statistical tests, verified-efficient reference solutions, and a checklist for what a credible performance benchmark should look like. It's essentially a "measurement hygiene" paper for a field that's been publishing performance claims on shaky foundations.

The broader point is one that keeps recurring in ML evaluation: if your benchmark can't distinguish signal from noise, every model looks the same. That's not proof the models are equivalent — it's proof your ruler is too short. As LLMs get better and the gaps get smaller, the quality of measurement infrastructure becomes the bottleneck for knowing whether progress is happening at all.