2026-09-04
If you've ever asked an AI to "fix this one bug" and gotten back a file where half the code has been quietly rewritten — different variable names, restructured loops, a new helper function you didn't ask for — this paper is about you. The authors call this behavior over-editing, and they set out to measure exactly how bad it is.
Here's the setup. Correctness is the usual bar for AI code repair: does the fixed code pass the tests? But in real software work, that's not enough. A good fix is also minimal — it changes only what needs to change. Minimal patches are easier to review, easier to trust, and less likely to introduce sneaky new bugs in code that wasn't broken in the first place. When a model rewrites a working function while fixing a bug in a nearby one, it's burning your reviewer's time and adding risk.
To study this rigorously, the authors built a clever benchmark. They took 400 problems from BigCodeBench (a well-known coding benchmark), and for each one they took the correct reference solution and injected a small, controlled bug at the syntax-tree level — say, flipping a comparison operator or swapping two arguments. Because they know exactly what corruption they introduced, they also know exactly what the minimal fix looks like: undo that one change. This gives them a ground-truth "smallest possible patch" to compare against.
Then they ran a range of LLMs on these repair tasks and measured two things: did the model fix the bug (correctness), and how much extra code did it touch beyond the minimal patch (fidelity)?
The findings, based on the abstract:
The deeper insight is that we've been evaluating code-editing models on the wrong thing. "Did the tests pass?" hides a lot of sloppy behavior that matters enormously in practice — especially as these tools get folded into pull-request workflows where humans have to read every changed line. If your AI reviewer's diff is three times bigger than it needs to be, the human on the other end starts rubber-stamping, and that's where bugs slip through.
