If you've been following AI coding agents at all, you've probably heard of SWE-bench — the benchmark where an AI is handed a real GitHub issue and has to produce a patch that fixes it. The scoring rule is simple: did the patch make the project's test suite go green? If yes, win. If no, loss. Progress on this benchmark has been the main way we measure whether coding agents are getting better.

The authors of SWE-Gate make a pointed observation: passing tests is not the same as being merge-worthy. When a real human submits a pull request to a real open-source project, the maintainer doesn't just run CI and click merge. They review it. They ask for renames. They complain about a missing docstring, a violated style rule, a change that touches more files than it should, a fix that solves the symptom but not the root cause, or an approach that clashes with how the rest of the codebase is written. These are review constraints, and current benchmarks completely ignore them.

SWE-Gate is a new benchmark built to test this. The authors mined real code-review conversations from actual open-source repositories and extracted the constraints reviewers imposed on patches — things like:

• "Use the existing helper function instead of writing your own."
• "This should be a private method, not public."
• "Add a test for the null-input case."
• "Don't touch this unrelated file."

Then they evaluated leading coding agents not just on whether their patches pass tests, but on whether the patches comply with these reviewer-imposed constraints.

The results are the interesting part. Agents that look impressive on SWE-bench take a real hit here. A patch can pass every test and still violate half a dozen things a human reviewer would flag on sight. This gap suggests today's agents are optimizing for "the tests are green" rather than "this is code a teammate would actually accept" — and those are meaningfully different targets.

The insight worth taking away: we've been grading coding agents on a rubric that's too easy. Real software engineering isn't just producing behavior-correct code; it's producing code that fits the project's conventions, respects its architecture, doesn't sprawl, and reflects taste. As agents get deployed into real dev workflows, benchmarks that capture the review dimension — not just the CI dimension — are what will actually predict whether they're useful teammates.