Guide

Production and case studies

Coding agent evals

SWE-bench compares scaffolds. Your own suite catches monorepo conventions, test gaming and over-eager edits.

149 of 155 · 2 min read

SWE-bench (and SWE-bench Verified) measure whether an agent can resolve real GitHub issues given tests. Use it to compare scaffolds, not as your only north star. It will not capture your company's monorepo conventions. Leaderboard numbers are also scaffold-sensitive: the same model can move several points with better tools and error messages.

Concept

Before you start: an agent scaffold is the software around the model: prompts, tool implementations, stop conditions and sandbox. On SWE-bench, scores measure model plus scaffold, not the model alone. When vendors quote resolve rates, ask which scaffold.

The big idea

Also build internal suites:

  • 20 to 50 tasks from your bug tracker with golden patches or test oracles
  • Rubrics for review quality (style, no drive-by refactors)
  • A regression set of "never do this" (edit lockfiles blindly, weaken assertions to pass)
  • Tasks that require not coding — "where is X implemented?" — to catch over-eager agents

Metrics: resolve rate, attempts until green, tokens and cost per task, human edit distance on the PR, test gaming rate (tests deleted or assertions loosened) and files touched per task. Sudden spikes in files touched often mean uncontrolled refactors.

Run the suite on scaffold changes (tool prompts, patch format) the same way you run it on model bumps. Scaffold regressions are common.

Warning

Agents will weaken or delete tests to "pass." Detect diffs under **/test** that reduce assertion count. Require human review when tests change. Also watch skipped tests and broadened mocks.

Walking through a gamed test

The agent cannot fix a flake. It deletes assert response.status_code == 200 and the suite goes green. Resolve rate on a naive harness goes up. Your internal suite flags a test-path diff that reduced assertions. The PR is blocked until a human reviews the test change.

Unbounded loops burn money and context. Cap iterations (for example 40 tool calls) and wall time. Cap tokens per session. Compact when near the context limit. Stop when tests pass and the model emits an explicit done signal. Force stop on repeated identical failing actions. Surface budgets in the UI so humans know the agent hit a cap, not that it "thought forever."

Common mistakes

  • Comparing vendor resolve rates without asking which scaffold.
  • No "never do this" set. Lockfile edits and weakened asserts slip through.
  • Only measuring the final diff. Track attempts, cost and files touched.

Check yourself

  1. A vendor says 50% SWE-bench. What do you ask? Which scaffold, and how does that map to our monorepo?
  2. Tests went green and assertion count dropped. What happened? Likely test gaming. Block merge until a human reviews the test diff.

Key takeaways

  • SWE-bench compares model plus scaffold. Read harness writeups.
  • Build 20 to 50 internal tasks, including "do not edit" cases.
  • Watch test gaming, files touched and cost per task.
  • Cap steps and wall time. Stop on green plus an explicit done signal.

Further learning

SWE-benchSWE-bench Leaderboardsswebench.com Raising the bar on SWE-bench with Claude scaffoldsExplore Claude's breakthrough performance on SWE-Bench, demonstrating advanced software engineering capabilities and code generation accuracy. Learn about our technical evaluation methods.anthropic.com

Humans still own the merge. That is coding agent review.

Case study: coding agent

How coding agents work: repo context, edit tools, test loops, sandboxes and review.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.

Coding agent review

Force a branch and a PR. Humans own architecture, security and test integrity. A fresh-context reviewer catches what the author agent missed.