AI Benchmarks Explained: SWE-bench, Terminal-Bench and the Numbers Behind Model Rankings

AI benchmark scores are useful when they are read as measurements of a defined experiment. They become misleading when a score is detached from the tasks, tools and conditions that produced it. A result labeled 80% may look precise while answering a question quite different from the one a developer or business owner is asking.

This AI-assisted explainer uses benchmark-maintainer documentation checked on September 5, 2026. It includes historical examples and original worked calculations. Coreqm has not run the benchmarks discussed here, and the article does not claim to provide a live ranking of all current models.

Start with the unit being evaluated

A language model is not always the complete system on a leaderboard. An agent may add search, a terminal, memory, multiple attempts or a review step. If two results use different systems around the model, their difference cannot be attributed to the model alone.

Write an evaluation label that includes the model, agent, benchmark version and relevant settings. Then ask what the score measures. Does success mean a repository test passed, a judge preferred the answer or a task was completed within a budget? Those are different outcomes, even when all are expressed as percentages.

This habit prevents a common mistake: using an impressive number as a general probability that an AI system will be correct. A benchmark samples a particular task distribution. Your application may have different inputs, tools, risks and success criteria.

What SWE-bench Verified measures

SWE-bench Verified contains 500 human-filtered software-engineering instances. Its maintainers describe checks for clear problem statements, correct tests and solvable tasks. The broader leaderboard can include different agent systems, while the bash-only setting uses a minimal environment to make model comparisons more controlled. Source: SWE-bench Verified overview.

For a reader, the key question is whether the reported setup resembles the work being considered. Repository issue resolution is informative for maintenance tasks, but it does not automatically measure visual design quality or the usability of a generated application. A test suite may validate behavior without judging whether a patch is easy for a team to maintain.

Treat the score as one piece of evidence. Inspect the setup, identify the kinds of work it covers and consider what your application still needs to test separately. A benchmark can be valuable without being comprehensive.

Why Terminal-Bench versions cannot be merged

The Terminal-Bench 2.1 release corrected 28 of the 89 tasks in version 2.0. Its maintainers describe problems involving external dependencies, resource mismatches and task specifications. In the release comparison, Claude Code with Opus 4.6 moves from 58.0% on 2.0 to 70.1% on 2.1. That is a historical benchmark-revision example, not evidence of a newly upgraded model. Source: Terminal-Bench 2.1 release analysis.

The lesson is larger than those particular numbers. If an evaluation changes, a score can change even when the model does not. Keep version labels attached and avoid placing results from different versions in one sortable column as though they share a denominator.

The agent matters too. A terminal controller determines how commands are issued, how errors are returned and how the model receives context. A result for one model-agent pair should not be renamed as a result for every application using that model.

Percentage points are not relative percentages

Using the historical example above, 70.1 minus 58.0 equals 12.1 percentage points. Dividing that increase by 58.0 gives approximately 20.9% relative improvement. Both calculations describe the same pair of numbers, but they communicate different quantities.

Label the quantity explicitly. Saying a score improved by 12.1% when you mean 12.1 percentage points can distort the reader's understanding. Conversely, a large relative increase from a low baseline can sound more decisive than the absolute change justifies.

The same care is needed when comparing costs. A system that halves token price does not necessarily halve cost per completed task if it uses more tokens or needs additional attempts. Percentages become meaningful only when the underlying quantity is clear.

Pass at one and multiple attempts answer different questions

First-attempt success asks whether one sampled attempt succeeds. A metric involving several attempts gives the system more opportunities and may use a specific estimator or selection rule. Consult the benchmark's definition; do not assume every label involving multiple attempts means the same thing.

As a deliberately simplified mathematical illustration, if independent attempts each had a 50% chance of success, five attempts would have a 96.875% chance of at least one success. Real agent attempts are not necessarily independent, and this is not a reported model result. The example shows why an extra attempt budget can transform a number without making the first response more reliable.

A production workflow may not know which attempt is correct. If success requires a judge or validator to select among candidates, include that mechanism and its cost. Multiple attempts are a legitimate system design, but the description should not hide them.

Read uncertainty and missing information honestly

A small gap between two scores may not establish a robust ordering. Results can vary across tasks, repeated runs and evaluation conditions. When a source provides an uncertainty interval, preserve it. When it does not, avoid supplying one from intuition.

Missing results should remain missing. A dash in a table does not mean zero capability, and the absence of a model from a leaderboard does not prove it is worse than the listed models. It may simply not have a comparable submitted result in the view being examined.

For an internal trial, save task-level outcomes and repeat a subset of runs. This helps reveal whether an average is stable or driven by a small number of cases. Report the sample size and avoid treating a tiny pilot as a universal finding.

Cost per success needs a clear denominator

Suppose a hypothetical evaluation costs $100 and produces 80 accepted tasks. Its API cost per accepted task is $1.25 if the $100 includes every attempt. Dividing only the cost of successful requests by successful tasks would omit the failures required to reach that outcome.

If you include human review, tool hosting or infrastructure, say so. If you exclude them, say so too. The goal is not one perfect metric for every organization; it is a calculation another reader can understand and reproduce.

Compare latency in the same disciplined way. Time to first token, total generation time and time to a verified outcome are separate measures. Choose the one relevant to the user's experience rather than whichever makes the chart look most favorable.

A checklist before sharing a model ranking

• Is the exact benchmark version shown?
• Are model and agent names both included?
• Are effort, attempts, tools and resource limits comparable?
• Is the metric defined, with uncertainty preserved where available?
• Are costs and exclusions stated clearly?
• Are vendor claims separated from independent reproduction?
• Does the conclusion stay within the work actually measured?

Frequently asked questions

Are benchmarks useless if conditions differ?

No. They can reveal progress, expose weaknesses and help choose candidates. The problem is not measurement; it is drawing conclusions beyond the measurement.

Can one score identify the best coding model?

Not for every team. Combine benchmark evidence with repository-specific acceptance tests using the coding model comparison framework.

What should a trustworthy review admit?

It should say what was tested, who tested it and what remains unknown. That standard applies to the Astra review, the DeepSeek V4 Pro analysis and any other model article. Precision about limitations is part of useful technical reporting, not a weakness to hide.