LLM Benchmarks Explained: Why Vendor Scores Mislead

Frontier-Bench v0.1, a new AI agent benchmark from the team behind Terminal-Bench and Harbor, launched on July 23, 2026 with 74 tasks spanning seven domains — and within a day, a vendor's own reported score on it did not match the number on Frontier-Bench's public leaderboard for the same model family. That gap is not a scandal; it is the normal, structural reason benchmark tables in launch blog posts should be read as marketing claims first and comparable data second.

What launched on July 23, 2026

Frontier-Bench started internally as "Terminal-Bench 3.0" before its creators spun it out as an independent, continuously updated benchmark. Version 0.1 ships with 74 tasks covering databases, ML checkpoints, RTL chip designs, formal proofs, source code, VM images and binaries, music scores, CAD files, scientific analyses and business decisions — seven domains in total, with coding tasks as the largest category and scientific tasks second. On launch day, the best-performing agent on the public leaderboard reached roughly 34% of the maximum possible score, according to Frontier-Bench's own announcement.

The methodology is the more interesting part for anyone who has to evaluate models for a living. Tasks go through a five-stage review pipeline — proposal, LLM-judge screening, human reviewer feedback, agent trial runs, and senior review — before they're added. Agent and verifier code run in separate containers specifically to prevent reward hacking, where an agent finds a shortcut that satisfies the grading script without actually doing the task. And unlike most benchmarks, which get replaced outright once models start saturating them, Frontier-Bench uses semantic versioning and result migrations so old scores stay comparable as new task revisions roll in.

Why Terminal-Bench needed a successor

Terminal-Bench became a standard reference for evaluating AI coding agents after its May 2025 launch, reaching 1,000 Discord members and 100 GitHub contributors by the time Terminal-Bench 2.0 shipped. But it measured one thing: terminal and CLI-based coding tasks. As agentic models started tackling multi-step work well outside a shell prompt — scientific analysis, hardware design, financial modeling — a single-domain benchmark stopped capturing the gap between models. Frontier-Bench is the response: same lineage, wider net.

The messy part: vendor numbers don't match the leaderboard

Here's the concrete example. Anthropic's July 24, 2026 launch announcement for Claude Opus 5 says the model "more than doubles" Opus 4.8's Frontier-Bench v0.1 score at maximum reasoning effort, and cites Opus 5 beating GPT-5.6 Sol on the same benchmark. Frontier-Bench's own public leaderboard, published a day earlier on July 23, lists GPT-5.6 Sol at 34.4% and Opus 4.8 at 21.1% — numbers that were never run at each model's highest effort setting and don't reflect Opus 5 at all, since it didn't exist yet.

Both numbers can be accurate and still be incomparable, because Frontier-Bench scores a mean reward across five attempts per task, and that mean shifts with the agent scaffold, the reasoning-effort configuration, and how many attempts get averaged in — none of which vendors are obligated to disclose in a press release. The scoring logic itself is simple; what varies is everything feeding into it:

# Frontier-Bench-style scoring: mean reward across N attempts, averaged over tasks
def task_score(attempt_rewards):
    return sum(attempt_rewards) / len(attempt_rewards)
 
def overall_score(all_task_rewards):
    return sum(task_score(r) for r in all_task_rewards) / len(all_task_rewards)

Change the harness, the effort setting, or the attempt count feeding task_score, and the headline percentage moves — without the model itself changing at all.

SWE-bench and GPQA, for comparison

Frontier-Bench sits alongside two other benchmarks developers see cited constantly, and they measure genuinely different things:

• SWE-bench scores an agent on resolving real, historical GitHub issues from open-source Python repositories — did the patch it produced make the project's own test suite pass. It's narrow and code-specific, which is exactly why GPT-5.6 Sol, Terra and Luna and most other coding-focused model launches lead with a SWE-bench variant.
• GPQA (Graduate-Level Google-Proof Q&A) is a multiple-choice test of PhD-level questions in biology, physics and chemistry, deliberately written so the answer can't be found by a quick web search. It measures reasoning under a fixed answer format, not agentic task completion.
• Frontier-Bench measures open-ended, multi-step agentic work — including but not limited to code — scored by whether the final artifact (a database schema, a formal proof, a CAD file) actually passes verification, not by matching a multiple-choice answer.

None of the three substitute for the others, and a model that leads on one can trail badly on another.

A checklist for reading benchmark claims

Before treating a vendor's benchmark table as a buying signal, check:

1. Vendor-reported or independent? A number in a launch blog post is a claim the vendor chose to publish, not an audited result.
2. Which harness ran it? The same model can score differently depending on the agent scaffold (a custom internal harness versus a public one like mini-SWE-agent) that executed the tasks.
3. What effort setting? "Maximum effort" and default settings on the same model can differ by double digits on agentic benchmarks, as the Opus 4.8 to Opus 5 comparison above shows.
4. What date, what version? Frontier-Bench's semantic versioning means a score from v0.1 and a score from a later revision on the same model aren't automatically the same measurement — check the benchmark version, not just the benchmark name.
5. Is there an independent leaderboard to cross-check? When one exists, as it does for Frontier-Bench, compare the vendor's number against it before repeating the vendor's framing.

The takeaway

Benchmark names give the illusion of a shared, objective scale, but the number next to the name depends on choices the vendor made and usually didn't disclose. Treat every benchmark score in a launch announcement as a claim to verify against the benchmark's own leaderboard where one exists, and expect the two numbers to disagree for reasons that have nothing to do with either party being dishonest.