# Benchmark

**TL;DR:** A standardized public test set used to compare models against each other, which is a genuinely different job from evaluating whether your system works.

Benchmarks are the shared yardsticks of the field. MMLU covers broad academic knowledge across 57 subjects. GPQA asks graduate-level science questions written to be hard to look up. HumanEval and its successors measure code generation from docstrings. SWE-bench uses real GitHub issues and checks whether the resulting patch passes the repository own tests. MT-Bench and LMArena capture human preference on open-ended conversation. Tool-use benchmarks like tau-bench test multi-turn agentic behavior against simulated environments. Each measures something specific, and the specificity is usually lost by the time the number reaches a launch post.

Two forces steadily erode their usefulness. Saturation comes first: once frontier models cluster in a narrow band at the top, the benchmark stops discriminating between them and differences fall inside the noise. Contamination is the more insidious one. Public test sets end up in training corpora, whether through direct inclusion or through the enormous volume of derivative discussion online, and a model that has effectively seen the answers scores well without having the capability. Nobody can fully rule this out for any public benchmark, which is why holdout and continuously refreshed variants keep appearing.

Then there is Goodhart law doing its usual work. Once a benchmark becomes a marketing target, optimizing for it becomes rational, and it stops being a measurement of the underlying ability and becomes a measurement of how much attention it received during training. This is not necessarily cheating; it is what happens to any metric with stakes attached.

The practical consequence is that benchmark rank is a weak predictor of how a model will do in your product. Your input distribution is not theirs. Your prompts, tools, retrieval quality, output format requirements, and latency budget are not represented anywhere in the score. A model two points down on MMLU can be substantially better on your task because it follows structured output instructions more reliably, and no public number will tell you that.

So use them for what they are good at. Benchmarks are excellent for shortlisting: they cheaply narrow a field of dozens of candidate models to three worth testing. Your own evals decide which of the three ships. The complement worth building is a private holdout set drawn from real traffic, kept out of any prompt sent to a provider that trains on inputs, since that is the only evaluation you can be sure no model has already seen.

## Why it matters

Model selection decisions get made off leaderboards constantly, and the leaderboard is measuring a different question than the one being asked. Understanding saturation, contamination, and distribution mismatch is what turns benchmarks from a source of false confidence into a useful filter. It also reframes eval investment correctly: the benchmark is free and generic, the private eval set is expensive and decisive, and only one of them can tell you whether to ship.

## Example

A team picks a model because it leads on a popular coding benchmark. In production, task success is worse than the model they were already using. The benchmark measures single-function generation from a clear specification; their product asks the model to navigate a large existing codebase, call four internal tools in sequence, and return structured JSON. The leading model is better at the first thing and less reliable at tool-call formatting, which is what actually determines whether their pipeline completes. A 200-case eval drawn from their own traffic surfaces this in an afternoon.

## Related terms

- [Evals (Evaluation Systems)](https://www.maximem.ai/glossary/evals)
- [LLM-as-a-Judge](https://www.maximem.ai/glossary/llm-as-a-judge)
- [End-to-End Eval](https://www.maximem.ai/glossary/end-to-end-eval)
- [Model Routing](https://www.maximem.ai/glossary/model-routing)
- [Dataset](https://www.maximem.ai/glossary/dataset)
- [LLM (Large Language Model)](https://www.maximem.ai/glossary/llm)
- [Loss Function](https://www.maximem.ai/glossary/loss-function)
- [Behavioral Evals](https://www.maximem.ai/glossary/behavioral-evals)

---

Source: [https://www.maximem.ai/glossary/benchmark](https://www.maximem.ai/glossary/benchmark)