Benchmark Saturation: Capability Ceiling, Contamination, or Measurement Collapse?

When a benchmark loses headroom — frontier scores cluster near the top, rank order becomes noisy, and the leaderboard stops separating systems — the field tends to reach for one of three explanations: the task was mastered, the test leaked, or the metric stopped measuring what it was built to measure. This article argues that saturation is a diagnostic alarm rather than a diagnosis, that the three explanations are interacting mechanisms rather than competing buckets, and that benchmark lifecycle design is necessary but not sufficient for evaluating frontier systems.

Coverage note: verified through May 2026.

What "saturation" is and is not

Benchmark saturation is an observation about scores, not a claim about underlying ability. Operationally it shows up as four signals that usually arrive together:

• High absolute scores. Frontier systems push the top of the leaderboard within a small fraction of the maximum.
• Compressed variance among frontier systems. The gap between the leading models shrinks below the resolution of the metric.
• Reduced rank stability. Re-running with different prompts, decoding strategies, or scoring scripts shuffles the ordering.
• Weakened linkage between score deltas and capability deltas. A 2-point gain on the leaderboard no longer predicts a measurable difference in downstream behavior.

These four signals are necessary for a diagnosis of saturation, but they do not by themselves identify the cause. A score that pegs near 95% can come from a task that has genuinely been solved, from a public test set that has entered the training ecosystem, from a metric that has collapsed onto a surface format, or — most often — from some weighted combination. The temptation is to treat saturation as a final verdict on a benchmark. The more useful frame treats it as a prompt to revalidate the relationship between score and capability.

This article will adopt a decision-tree posture for that revalidation:

1. Does performance remain high on fresh, matched, private, and format-perturbed variants? If so, the benchmark may genuinely have ceased to stress the underlying construct at the frontier.
2. Does performance fall on contamination-resistant variants while remaining high on the public set? If so, leakage, near-duplicate exposure, or public-leaderboard selection pressure is implicated.
3. Does performance fall under semantically equivalent reformatting, paraphrasing, or scaffold changes? If so, the metric has likely collapsed onto format-matching rather than the construct it was built around.

Each branch admits partial answers, and any saturated benchmark can sit on more than one branch at once. The point of the tree is to discipline what would otherwise be a one-cause story.

Construct Validity in AI Evaluation — what counts as the underlying capability a benchmark is supposed to measure, and why "high score" is not a substitute for that question.

The historical record, benchmark by benchmark

A common framing claims that modern benchmarks "saturate within months of release." This is rhetorically useful but empirically loose. The named benchmarks have different release contexts, different scoring formats, different exposure patterns, and different saturation arcs. The following table reconstructs the record from primary sources.

Benchmark	Release	Construct	Saturation marker	Successor / counter-design
GLUE	2018 (Wang et al.)	Multi-task NLU	Surpassed by ~2019	SuperGLUE
SuperGLUE	2019 (Wang et al.)	Harder NLU	Human macro-average surpassed by DeBERTa, Jan 2021 (Microsoft Research)	None canonical; pivoted to MMLU and code
MMLU	2020 (Hendrycks et al.)	57-subject knowledge & reasoning	Frontier scores >85% by 2023; reduced discrimination among top models	MMLU-Pro (Wang et al.); MMLU-Redux (Gema et al.)
HumanEval	2021 (Chen et al.)	Short Python function synthesis	pass@1 > 90% by 2024	HumanEval+; LiveCodeBench (Jain et al.)
GSM8K	2021 (Cobbe et al.)	Grade-school word problems	Frontier > 95% by 2024	GSM1k (Zhang et al.); GSM-Symbolic (Mirzadeh et al.)
MATH	2021 (Hendrycks et al.)	Competition mathematics	Frontier scores rising fast; superseded as headline by MATH-500 subset	MATH-500 (process-supervision subset)

A few details from this record deserve to be foregrounded, because they cut against the "uniform months-to-saturation" framing.

GLUE → SuperGLUE was relatively fast, but the saturation claim is doing more work than the timeline alone supports. The SuperGLUE paper introduces itself as a response to GLUE, "introduced a little over one year ago," already being surpassed at the human-baseline level on most subtasks (Wang et al.). That is a year, not months, and the human baseline being beaten is not the same as the underlying language understanding being solved. SuperGLUE itself was passed on the human macro-average by DeBERTa in January 2021 — Microsoft's announcement reports a single-model score of 89.9 against a human baseline of 89.8 and an ensemble at 90.3 (Microsoft Research). That sequence is a real example of headroom collapse on a public NLU leaderboard, but it spans roughly two and a half years from GLUE's introduction to SuperGLUE saturation, not a single short cycle.

MMLU's saturation story is entangled with benchmark quality. The original 2020 paper showed GPT-3 well below expert human performance across 57 academic and professional subjects (Hendrycks et al.). By 2023–2024, frontier systems were posting scores in the high 80s and low 90s, but two follow-up efforts complicate the simple "MMLU has been solved" reading. Are We Done with MMLU? documents pervasive annotation and question-quality issues across the benchmark and proposes MMLU-Redux, a re-audited version, finding non-trivial label noise that confounds high-end score interpretation (Gema et al.). MMLU-Pro deliberately makes the benchmark harder — more reasoning-heavy questions, expanded answer choices — and reports that frontier models drop 16 to 33 points relative to MMLU, while prompt sensitivity falls from roughly 4–5% on MMLU to about 2% on MMLU-Pro (Wang et al.). Together these results suggest that part of MMLU's headline saturation reflects genuine progress, part reflects label noise, and part reflects models having absorbed the format and answer-pattern statistics of a public multiple-choice test.

HumanEval saturated as a static set; the response was time-indexed evaluation rather than a harder static replacement. The original Codex paper introduces 164 hand-written Python problems with unit tests, and reports Codex at 28.8% pass@1 versus GPT-3 at 0% (Chen et al.). HumanEval+ extended the test suite. LiveCodeBench took the more interesting design step: it continuously ingests problems from LeetCode, AtCoder, and Codeforces, indexes them by release date, and supports evaluation on problems posted after a model's training cutoff (Jain et al.). This is not a contamination-proof benchmark — public discussion of contest problems leaks into training data, and contest formats are themselves an artifact — but it is a contamination-resistant design in the strict sense that it lets you separate pre-cutoff from post-cutoff items for any given model.

GSM8K's saturation is the cleanest case of a partially recoverable diagnosis. The 2021 release introduced 8,500 grade-school math word problems, and even very large transformers of the era performed poorly (Cobbe et al.). By 2024, top reported scores exceeded 95%. The GSM1k construction is precisely the kind of fresh matched test the empirical literature has been calling for: a new, human-written, distribution-matched 1,250-item set, with concealed evaluation. Frontier models maintained near-public performance, while certain open and mid-tier model families dropped by up to 8% — implicating overfitting in those families rather than across-the-board memorization (Zhang et al.). The GSM-Symbolic construction adds a second probe: replacing names and numbers with symbolic templates, and adding semantically inert clauses to the problem statement. Performance falls under both perturbations, with the dropoff growing as more clauses are added (Mirzadeh et al.). The combined picture is that grade-school math has been substantially learned, that some of the public-set advantage was overfit, and that part of the remaining score reflects sensitivity to surface form rather than to the reasoning construct.

MATH → MATH-500 is a transition that should be treated cautiously. MATH is a 12,500-problem competition-mathematics set introduced in 2021 (Hendrycks et al.). MATH-500 emerged as a 500-problem subset popularized through process-supervision and step-level evaluation work, where its smaller size made it tractable for stepwise scoring. It has subsequently become a default headline metric. The narrowing is convenient — cheaper to run, easier to compare across labs, more stable to small changes — but it is not a decontamination story. A canonical 500-problem subset, repeatedly cited and re-evaluated, accumulates exposure faster than the parent set and concentrates leaderboard incentives onto a smaller construct slice. Treating MATH-500 as a benchmark renewal rather than a trade-off would mistake convenience for validity.

The honest reading of the table is that score compression is a robust phenomenon, that arrival times vary by benchmark and by model generation, and that the response patterns vary correspondingly: replacement (GLUE → SuperGLUE), refinement (MMLU → MMLU-Pro, MMLU-Redux), counter-design (HumanEval → LiveCodeBench), fresh matched probing (GSM8K → GSM1k, GSM-Symbolic), and convenience subsetting (MATH → MATH-500). Each response says something different about the failure mode it was attacking.

Three explanations that interact

The three standard explanations for saturation are best treated as overlapping mechanisms rather than mutually exclusive bins. Any saturated benchmark almost certainly carries weight from each.

Capability ceiling

The cleanest version of the story: the benchmark was constructed with a particular skill in mind, models acquired that skill at the level the benchmark could discriminate, and further gains in the underlying capability outrun the benchmark's resolution. Grade-school arithmetic, short Python function synthesis, and many natural-language inference tasks have plausibly crossed this threshold for frontier systems. The empirical signature is that scores remain high on fresh, matched, format-perturbed, and label-audited variants — that is, the score does not depend on having seen the public test set, and it does not depend on a particular surface format.

The capability-ceiling reading is real, and resisting it has its own failure modes. Evidence for genuine progress on fresh tests should not be dismissed as residual leakage, especially when a fresh holdout is constructed independently and paid attention to label quality.

The honest qualifier is that "ceiling" is local to the construct the benchmark was built around. A model can hit the GSM8K ceiling on grade-school word problems while falling sharply under symbolic reformatting, which means the ceiling reflects the public test's construct rather than the broader reasoning capability the construct was supposed to proxy for (Mirzadeh et al.).

Contamination, treated as a spectrum

Contamination is usually discussed as a binary — was the test in the training data or wasn't it — and that framing misses most of the actual mechanism. The relevant mechanisms form a spectrum:

• Exact-item memorization. The literal benchmark item appears in pretraining or post-training data, and the model retrieves it.
• Near-duplicate exposure. The item appears with paraphrased wording, translated text, or stylistic variants.
• Solution-pattern exposure. Discussions, blog posts, or tutorials that walk through the benchmark or close analogs are in the training data, supplying solution templates rather than literal answers.
• Synthetic-data tuning toward benchmark-like distributions. Distillation and self-improvement loops generate problems that match the benchmark's distribution, even when the benchmark items themselves are excluded.
• Checkpoint and recipe selection on public leaderboards. Labs use the public score as a model-selection signal, biasing released checkpoints toward configurations that score well on the public set.
• Protocol adaptation. Prompts, decoding strategies, instruction templates, and scaffolding choices are tuned to the benchmark's exact format.

Only the first two are detectable by canonical n-gram or near-duplicate audits. The remainder are harder to bound. They are also harder to call cheating: prompt tuning against a visible benchmark is a normal part of iteration, and synthetic-data recipes are a default ingredient in modern post-training. The public benchmark becomes a de facto optimization target whether or not anyone looks at the test items.

This is why exact-match deduplication and canary tests can produce false reassurance. A clean canary score does not imply a clean construct — it implies that one specific contamination pathway is closed.

LiveCodeBench's value is that it directly cuts the first two pathways: by indexing problems by release date and supporting evaluation on post-cutoff items, it provides a contamination-resistant temporal slice for any model whose training cutoff is known (Jain et al.). It does not close the other pathways. Public discussion of contest problems still flows into training data, and contest formats are themselves a target.

Train-Test Leakage Detection — the gap between exact-match audits and the contamination spectrum that actually matters.

Measurement collapse

The third mechanism is the subtlest and the one most likely to be underestimated. A benchmark "saturates" in the measurement-collapse sense when the score becomes high not because the underlying capability is mastered, but because the model has learned the benchmark's surface conventions:

• Format priors. Multiple-choice priors, A/B/C/D answer-letter biases, common stem patterns, and four-option versus ten-option distributions.
• Scoring-script idiosyncrasies. Exact-match scoring that ignores semantic equivalence; pass@k scoring that rewards verbose attempts; answer-extraction regexes that reward specific phrasing.
• Prompt and instruction conventions. "Let's think step by step," chain-of-thought prefixes, and other prompt-template effects that drive several points of score variation without changing capability.
• Item-template shortcuts. Patterns of the form "X is to Y as A is to ___" that admit format-level pattern completion without engaging the underlying reasoning.

The empirical signatures of measurement collapse are large variance under semantically equivalent reformatting, sensitivity to label corrections, and large gaps between original-format performance and performance on construct-preserving variants. GSM-Symbolic shows this directly for grade-school math: the same underlying problems, restated in a templated symbolic form or extended with semantically inert clauses, produce performance drops that grow with the manipulation (Mirzadeh et al.). MMLU-Pro's reduced prompt sensitivity (~2% versus 4–5%) is partial evidence that some of MMLU's score variance was format noise rather than knowledge variation (Wang et al.).

Measurement collapse is the mechanism most relevant to claims that "the benchmark is solved." It is possible to score 95% on a static benchmark and 70% on a construct-preserving rewrite with the same items. When that gap exists, the original score is a measurement artifact, not a capability statement.

Why these mechanisms cannot be cleanly separated

Capability ceiling, contamination, and measurement collapse co-occur, and treating them as a three-way bucket is a category error. A real saturated benchmark might be carrying:

• 50% genuine progress on the underlying construct,
• 20% direct or near-duplicate exposure of the public test set,
• 20% selection pressure from labs tuning recipes against the visible leaderboard,
• 10% format-priors and scoring-script effects,

and the relative weights will shift across model families and across benchmark generations. The article-level point is that any narrative that picks one explanation as the single cause is almost certainly wrong about a saturated public benchmark in 2024–2026.

Goodhart's Law and Benchmark Optimization — once a measurement becomes an institutional target, the relationship between measurement and underlying construct degrades.

Counter-design case studies

The benchmark literature has produced several specific responses to saturation. Each is best read as a counter-design aimed at a particular mechanism rather than as a general cure.

LiveCodeBench: temporal holdout against literal contamination

LiveCodeBench addresses one mechanism — exact-item exposure from the public web — through time-indexing. By continuously scraping problems from LeetCode, AtCoder, and Codeforces with publication dates attached, the benchmark lets evaluators score any model only on problems released after that model's training cutoff (Jain et al.). Reported scores are time-segmented, and the gap between pre-cutoff and post-cutoff performance is itself a contamination probe.

The honest characterization is contamination-resistant by design, not contamination-proof. Solution discussions and editorial walkthroughs of contest problems still flow into pretraining and post-training corpora, and the contest-programming construct is narrower than software engineering as practiced in real repositories. LiveCodeBench is a strong improvement over static HumanEval-style sets for the question "did the model see this exact problem?" It is not the answer to "does the model write production code?"

Dynabench and Dynaboard: dynamic adversarial collection

Dynabench reframes evaluation as model-in-the-loop, human-adversary data collection, where humans probe models for failures and failure cases enter future evaluation rounds (Kiela et al.). Dynaboard extends the framing into a leaderboard with multiple metrics — accuracy, robustness, fairness, throughput, memory — combined into a utility score that captures non-accuracy properties (Ma et al.).

These designs attack a different mechanism: static-leaderboard incentives that reward accuracy on a fixed distribution. The trade-off is that dynamic, model-in-the-loop benchmarks make longitudinal comparison harder. A 2024 score and a 2026 score on a dynamic benchmark may not measure the same population of items, because the live distribution shifts as new failure cases enter and as the adversary pool changes. Comparability across rounds becomes a governance problem rather than a measurement guarantee.

The institutional cost is also real. Dynamic benchmarks need hosting, evaluator pools, governance over how the live distribution evolves, and adjudication when the metric mix is challenged. These are harder to sustain than a static dataset and a published leaderboard.

Dynamic Benchmark Design — model-in-the-loop adversarial collection, comparability under evolving distributions, governance overhead.

GSM1k: matched fresh holdout against overfitting

The GSM1k construction targets the overfitting variant of contamination. Rather than replacing GSM8K with a different task, it builds a 1,250-item, human-written, distribution-matched set held in private until evaluation (Zhang et al.). When models are evaluated on both the public GSM8K and the matched private set, the gap is a direct overfitting probe.

Frontier systems showed small gaps; certain mid-tier and open model families showed drops of up to 8%. This is informative because it neither vindicates the "everything is contamination" view nor the "models have solved math" view. It supports the more careful conclusion that overfitting is real and family-specific, and that the construct is partially mastered at the frontier.

Matched fresh holdouts are the cheapest decisive probe for the contamination question on a specific benchmark. They are also limited: a fresh matched set tells you about generalization across items drawn from the same distribution, not about generalization across distributions. A model can pass GSM1k and still fail mathematical reasoning that does not look like grade-school word problems.

MMLU-Pro and MMLU-Redux: refinement versus replacement

MMLU-Redux is a re-audit of the original MMLU items, finding annotation errors and question-quality issues at scale (Gema et al.). MMLU-Pro is a harder replacement, with more reasoning-intensive questions and ten answer choices instead of four, designed to restore discrimination at the frontier (Wang et al.).

These two responses are complementary, and together they illustrate why "make a harder benchmark" is not a complete fix. The MMLU-Redux work shows that part of the saturation story was always about label noise — high reported scores were partly a product of mismeasurement, not of mastery. MMLU-Pro shows that re-engineering the construct can recover discrimination, but at the cost of breaking longitudinal comparability with MMLU and starting the saturation clock again on a new public artifact.

MATH-500 as cautionary trade-off

MATH-500 is the best illustration of why "stable subset" should not be confused with "decontaminated benchmark." A 500-problem slice of MATH became a default evaluation surface because it was small enough for fine-grained step-level scoring and stable enough for cross-paper comparison. The same properties that make it useful — small, fixed, high-attention — make it a concentrated overexposure target. Each citation, each paper that reports MATH-500 as a headline number, increases the incentive to optimize specifically for those 500 problems.

This is not a unique problem with MATH-500. It is the generic failure mode of any compact, widely cited evaluation subset. The lesson is that subset selection for convenience and subset selection for validity are different problems, and conflating them is a benchmark-lifecycle hazard.

What evidence would actually separate the explanations

The article-level decision tree above is implementable as a concrete experiment, and laying it out helps clarify which evidence is informative and which is not.

For a saturated benchmark, the cheapest discriminating experiment is something like:

1. Build a fresh matched set. 300–1,000 items, human-written, distribution-matched to the original benchmark, with independent label review. Concealed until evaluation.
2. Construct three probes per item.
   • Exact-format replicas — same construct, same surface form as the original.
   • Paraphrased variants — same construct, equivalent restatement.
   • Construct-preserving format shifts — the same underlying capability, presented in a different surface format (symbolic templating, alternate answer-extraction format, restructured prompt).
3. Evaluate with frozen prompts and preregistered scoring. Multiple answer-extraction protocols, reported separately. Decoding parameters fixed.
4. Audit labels. A non-trivial fraction of saturated benchmarks contain label errors that inflate scores; the audit is part of the experiment, not a separate concern.
5. Compare four numbers per model: public-set score, fresh-replica score, paraphrased score, format-shifted score.

The interpretation rules follow directly from the decision tree:

• If all four scores are similar and high, the capability-ceiling account is supported on the construct as defined.
• If the public-set score is much higher than the fresh-replica score, contamination or overfitting is implicated. The mechanism is narrower if the gap is family-specific, broader if it is universal.
• If the format-shifted score is much lower than the paraphrased and replica scores, measurement collapse is implicated. The benchmark was scoring the format, not the construct.
• Mixed gaps imply mixed causes, which is the realistic case.

This experimental pattern is directly visible in the GSM8K → GSM1k → GSM-Symbolic sequence (Zhang et al.; Mirzadeh et al.). It is partially visible in the MMLU → MMLU-Redux → MMLU-Pro sequence (Gema et al.; Wang et al.). It is the experimental design the field tends to converge on when it takes a saturated benchmark seriously.

It is also expensive enough that most labs do not run it before claiming "X% on Y benchmark." Reading public scores without the matched-fresh, perturbation, and label-audit context is reading a measurement without its calibration data.

Lifecycle design and its limits

The standard set of lifecycle responses to saturation can be summarized in three families, each with a distinct cost profile.

Approach	What it does well	What it costs
Static benchmark with stricter hygiene (canary splits, near-duplicate audits, provenance tracking, private holdouts)	Maximum reproducibility, historical comparability, low operational overhead	Still accumulates overexposure; private holdouts reduce independent verifiability; selection pressure persists
Dynamic benchmark lifecycle (refresh items, time-indexed evaluation, model-in-the-loop adversarial collection, private evaluation windows)	Better against contamination and stale format-matching; reflects evolving capability frontier	Harder governance, higher hosting and evaluator cost, weaker year-to-year comparability, vulnerable to non-stationary distribution drift
Beyond benchmarks (task-based audits, longitudinal deployment evaluations, agent work products, human-in-the-loop studies)	Better aligned with frontier behavior in the wild; harder to game	Expensive, less standardized, harder to cite as clean leaderboards, requires per-task design

The recommendation is layered rather than singular. Static anchors with hygiene give continuity; dynamic and private components give validity at the frontier; broader paradigms give signal on capabilities that benchmarks cannot compress cleanly. Treating any layer as a complete solution misreads the trade-offs.

Second-order costs of each fix

Each lifecycle response carries failure modes that are themselves worth naming.

Held-out private splits reduce contamination risk but create a trust problem. When a benchmark provider reports private-split scores, the community must accept the provider's word about score changes — independent verification disappears. Over time, this creates a benchmark-provider governance burden that did not exist for static public sets.

Dynamic refresh reduces staleness but breaks longitudinal claims. A model that improves from one refresh to the next may be improving because it is better, or because the refresh shifted the distribution. Without careful versioning and parallel reporting on overlapping splits, the comparison degenerates into "the score went up."

Frequent refresh cadences reduce overexposure but incentivize benchmark providers to optimize for novelty rather than construct validity. There is a meta-Goodhart problem: when "fresh" becomes the metric, providers optimize for freshness at the expense of careful construct design.

Contamination audits can reassure falsely. Exact-match deduplication misses paraphrases, derivative training material, and synthetic-data contamination through public discussions. A clean canary report is a clean report on canaries, not on the broader contamination spectrum.

Adversarial dynamic collection introduces evaluator-quality variance. The benchmark is now only as good as the adversary pool, and "this is harder than the previous round" can reflect smarter adversaries rather than weaker models.

The article-level point is not that lifecycle redesign is hopeless — it is that every fix moves the failure mode rather than eliminating it. The job is to distribute the failure modes across enough independent layers that no single mechanism dominates the measurement.

Held-Out Evaluation Splits · Time-Indexed Benchmarks · Perturbation-Based Robustness Testing · Label Quality Auditing

Detection signals: when a benchmark is failing

A saturated benchmark broadcasts its failure modes if you watch for the right signals. The following are practical heuristics for whether a public number should be trusted.

• Public-private gap. When a public-set score is much higher than performance on a matched private set, treat the public number as inflated. The gap is the contamination/overfitting estimate.
• Format perturbation sensitivity. Large score variance under semantically equivalent reformatting is direct evidence of measurement collapse. If swapping multiple-choice letters or restructuring the answer format moves the score by more than a couple of points, the format is part of what is being measured.
• Label-correction sensitivity. When audits like MMLU-Redux find that score rankings shift after correcting label errors, the original benchmark was partly measuring annotation noise.
• Post-release score discontinuities. Large jumps shortly after a benchmark is released and discussed publicly suggest leakage, recipe adaptation, or selection pressure rather than capability progress.
• Ranking churn after refreshes. When a benchmark refresh reorders the leaderboard substantially, the previous ordering was tracking benchmark-specific overfitting rather than general capability.
• Saturation in narrow constructs but not in cousins. When models hit the ceiling on GSM8K but stumble on GSM-Symbolic, the saturation is local to the construct's surface form, not to the underlying skill.

These signals are diagnostic rather than dispositive. None of them alone proves contamination or measurement collapse; all of them together constitute the calibration data that should accompany any saturated public number.

The open question: measurement repair, or different paradigms?

The core unresolved question is whether benchmark saturation is intrinsically a measurement-design problem — solvable in principle by better hygiene, dynamic refresh, and stricter construct-validity work — or whether it is evidence that frontier evaluation needs different paradigms entirely.

The case for measurement repair is real. Many of the response designs cited in this article work. LiveCodeBench's temporal indexing closes a meaningful contamination channel (Jain et al.). GSM1k separates overfitting from generalization on a specific construct (Zhang et al.). MMLU-Redux recovers calibration that label noise had obscured (Gema et al.). MMLU-Pro restores discrimination at the frontier with construct-preserving redesign (Wang et al.). In each case, careful lifecycle work converts a saturated, ambiguous public number into a more informative measurement.

The case for paradigm shift is also real. Static and dynamic benchmarks alike measure short, well-bounded tasks with automatic scoring. They do not measure sustained work, multi-step tool use, calibration under uncertainty, behavior under distribution shift, robustness to adversarial prompts, or alignment of declared and actual capability. These properties matter increasingly for systems deployed as agents, and they do not compress cleanly into a public leaderboard. Evaluation designs aimed at them — agent task suites, longitudinal deployment studies, scenario-based audits, capability-specific scientific probes — look more like measurement infrastructure than like benchmarks in the traditional sense.

The honest verdict is that both cases are simultaneously correct. Lifecycle repair is necessary because public static benchmarks remain the cheapest, most reproducible cross-lab comparison the field has, and giving them up is a coordination failure even when individual benchmarks deserve retirement. Paradigm shift is also necessary because the most capability-relevant evaluation questions for frontier systems are not the questions traditional benchmarks were built to answer.

What this article will not do is resolve the tension. A clean ending — "the answer is harder benchmarks" or "the answer is to abandon benchmarks for deployment audits" — would be a forced synthesis that the evidence does not support. The defensible position is plural: use lifecycle repair to slow the obvious failure modes; use fresh matched, perturbation, and label-audit probes to separate the three mechanisms when a benchmark saturates; use task-specific and deployment-grounded evaluations when the static benchmark stops tracking the capability that actually matters; and treat the public leaderboard as one signal among several rather than the primary score.

Saturation is not the disease, in this reading. Saturation is what a public benchmark does when it succeeds at being interesting enough that a research community optimizes against it. The diseases are the underlying mechanisms — exposure, recipe selection, format priors, label noise, narrow construct definition — that the saturation event invites you to investigate. Treat saturation as a prompt to do that investigation, and the benchmark lifecycle becomes coherent. Treat saturation as a verdict, and you will end up over-claiming progress, over-accusing contamination, or over-engineering a measurement-design solution to what is partly an institutional incentive problem.

Capability Ceilings vs Format Ceilings · The Public Leaderboard Incentive Problem

Companion entries

Core theory:

• Construct Validity in AI Evaluation
• Goodhart's Law and Benchmark Optimization
• Capability Ceilings vs Format Ceilings
• The Public Leaderboard Incentive Problem

Practice:

• Held-Out Evaluation Splits
• Time-Indexed Benchmarks
• Dynamic Benchmark Design
• Perturbation-Based Robustness Testing
• Label Quality Auditing
• Train-Test Leakage Detection
• The MMLU Family Tree
• Contamination-Resistant Coding Evaluation

Counterarguments and limits:

• Why Static Benchmarks Still Matter
• The Reproducibility Cost of Private Holdouts
• Dynamic Benchmarks and the Comparability Problem

Related paradigms:

• Evaluation Beyond Benchmarks: Agent Tasks and Deployment Studies
• Longitudinal Deployment Evaluation
• Capability-Specific Scientific Probes
• Red Teaming as Evaluation Infrastructure