FrontierMath: Research-Level Mathematics Evaluation
Status at a glance
| Dimension | FrontierMath claim | Current assessment |
|---|---|---|
| Core purpose | Measure advanced mathematical reasoning beyond contest benchmarks and ordinary graduate problem sets. | Strong conceptual fit for Reasoning Benchmarks, especially when compared with saturated exams such as MATH/AIME-style tasks. |
| Dataset design | Original problems authored and vetted by expert mathematicians; answer format chosen for automated checking. | Methodologically strong, but the May 2026 error audit materially weakens any simple “gold standard” framing. arXiv |
| Difficulty | Initial frontier models scored below 2%; later reasoning models crossed 20% and then 50% on some tiers under high-compute or vendor-reported settings. | Real performance growth is clear, but leaderboard comparability is difficult because scaffolds, token budgets, tool access, and problem subsets differ. arXiv+2The Verge+2 |
| Evaluation style | Closed-answer, mostly exact numeric/symbolic outputs, all-or-nothing scoring. | Excellent for unambiguous grading; weak as a proxy for proof writing, conjecture formation, long-horizon research, and mathematical taste. |
| Benchmark longevity | Designed to resist saturation by current models. | It may remain useful as a maintained evaluation program, but the static Tiers 1–4 set is unlikely to remain a durable ceiling without renewal, audit, and versioning. |
What FrontierMath is
FrontierMath, introduced by Glazer et al. in 2024, is a benchmark of original mathematical problems intended to be difficult for frontier language models and useful for tracking progress toward Research-Level AI. The original paper describes “hundreds” of new problems written and vetted by expert mathematicians, spanning areas such as number theory, real analysis, algebraic geometry, and category theory; the authors reported that then-current state-of-the-art models solved fewer than 2% of the problems. arXiv
The benchmark is not simply “hard math.” It is a deliberately engineered compromise between research mathematics and machine evaluation: problems should require nontrivial mathematical reasoning, but their final answers should be checkable automatically. This places FrontierMath in the family of Closed-Answer Evaluation rather than proof-assistant benchmarks, theorem-proving benchmarks, or open-ended research-agent evaluations. Epoch AI
Epoch’s current public description divides the FrontierMath program into Tiers 1–4 and a separate Open Problems track. Tiers 1–3 range from difficult undergraduate or graduate-level mathematics through very advanced material, while Tier 4 is described as research-level; the Open Problems track is distinct and contains known unsolved mathematical problems rather than benchmark-style hidden-answer tasks. Epoch AI
This distinction matters. Solving a FrontierMath problem is evidence that a model can execute a difficult chain of mathematical reasoning under a benchmark scaffold. It is not direct evidence that the model can originate a research program, choose fruitful abstractions, write a publishable proof, survive adversarial peer review, or decide which mathematical questions are worth asking. FrontierMath is best treated as a high-ceiling probe of AI Mathematical Problem Solving, not as a complete operational definition of mathematical research ability.
Design goals
FrontierMath was created against two background problems in AI evaluation: benchmark saturation and data contamination. Saturation occurs when a benchmark no longer discriminates among frontier systems because many of them score near the top; contamination occurs when test examples, solutions, or near-duplicates enter model training data or evaluation scaffolds, undermining the interpretation of scores. Glazer et al. explicitly frame FrontierMath as a response to these problems by using newly authored, unreleased problems that are hard enough to leave substantial headroom. arXiv
The design aims can be summarized as follows:
| Goal | Implementation choice | Why it matters |
|---|---|---|
| High ceiling | Problems written by expert mathematicians, often requiring advanced graduate or research-level ideas. | Prevents immediate saturation by models that already perform well on competition math. |
| Objective grading | Final answers are exact, symbolic, numeric, or otherwise machine-checkable. | Reduces dependence on LLM-as-judge grading, which is fragile for subtle mathematics. |
| Contamination resistance | Problems are newly authored, privately held, and checked against public sources. | Makes memorization or retrieval less plausible than on public datasets. |
| Comparability | Models solve under a standardized scaffold and submit answers through a constrained interface. | Supports leaderboard-style evaluation, though scaffold differences still matter. |
| Mathematical breadth | Problems cover many Mathematical Subject Classification areas. | Avoids reducing “mathematics” to olympiad combinatorics or calculus-style exercises. |
The central design tension is visible here: FrontierMath wants problems that are close enough to research mathematics to be hard, but constrained enough to admit automatic scoring. That tension explains both its value and many of its weaknesses.
Construction methodology
Expert authorship
The original construction pipeline recruited more than 60 mathematicians across more than a dozen countries, including contributors with strong mathematical credentials such as International Mathematical Olympiad gold medals and a Fields Medalist. The cited subject areas included algebraic geometry, number theory, topology, combinatorics, category theory, partial differential equations, probability, differential geometry, logic, and theoretical computer science. arXiv
This is a major departure from many earlier math benchmarks. Datasets such as GSM8K, MATH, and olympiad-style collections are often assembled from existing educational or competition material. FrontierMath instead pays or recruits specialists to create new problems with an intended adversarial property: the problems should not be solvable by pattern-matching against familiar public examples.
Expert authorship also introduces a bias. The benchmark inherits the mathematical taste, local norms, and problem-design habits of its authors. A problem that an algebraic geometer considers elegant may be adversarially obscure to a model trained mostly on public exposition; a number theorist may naturally choose problems whose final answer is a large integer because those are easier to verify automatically. These are not fatal flaws, but they mean FrontierMath measures “performance on this authored distribution of hard mathematical puzzles,” not “mathematical ability” in the abstract.
Problem requirements
Glazer et al. describe several requirements for problem submission: originality, automated verifiability, resistance to guessing, computational tractability, a detailed solution, a verification script, and metadata. Many answers are represented as integers, rational numbers, or SymPy-compatible symbolic objects, allowing a model’s output to be checked by code rather than by a human evaluator. arXiv
This requirement strongly shapes the benchmark. It selects for mathematical questions whose endpoints are crisp: compute this invariant, determine this coefficient, find this exact value, identify this finite structure, or return a symbolic expression. It selects against many forms of real mathematical work: proving a new theorem, finding the right definition, comparing proof strategies, writing exposition, diagnosing a false conjecture, or building a theory over months.
The result is an intentionally narrow but valuable instrument. In evaluation terms, FrontierMath trades ecological validity for scoring reliability. That trade is reasonable, but it should be made explicit whenever scores are cited.
Verification and peer review
The original paper reports that each submitted problem was blind peer reviewed by at least one expert mathematician, who checked correctness, ambiguity, difficulty, metadata, and susceptibility to guessing. The authors also discuss additional review stages and note that passive review has limitations: in a second-review sample, some problems had incorrect answers, missing hypotheses, or guessability issues. arXiv
That admission is important. Even before the 2026 audit, the paper itself did not justify the naive claim that expert-authored benchmark problems are automatically correct. Hard mathematics is brittle: small missing hypotheses can change a theorem; a hidden degeneracy can break a construction; an answer-checking script can encode the author’s mistake; a problem may be solvable by a shortcut the author did not anticipate.
The May 2026 Epoch update makes this caveat central rather than peripheral. Epoch now reports that an AI-assisted review flagged fatal errors in about one third of Tiers 1–4 problems, while also stating that they believe most problems remain valid and that updated scores will be released after human review. Epoch AI
For a benchmark whose main attraction is unambiguous correctness, this is a serious event. It does not invalidate FrontierMath as a benchmark concept, but it weakens retrospective leaderboard claims and shows that research-level benchmark construction needs continuous audit, not one-time expert review.
Tiered difficulty
Epoch currently describes the private Tiers 1–3 dataset as 300 base problems plus 50 Tier 4 problems, with 10 public Tiers 1–3 examples and two public Tier 4 examples. Public hub scores are reported on the private sets unless otherwise stated. Epoch AI
The tiering system is meant to create a difficulty gradient. Tiers 1–3 cover hard but bounded problems across undergraduate, graduate, and advanced graduate mathematics. Tier 4 is intended to be research-level. The Open Problems track is separate, and it is more properly read as a stress test for autonomous mathematical discovery than as a conventional benchmark. Epoch AI
The small size of Tier 4 is one of the benchmark’s most important statistical limitations. A 48-problem private Tier 4 set means one problem corresponds to about 2.1 percentage points. If several problems are ambiguous, invalid, or unusually scaffold-sensitive, leaderboard movements of several percentage points may reflect benchmark artifacts rather than model capability.
Evaluation protocol
The current Epoch hub protocol asks models to submit a Python answer() function with no parameters, returning a valid answer object such as an integer, string, fraction, list, dictionary, or SymPy object. Models can reason, use Python code, and produce intermediate work, but only the final submitted answer is scored; scoring is all-or-nothing. The current hub has used a hard 1,000,000-token limit and allows code execution before final answer submission. Epoch AI
This is a crucial detail for interpreting scores. FrontierMath is not a pure next-token-prediction exam. It is an evaluation of an AI system operating under a scaffold that may include chain-of-thought-style reasoning, scratch work, Python execution, repeated attempts, and a final answer wrapper. In other words, FrontierMath is already partly a benchmark of Test-Time Compute and Tool-Augmented Reasoning, not merely of the base model.
The original paper’s experimental setup was more constrained, including a 10,000-token limit in the reported experiments. The current hub differs from both the original paper setup and some private vendor evaluations, and Epoch has explicitly noted that score differences can arise from scaffolds, test-time compute, and problem-subset differences. arXiv
This has two consequences. First, a FrontierMath score should always be reported with its scaffold, token budget, tool policy, number of attempts, and problem subset. Second, “model X scored Y%” is usually underspecified. The more accurate statement is “model X, under scaffold S, with compute budget C, on subset D, under scoring rule R, scored Y%.”
Dataset composition and domain coverage
The original paper emphasizes mathematical breadth, with coverage across many Mathematical Subject Classification areas. It also reports a skew toward areas amenable to exact answer verification, including substantial representation from number theory and combinatorics. arXiv
This skew is not accidental. Number theory and combinatorics often yield compact exact answers and can be made highly nontrivial without requiring long prose proofs. Algebraic geometry, category theory, topology, and analysis can also be represented, but only through problem formats that culminate in a checkable answer. The more a domain’s difficulty lies in proof architecture, definitions, or qualitative reasoning, the harder it is to include without moving away from automated grading.
The benchmark therefore covers “advanced mathematics” better than most earlier AI benchmarks, but it does not cover mathematics uniformly. It is especially strong for problems with hidden structure and exact endpoints. It is weaker for proof-oriented mathematical practice, experimental conjecture formation, long-term theory building, and collaborative research workflows.
Empirical performance trajectory
Initial frontier models: below 2%
In the original FrontierMath paper, six early frontier models were evaluated and none reached 2% accuracy. The authors also reported that only four problems were solved at least once across those initial evaluations, making rankings among early models low-signal. arXiv
This result established FrontierMath’s early reputation: it was one of the rare benchmarks where contemporary frontier models looked almost completely ineffective. That mattered because many public AI benchmarks by 2024 had already entered a saturation regime or were vulnerable to training contamination.
The initial result also had a downside. When almost every model scores near zero, the benchmark demonstrates difficulty but provides limited resolution. A benchmark that is too hard can be as uninformative as one that is too easy, at least for ranking systems within a narrow capability band.
o3 and o3-mini: the first visible jump
OpenAI’s December 2024 o3 announcement reported a major jump, with o3 solving 25.2% of FrontierMath problems under the evaluation conditions OpenAI described, compared with prior models below 2%. The Verge
OpenAI’s January 2025 o3-mini release page reported that o3-mini, with high reasoning effort and Python-tool access, solved over 32% of FrontierMath on a first attempt, with more than 28% on Tier 3; OpenAI marked those numbers as provisional. OpenAI
These results changed the interpretation of FrontierMath. The benchmark was no longer just a “models cannot do this” demonstration. It became a measurement surface for the interaction between frontier models, reasoning-specific training, tool use, and high test-time compute.
However, independent comparability remained difficult. Reporting at the time noted discrepancies between OpenAI’s internal o3 numbers and Epoch hub results, with differences attributed to factors such as aggressive test-time compute, scaffolding, and different problem subsets. TechCrunch
GPT-5-era models: high scores under high-compute settings
By 2026, recent reasoning models had pushed much higher. OpenAI’s GPT-5.5 system card-style release table reports FrontierMath Tiers 1–3 scores of 51.7% for GPT-5.5 and 52.4% for GPT-5.5 Pro, and Tier 4 scores of 35.4% for GPT-5.5 and 39.6% for GPT-5.5 Pro. The same table notes that these benchmark runs were conducted with high reasoning effort in a research environment and may differ from production ChatGPT behavior. OpenAI
Epoch’s March 2026 discussion of GPT-5.4 similarly reported record-setting FrontierMath performance, including 50% on Tiers 1–3 and 38% on Tier 4 for GPT-5.4 Pro under their described setup. Epoch also noted that OpenAI had access to many problem statements and solutions, while held-out/non-held-out comparisons were not statistically significant in the reported analysis. LinkedIn
These numbers justify saying that recent reasoning models have broken the 20% barrier and, under some high-compute or vendor-reported conditions, are approaching or exceeding 50% on the lower FrontierMath tiers. They do not justify saying that FrontierMath is solved. Tier 4 remains materially harder, Open Problems are a different category, score comparability is scaffold-sensitive, and the May 2026 problem-audit issue means corrected scores may differ from currently cited numbers. Epoch AI
What the trajectory shows
The empirical trajectory is best summarized as a three-stage shift:
| Stage | Approximate period | What changed | Interpretation |
|---|---|---|---|
| Near-zero regime | 2024 paper-era models | Frontier systems scored below 2%. | FrontierMath successfully exposed a capability gap. |
| Breakthrough regime | o3 / o3-mini period | Reasoning-specialized systems crossed the 20%–30% range under high-compute/tool settings. | Reasoning Models and test-time compute mattered more than raw pretraining scale alone. |
| Partial-saturation regime | GPT-5-era systems | Tiers 1–3 reached roughly 50% in leading reported results; Tier 4 reached roughly high-30s to around 40%. | The benchmark still has headroom, but its static lower tiers are no longer safely far from saturation. |
The most important inference is not that models “became mathematicians.” It is that the combination of stronger reasoning training, better scaffolds, larger context/token budgets, Python execution, and repeated search can solve a nontrivial fraction of problems that were designed to resist near-term frontier models. That is a genuine capability gain.
Methodological strengths
1. Unambiguous correctness as an evaluation target
FrontierMath’s strongest idea is to force advanced mathematics into answer formats that can be checked precisely. This avoids a common failure mode in evaluating mathematical reasoning: the model writes a plausible proof-like explanation, the grader is another model or a non-expert human, and errors slip through. By scoring the final answer against a ground-truth checker, FrontierMath makes correctness harder to fake.
This strength should now be stated carefully. The benchmark’s design targets unambiguous correctness, but the May 2026 audit shows that problem-level correctness was not fully secured in practice. The right conclusion is not “automated answer checking is useless,” but rather “automated answer checking is only as reliable as the mathematical specification and reference solution behind it.” Epoch AI
2. Authored ground truth
Because problems are authored with solutions, verification code, and metadata, FrontierMath can attach each problem to an intended mathematical construction. This is preferable to scraping arbitrary hard problems from the internet, where answer keys may be missing, ambiguous, or already present in training corpora.
Authored ground truth also enables adversarial design. Authors can create problems that require combining methods rather than applying a standard theorem template. The original paper describes broad technique coverage and emphasizes problems requiring unusual combinations of ideas. arXiv
The weakness is that authored ground truth is not the same as independently formalized truth. Unless solutions are machine-checked in systems such as Lean, Isabelle, or Coq, expert-authored answers remain fallible. FrontierMath’s 2026 audit is evidence for that failure mode.
3. Contamination resistance
FrontierMath’s original contamination story is stronger than that of public exam datasets. Problems are newly written, privately managed, and checked against public sources using plagiarism-detection tools; Glazer et al. report no significant public matches in their checks. arXiv
But contamination resistance is not binary. Epoch later clarified that OpenAI commissioned hundreds of questions, owns and has access to many problems and solutions, and that disclosure to contributors was imperfect in the early process. Epoch also described a held-out subset intended to preserve independent evaluation. Epoch AI
This governance history does not prove that models trained on FrontierMath solutions. It does mean that “private benchmark” should not be read as “unknown to every model developer.” For high-stakes leaderboard interpretation, access controls, held-out splits, and disclosure matter almost as much as the mathematical content.
4. High ceiling
The original under-2% result demonstrated an unusually high ceiling relative to common AI math benchmarks. Even after large performance gains, FrontierMath Tier 4 and the Open Problems track retain headroom. Epoch’s later pass@N analysis argued that, even with many attempts, GPT-5-era performance appeared to plateau well below full coverage, although this analysis predates the May 2026 correction process. Epoch AI
A benchmark with headroom is valuable because it can distinguish not just today’s best model from last year’s, but also different styles of progress: better base reasoning, better search, better theorem retrieval, better symbolic computation, better self-verification, and better long-context planning.
5. Tool realism
FrontierMath’s Python-enabled scaffold is closer to how many mathematicians actually work than a no-tools exam. Mathematicians compute examples, test conjectures, factor polynomials, search small cases, and use software to guide intuition. A model that can coordinate abstract reasoning with code execution is closer to a useful mathematical assistant than a model limited to static text completion.
This is also why FrontierMath results are difficult to compare. Tool use creates degrees of freedom: code interpreter quality, search strategy, number of attempts, timeout policy, token budget, and scaffold design can all change scores. But the alternative—no tools at all—would be less faithful to modern mathematical practice.
Methodological weaknesses
1. Small N and large error bars
The private Tiers 1–3 set contains 290 private problems, while Tier 4 contains 48 private problems under the current public description. Epoch AI
This is large compared with some hand-built expert benchmarks, but small compared with ordinary machine-learning test sets. At Tier 4, each problem is about 2.1 percentage points. A single invalid problem, duplicated trick, unusually easy shortcut, or scorer bug can materially affect reported performance. A model improving from 38% to 40% on Tier 4 may have solved one additional problem, not crossed a robust capability threshold.
Small N also interacts with domain skew. If a model is unusually strong at computational number theory but weak at analysis, a handful of domain-heavy problems can make it look more or less capable than it is. For a benchmark this expensive, some small-N problem is unavoidable, but leaderboard discourse should not ignore it.
2. Narrow domain coverage despite broad labels
FrontierMath covers many mathematical areas, but only through answer-checkable tasks. That makes it narrower than “mathematical research.” It does not directly evaluate proof writing, formalization, peer-review resilience, literature search, taste, collaboration, or the ability to discover a useful intermediate lemma without a prompt.
This matters because the strongest AI systems may learn to optimize for closed-answer mathematical puzzles. A model can become excellent at computing hard invariants while still being unreliable as a proof author. Conversely, a theorem-proving agent may perform poorly on FrontierMath if it lacks numerical search or problem-specific heuristics.
3. Expert authorship bias
Expert-authored benchmarks can overrepresent the kinds of problems experts enjoy writing. They can also overrepresent problems that are elegant to humans but pathological for machines, or vice versa. The problem distribution is not sampled from “all mathematics”; it is designed.
That design is useful if the benchmark’s purpose is stress testing. It is less useful if one wants a representative estimate of productivity in mathematical research. The right mental model is adversarial evaluation, not population sampling.
4. Ground-truth fragility
FrontierMath’s greatest methodological strength—authored exact answers—is also a point of fragility. If the problem statement is missing a hypothesis, if there are multiple interpretations, if the checker accepts the wrong object, or if the intended solution contains a hidden gap, all-or-nothing scoring becomes misleading.
The May 2026 Epoch update is therefore not a minor footnote. It is a direct stress test of the benchmark’s reliability model. The fact that AI-assisted review could flag fatal errors in about a third of Tiers 1–4 problems suggests that future versions need stronger pre-release validation, independent re-solving, adversarial review, and ideally formal or semi-formal verification for at least some problem classes. Epoch AI
5. Scaffold and compute sensitivity
FrontierMath performance is highly sensitive to test-time compute. OpenAI’s o3 and o3-mini results were tied to high reasoning settings and Python tool access; Epoch and other analyses later emphasized that different scaffolds and problem subsets can lead to different observed scores. OpenAI
This means FrontierMath is partly a benchmark of Inference-Time Search. That is not a defect if reported transparently. It is a defect when leaderboards collapse model, scaffold, compute budget, and sampling policy into one number.
A useful FrontierMath report should include at minimum:
| Required metadata | Why it matters |
|---|---|
| Model version | Capability can change rapidly across releases. |
| Problem subset and benchmark version | Current scores are not comparable across invalidated or revised problem sets. |
| Token budget | More reasoning budget can change solve rates dramatically. |
| Tool policy | Python access and symbolic libraries affect performance. |
| Number of attempts / pass@k | Single-shot accuracy and multi-attempt search measure different capabilities. |
| Scaffold details | Prompting, verification loops, and self-critique can move scores. |
| Access status | Developer access to statements or solutions changes contamination interpretation. |
Relationship to Humanity’s Last Exam
Humanity’s Last Exam is a broad expert-level benchmark spanning many academic subjects, not a mathematics-specific benchmark. The Nature article describing HLE characterizes it as a 2,500-question multimodal benchmark across dozens of subjects, with multiple-choice and short-answer questions, expert review, automated grading, and attempts to select questions resistant to quick internet retrieval. Nature
HLE’s strength is breadth. It tests whether a model can answer hard questions across the sciences, humanities, mathematics, and multimodal domains. It is useful for measuring general expert-level factual and reasoning competence.
FrontierMath’s strength is depth. It trades disciplinary breadth for mathematical difficulty, exact answer checking, and problem originality. A model can score well on HLE because it is broadly knowledgeable and tool-competent while still failing FrontierMath Tier 4. Conversely, a system specialized for symbolic mathematics could perform well on FrontierMath while being mediocre on HLE’s humanities or multimodal questions.
| Benchmark | Domain | Typical answer style | Main strength | Main limitation |
|---|---|---|---|---|
| FrontierMath | Advanced mathematics | Exact numeric/symbolic/programmatic answer | Deep math reasoning with low random-guessability | Narrow domain; small N; current QA concerns |
| Humanity’s Last Exam | Broad expert knowledge across disciplines | Multiple-choice and short-answer, including multimodal | Breadth and interdisciplinary coverage | Less focused on deep mathematical problem solving |
| Open Problems track | Actual unsolved math problems | Open-ended research solution | Closest to true discovery | Not a stable benchmark in the usual sense |
The two benchmarks should be treated as complementary. HLE asks whether an AI system has broad expert-level competence. FrontierMath asks whether it can solve deliberately difficult mathematical problems with exact endpoints.
Relationship to GPQA Diamond
GPQA Diamond is a hard graduate-level science question benchmark derived from GPQA, which contains expert-written multiple-choice questions in biology, physics, and chemistry. The original GPQA paper describes 448 questions written by domain experts, with PhD-level experts performing far above skilled non-experts even when those non-experts had web access. arXiv
Epoch’s GPQA Diamond page describes the Diamond subset as 198 high-quality, challenging multiple-choice questions where domain expert annotators answered correctly and many non-domain experts answered incorrectly. It uses four-option multiple-choice answers, which implies a 25% random baseline. Epoch AI
The contrast with FrontierMath is sharp:
| Dimension | FrontierMath | GPQA Diamond |
|---|---|---|
| Field | Mathematics | Biology, chemistry, physics |
| Difficulty source | Original hard mathematical constructions | Expert-level scientific knowledge and reasoning |
| Answer space | Exact numeric/symbolic/programmatic | Four-option multiple choice |
| Random baseline | Usually negligible or problem-specific | 25% |
| Contamination posture | Private, newly authored problems | Public benchmark with controlled subset history |
| Ceiling as of 2026 | Still substantial on Tier 4, but lower tiers have high scores | Many frontier systems report very high scores |
| Main use | Stress-test mathematical reasoning | Test expert scientific QA and reasoning |
OpenAI’s GPT-5.5 release table reports GPQA Diamond scores above 90% for several leading systems, while FrontierMath Tier 4 remains far lower. OpenAI
That does not mean GPQA Diamond is “bad.” It means its difficulty ceiling is different. GPQA Diamond is a strong benchmark for expert scientific question answering, but multiple-choice format and public benchmark dynamics make it less analogous to research-level mathematical problem solving than FrontierMath.
What FrontierMath scores do and do not mean
A high FrontierMath score is evidence that a system can solve a meaningful fraction of hard, closed-answer mathematical problems under a specific scaffold. It suggests competence in theorem recall, symbolic manipulation, decomposition, computational exploration, and self-checking.
It is not direct evidence of autonomous mathematical research capability. In particular, FrontierMath does not require the system to:
| Research activity | Tested by FrontierMath? | Comment |
|---|---|---|
| Prove a theorem in publishable prose | Partly, indirectly | The model may reason internally, but only final answers are scored. |
| Formalize a proof in Lean or Coq | No | Formal proof benchmarks test a different skill. |
| Choose an important research problem | No | The benchmark supplies the problem. |
| Conduct literature review | Mostly no | Public retrieval is not the central task. |
| Invent a new theory | No | Most tasks are bounded puzzle-like problems. |
| Persist over weeks or months | No | Problems are intended for hours or days, not full research programs. |
| Collaborate with humans | No | The evaluation is single-agent or scaffolded system performance. |
| Verify its own proof rigorously | Only through final answer checking | A correct answer can still be reached through flawed reasoning or lucky search. |
Epoch’s qualitative analysis of o3-mini traces supports this distinction. Mathematicians reviewing raw traces described the model as extremely knowledgeable and often able to invoke relevant results, but also found recurring failures in precision, depth, creativity, and hallucination. Epoch AI
This is one of the most useful ways to read FrontierMath. It is not a binary test of “real reasoning.” It is an empirical window into how current systems combine knowledge, heuristic search, computation, and brittle self-verification on problems where shallow memorization should be ineffective.
Will FrontierMath saturate?
The saturation question has two different meanings.
First, will the original static Tiers 1–4 problem set saturate? Probably yes, at least in part. Scores on Tiers 1–3 have already risen from below 2% to around 50% in leading reported results, and high-compute pass@k methods can extract additional solves. Static private benchmarks also become less private over time as more developers obtain access, more public discussion accumulates, and more solved examples leak into the ecosystem. OpenAI
Second, will the FrontierMath methodology remain useful? Probably yes, if treated as a living benchmark program rather than a fixed dataset. The concept of expert-authored, private, automatically checkable, high-difficulty mathematical problems remains sound. But it needs continuous replenishment, independent audit, transparent versioning, and separate reporting by tier, scaffold, and compute budget.
Epoch’s pass@N analysis argued that less than 70% of FrontierMath appeared within reach of GPT-5 under aggressive sampling assumptions, though this analysis should be interpreted in light of the later problem-audit caveat. Epoch AI
The deeper open question is whether future systems will saturate the style of FrontierMath: bounded, exact-answer, expert-written problems. There are three plausible futures:
| Scenario | What happens | Implication |
|---|---|---|
| Static saturation | Models learn to solve most Tiers 1–4 problems under high compute. | FrontierMath becomes historically important but no longer frontier-discriminating. |
| Moving-target durability | New problems, audits, and harder tiers preserve headroom. | FrontierMath becomes an ongoing evaluation program similar to a maintained exam. |
| Capability phase change | Models become strong enough that any human-authored closed-answer benchmark is rapidly solved. | Evaluation must move toward open-ended research, formal verification, and adversarial human-AI co-discovery. |
The third scenario is the most interesting. If models can routinely solve newly authored FrontierMath-style problems, the bottleneck shifts from “can the model solve this?” to “can humans generate, verify, and withhold enough good problems?” At that point, the benchmark’s limitation is no longer model weakness but evaluation supply.
FrontierMath and the philosophy of AI evaluation
FrontierMath exposes a recurring problem in AI Evaluation Philosophy: the more precisely one defines a capability, the easier it is to miss the broader phenomenon. FrontierMath precisely measures a narrow target: hard mathematical problem solving with exact answers. That precision is valuable. It also excludes much of what mathematicians mean by research.
This is not a reason to dismiss the benchmark. It is a reason to avoid inflated interpretations. The right evaluation stack for mathematical AI should include:
| Layer | Example benchmark style | Capability tested |
|---|---|---|
| Closed-answer hard problems | FrontierMath | Exact advanced problem solving |
| Formal theorem proving | Lean/Coq/Isabelle tasks | Machine-checkable proof construction |
| Proof critique | Human or formal review of generated arguments | Error detection and rigor |
| Literature-grounded research | Agent tasks over papers and definitions | Contextual mathematical work |
| Open conjecture generation | Human-reviewed research proposals | Taste and abstraction |
| Long-horizon collaboration | Multi-week human-AI projects | Persistence, correction, and integration |
FrontierMath is strongest in the first layer. It can contribute to the others only indirectly.
Governance and leaderboard interpretation
The OpenAI/Epoch relationship is central to interpreting recent scores. Epoch clarified in 2025 that OpenAI commissioned a set of FrontierMath questions and had ownership/access rights to many problems, while Epoch maintained evaluation rights and a held-out set. Epoch also acknowledged that contributors were not systematically informed of OpenAI’s involvement during early collection and described future disclosure improvements. Epoch AI
This history does not make FrontierMath unusable. It does mean that independent benchmark governance is not a cosmetic issue. For frontier-model evaluation, the following distinctions matter:
| Term | Meaning | Why it matters |
|---|---|---|
| Private | Not fully public on the internet | Reduces ordinary training contamination. |
| Held-out | Withheld from a specific evaluated developer | Supports stronger inference about generalization. |
| Unseen by model | Not in training data, tuning data, prompts, or tools | The strongest contamination claim, often hard to verify. |
| Audited | Independently checked for correctness and ambiguity | Necessary for reliable all-or-nothing scoring. |
| Versioned | Problem set and scoring rules are frozen and labeled | Required for meaningful longitudinal comparison. |
FrontierMath should be reported with these distinctions. A private but developer-accessible problem set is not equivalent to a held-out blind test. A corrected benchmark version is not equivalent to the pre-correction benchmark. A vendor-reported high-compute score is not equivalent to a public hub score.
Practical interpretation for AI engineering
For AI engineers, FrontierMath is useful less as a single leaderboard and more as a diagnostic target for Self-Improving Systems.
A system that improves on FrontierMath may be improving through several different mechanisms:
| Mechanism | FrontierMath manifestation | Engineering implication |
|---|---|---|
| Better mathematical priors | Chooses relevant theorems and transformations earlier. | Improves base model training or domain adaptation. |
| Better decomposition | Breaks problems into lemmas, computations, and cases. | Improves planning and scratchpad structure. |
| Better tool use | Uses Python/SymPy to test examples and verify candidates. | Improves tool-calling, code execution, and symbolic interfaces. |
| Better search | Samples many approaches and selects consistent answers. | Improves pass@k, tree search, verifier loops, and compute allocation. |
| Better self-verification | Detects contradictions and rejects bad shortcuts. | Improves internal critique and external verifier design. |
| Better retrieval | Finds relevant mathematical facts or analogous constructions. | Improves literature and theorem-database integration. |
The benchmark alone cannot tell which mechanism caused a score increase. Raw traces, ablations, pass@k curves, tool logs, and human review are needed. This is why Epoch’s qualitative trace analysis is valuable: it shows that even successful systems may rely on brittle heuristic reasoning, hallucinated citations, or opportunistic shortcuts rather than robust proof understanding. Epoch AI
Bottom line
FrontierMath is one of the most important AI mathematics benchmarks because it attacks a real weakness in earlier evaluation: public, saturated, or shallow math tests were no longer adequate for frontier systems. Its use of newly authored expert problems, exact-answer verification, and high mathematical difficulty makes it a serious probe of advanced reasoning.
The benchmark’s weaknesses are equally serious. The problem set is small, domain-skewed, scaffold-sensitive, partially entangled with developer access, and—after the May 2026 audit—no longer safely describable as cleanly verified ground truth without qualification. Epoch AI
The most accurate assessment is therefore conditional: FrontierMath is a strong benchmark design pattern and a historically important measurement surface, but the current Tiers 1–4 leaderboard should be interpreted with versioning, audit status, compute budget, and access controls in view. Its long-term value will depend less on the original problem set than on whether the community can maintain a pipeline of audited, private, high-ceiling mathematical evaluations.
Primary references used
| Reference label | Role in this article |
|---|---|
| Glazer et al., “FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI” | Original benchmark design, construction methodology, initial under-2% model performance, discussion of limitations. arXiv |
| Epoch AI, “FrontierMath Tiers 1–3” and “Tier 4” benchmark pages | Current dataset sizes, evaluation protocol, tier definitions, and May 2026 audit warning. Epoch AI |
| Epoch AI, “Clarifying OpenAI’s involvement with FrontierMath” | Governance, OpenAI commissioning/access, and disclosure caveats. Epoch AI |
| OpenAI o3 / o3-mini / GPT-5.5 release materials | Reported high-compute performance jumps on FrontierMath and comparison with GPQA/HLE. OpenAI |
| Epoch AI leaderboard analyses | GPT-5-era performance interpretation, pass@N saturation analysis, and qualitative trace review. LinkedIn+2Epoch AI+2 |
| Humanity’s Last Exam and GPQA Diamond sources | Cross-benchmark comparison of domain breadth, answer format, and difficulty ceilings. Nature |
Companion entries
Core theory: Mathematical Reasoning in LLMs, Reasoning Benchmarks, Closed-Answer Evaluation, Benchmark Saturation, Data Contamination, AI Evaluation Philosophy
Frontier benchmark ecosystem: FrontierMath, Humanity’s Last Exam, GPQA Diamond, MATH Benchmark, AIME as an AI Benchmark, Graduate-Level AI Evaluation
AI systems and scaffolding: Reasoning Models, Test-Time Compute, Tool-Augmented Reasoning, Inference-Time Search, Verifier-Guided Decoding, Pass@k Evaluation
Mathematical AI practice: Formal Proof Assistants, Lean for AI Evaluation, Symbolic Computation, AI Mathematical Problem Solving, Mathematical Research Automation
Governance and methodology: Private Benchmarks, Held-Out Evaluation, Benchmark Versioning, Expert-Authored Datasets, Leaderboard Governance, Evaluation Auditing
Counterarguments and caveats: Goodhart’s Law in AI Evaluation, Small-N Benchmark Risk, LLM Benchmark Contamination, Expert Authorship Bias, Why Correct Answers Are Not Proofs