GPQA Diamond
Abstract. GPQA Diamond is the 198-question, highest-quality subset of GPQA, a graduate-level science benchmark designed to test whether language models can solve hard physics, chemistry, and biology questions that skilled non-experts fail even with internet access. Its importance changed quickly: in late 2023, GPT-4 was near 39%; by 2024, reasoning models such as o1 were near 78%; by 2025–2026, frontier systems clustered around 90% or above, turning GPQA Diamond from a capability frontier into a case study in Benchmark Saturation, contamination control, and the ambiguity of “reasoning” scores.
Coverage note: verified through May 11, 2026.
GPQA Diamond: Graduate-Level Science Benchmark
1. What GPQA Diamond is
GPQA Diamond is the most selective split of the Graduate-Level Google-Proof Q&A benchmark introduced by Rein et al. The full GPQA project collected hundreds of multiple-choice questions in biology, physics, and chemistry, written by domain experts with or pursuing PhDs, with the explicit goal of producing questions that are easy enough for relevant experts to answer but hard for skilled non-experts to solve even with unrestricted web access. The arXiv abstract describes GPQA as 448 multiple-choice questions; domain experts reached 65% accuracy, or 74% after discounting cases where experts judged their own mistakes clear, while highly skilled non-experts reached only 34% despite spending more than 30 minutes per question with web access. arXiv
The Diamond subset is the benchmark’s “cleanest” or highest-confidence slice: 198 questions retained only when both expert validators agreed, and when non-expert validators mostly failed. In the GPQA paper’s split table, GPQA Diamond has 198 items, reported expert accuracy of 81.3% under the split’s selection conditions, non-expert accuracy of 22.1%, and “sufficient expertise” coverage of 97.0%; those figures carry selection effects because Diamond was defined partly by expert agreement and non-expert failure. OpenReview
The phrase Google-proof matters. GPQA was not merely an exam-question benchmark. Its intended niche was Scalable Oversight: questions whose answer can be validated by experts but cannot be cheaply recovered by a non-expert searcher. Rein et al. explicitly framed the benchmark as a way to track progress toward capabilities that may outrun ordinary human supervision, and the repository includes dataset-release precautions such as a canary string, passworded release files, and guidance about prompt formats and baselines. OpenReview
2. Dataset design: how GPQA tried to be “Google-proof”
GPQA’s core design idea is simple but demanding: ask PhD-level scientists to write questions in their own subfields, then test whether other experts can answer them and whether highly educated outsiders cannot. The paper describes a four-stage collection pipeline: question writing, expert validation, revision, and non-expert validation. The contractors were recruited mainly through Upwork; they had or were pursuing PhDs in relevant domains, were asked to write difficult but answerable questions, and were paid at rates the authors report as roughly $95/hour on average, with a maximum of $150/hour. OpenReview
The validation design matters more than the multiple-choice format. Non-expert validators were PhDs or PhD candidates in other domains, and they were allowed unrestricted time and web access, but not language-model assistants. They were instructed to spend at least 15 minutes per question; the reported average was 37 minutes and the median was 30 minutes. That is why the 34% non-expert score in the paper is not a trivial “layperson baseline.” It is a benchmark for failure by smart, technically trained, internet-enabled outsiders. OpenReview
GPQA also avoided publishing worked examples from the main set in the paper because public examples would increase leakage risk. The paper includes an explicit request not to reveal examples and includes a canary string to help detect whether the paper’s content appears in training data. That is a notable early example of Contamination-Resistant Evaluation practice, although not a guarantee against contamination once the dataset is publicly downloadable and benchmarked by many labs. OpenReview
GPQA splits
| Split | Items | Inclusion rule, simplified | Human validation signal | Intended use |
|---|---|---|---|---|
| GPQA Extended | 546 | Broadest retained set after held-out exclusions | Expert accuracy 64.8%; non-expert accuracy 34.1% | Maximum coverage; noisier |
| GPQA Main | 448 | Excludes items where both experts were wrong or all non-experts succeeded | Expert accuracy 71.9%; non-expert accuracy 30.4% | Main paper benchmark |
| GPQA Diamond | 198 | Both experts answer correctly and most non-experts fail, under the paper’s selection criteria | Expert accuracy 81.3%; non-expert accuracy 22.1% | Highest-quality, hardest-to-game subset |
The asterisks and qualifications in the original table are important: “expert accuracy” in the Main and Diamond splits is conditioned on selection rules, so it is not an independent estimate of how all PhDs would perform on a fresh sample. In the authors’ wording, expert agreement can include cases where the expert later made a demonstrable mistake or showed clear understanding in their explanation. OpenReview
3. Domain coverage: graduate-level, but narrow
GPQA is a science benchmark, not a general intelligence benchmark. Its domains are biology, physics, and chemistry, with uneven subdomain distribution. In the extended set, large concentrations include organic chemistry, molecular biology, quantum mechanics, high-energy particle physics, astrophysics, and general chemistry; small subareas such as optics/acoustics, physical chemistry, analytical chemistry, and condensed matter physics are represented by only a handful of questions. OpenReview
That distribution is defensible for a first high-quality expert benchmark, but it limits interpretation. A model that does well on GPQA Diamond may have strong scientific problem-solving behavior on the benchmark distribution; it does not thereby demonstrate strong ability in clinical medicine, engineering design, experimental planning, theorem proving, bioinformatics workflows, or open-ended research. This is the first major interpretive constraint: GPQA Diamond is expert-level, but it is not broad expert-level cognition.
The second constraint is that the questions are closed-ended. Four-answer multiple choice is useful for reproducible automated evaluation, but it compresses the task. A model need not produce a derivation, choose experiments, inspect raw data, or defend an answer under adversarial questioning. Random guessing has a 25% baseline, and answer-option artifacts can matter. GPQA’s strength is that it made the answer choices hard enough that non-expert searchers still failed; its weakness is that final performance remains a single closed-ended accuracy number. OpenReview
4. Why Diamond became the canonical split
Diamond became the public shorthand for GPQA because it is the most filtered split. In benchmark practice, “hard but noisy” is less useful than “hard and defensible.” The Diamond subset is smaller, but its items have stronger evidence that the answer key is correct and that the question is genuinely discriminative between domain experts and outsiders. OpenReview
The tradeoff is statistical power. With only 198 items, one question is roughly 0.5 percentage points of accuracy. A 1-point difference between two models is about two questions; a 3-point difference is about six questions. At an observed accuracy of 94%, the binomial standard error is roughly 1.7 percentage points, so a crude 95% interval is about ±3.3 percentage points, before accounting for correlated item errors, prompt variance, answer-order effects, or harness differences. At 90%, the crude interval is about ±4.2 percentage points. This means GPQA Diamond can clearly separate 40%, 78%, and 94% systems, but it is a weak instrument for ranking systems separated by less than a few points.
This matters because 2025–2026 frontier reports often cluster near the low-to-mid 90s. A reported 94.3% versus 93.6% may look meaningful on a leaderboard, but on a 198-item benchmark it can amount to roughly one or two answer differences unless averaged across repeated trials under identical conditions. The saturation question is therefore partly empirical and partly statistical: the benchmark can still catch regressions, but it no longer has much headroom as a fine-grained frontier discriminator.
5. Empirical trajectory: from GPT-4 to reasoning models to 90%+
The original GPQA paper captured a short but striking acceleration. At release in November 2023, the best GPT-4 baseline was about 39%; by March 2024, Claude 3 Opus was reported around 60% under stronger sampling/self-consistency settings. The OpenReview version of the GPQA paper explicitly notes this early jump from GPT-4’s 39% to Claude Opus at roughly 60% within months. OpenReview
OpenAI’s o1 report marked the next major step. OpenAI described o1 as trained with large-scale reinforcement learning to reason using chain of thought, with performance improving as train-time and test-time compute increased. In the GPQA Diamond appendix table, GPT-4o scored 50.6% pass@1 and 56.1% consensus@64; o1 scored 77.3% pass@1 and 78.0% consensus@64, while o1-preview scored 73.3% pass@1 and 78.3% consensus@64. OpenAI
OpenAI’s o3 period is harder to summarize as a single clean score. Public reporting around OpenAI’s December 2024 o3 announcement cited roughly 87.7% on GPQA Diamond, which is the “~88%” figure often repeated in discussions of o3. Later official OpenAI comparison tables for released o-series and open-weight models list o3 at 83.3% on GPQA Diamond without tools, alongside o4-mini at 81.4% and gpt-oss-120b at 80.1%; this is a useful reminder that reported GPQA numbers depend on release status, harness, tool policy, reasoning effort, and aggregation method. Ars Technica
By the GPT-5 generation, the frontier was near 90% even under no-tool settings. OpenAI’s GPT-5 page reports GPT-5 Pro at 88.4% on GPQA without tools and says GPT-5 outperformed o3 with substantially fewer output tokens on capabilities including graduate-level scientific problem solving. OpenAI
By 2026, official benchmark tables had GPQA Diamond clustered in the 90s. OpenAI’s GPT-5.5 page reports GPT-5.5 at 93.6%, GPT-5.4 at 92.8%, GPT-5.4 Pro at 94.4%, Claude Opus 4.7 at 94.2%, and Gemini 3.1 Pro at 94.3% in its academic comparison table. Google’s Gemini 3.1 Pro model card independently reports Gemini 3.1 Pro Thinking High at 94.3% and Gemini 3 Pro Thinking High at 91.9% on GPQA Diamond, with results listed as of February 2026. OpenAI
Trajectory table
| Period | Model or model family | Reported GPQA Diamond performance | Evaluation caveat |
|---|---|---|---|
| Late 2023 / original GPQA paper | GPT-4 baselines | About 39%; GPT-4-0613 few-shot CoT around 38.8% on Diamond | Original paper baselines; prompt and sampling choices matter. arXiv |
| Early 2024 | GPT-4-0125-preview and Claude 3 Opus | GPT-4-0125-preview around low-to-mid 40s; Claude 3 Opus up to mid/high 50s with self-consistency | Same benchmark, but stronger models and aggregation settings. OpenReview |
| Sept.–Dec. 2024 | GPT-4o, o1-preview, o1 | GPT-4o 50.6% pass@1 / 56.1% cons@64; o1 77.3% pass@1 / 78.0% cons@64 | OpenAI evaluation; o1 framed as reasoning model with test-time compute. OpenAI |
| Dec. 2024–2025 | o3 / o4-mini | o3 widely reported around 87.7%; later official no-tool comparison lists o3 at 83.3%, o4-mini at 81.4% | Not one apples-to-apples number; release and harness details differ. Ars Technica |
| 2025 | GPT-5 Pro | 88.4% without tools | Official OpenAI GPT-5 benchmark table. OpenAI |
| 2026 | GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.7 comparisons | Low-to-mid 90s; OpenAI reports GPT-5.5 93.6%, Gemini 3.1 Pro 94.3%, Claude Opus 4.7 94.2%; Google reports Gemini 3.1 Pro 94.3% | At this level, small-N uncertainty and harness differences dominate fine ranking. OpenAI |
6. Evaluation conventions that change the number
A GPQA Diamond score is not fully interpretable without the evaluation recipe. At least six choices matter.
First, pass@1 versus consensus or majority voting changes the result. The GPQA paper reports multiple prompting and self-consistency variants, and OpenAI’s o1 table distinguishes pass@1 from consensus@64. Consensus can improve scores by sampling many reasoning traces and selecting the most frequent answer; that is a legitimate capability measurement if one cares about test-time compute, but it is not the same as a single-shot answer. OpenReview
Second, reasoning effort changes results for reasoning models. OpenAI’s o-series system card describes o3 and o4-mini as models with tool-enabled reasoning and chains of thought; the o1 report frames the model family as improving with more train-time and test-time compute. Once “thinking longer” becomes part of the model interface, benchmark scores measure a policy plus an inference budget, not a static model in the older sense. OpenAI
Third, tool access can make a score non-comparable. GPQA was originally designed so web access would not let non-experts solve the questions, and GPT-4 search baselines in the paper did not trivially solve the benchmark. But modern tool-using models can search, calculate, run Python, inspect files, and use intermediate computations inside their reasoning process. OpenAI’s o3/o4-mini system card explicitly says those models can use tools such as web browsing and Python during their thought process. OpenReview
Fourth, answer randomization and option formatting matter. A multiple-choice benchmark can be subtly sensitive to whether answer labels are permuted, whether explanations are required, whether the answer is extracted by regex or judged by another model, and whether the prompt includes examples. GPQA’s repository documents several prompt variants and retrieval baselines, underscoring that the benchmark is not just a file of questions but an evaluation protocol. GitHub
Fifth, public versus private exposure matters. GPQA took contamination seriously at release, but it is a public dataset. Once a benchmark becomes canonical, models may encounter it through training data, benchmark discussions, solution traces, or evaluation-tuned reinforcement learning. That does not imply any specific reported GPQA score is contaminated; it means the burden of proof rises as scores approach the ceiling.
Sixth, domain-level reporting matters. The o1 table shows much stronger performance in some GPQA subdomains than others; for example, OpenAI’s appendix reports o1 physics pass@1 above 90%, but chemistry and biology substantially lower. A single averaged Diamond score can hide whether the model improved broadly or mostly learned particular scientific distributions. OpenAI
7. Methodological strengths
GPQA Diamond’s first strength is expert authorship. Many benchmarks call themselves “hard” because models score poorly; GPQA’s difficulty was constructed through domain expertise and validated by expert/non-expert contrast. That is a stronger design than simply mining obscure trivia or repackaging exam questions.
Its second strength is adversarial non-expert validation. The non-expert validators were not average crowdworkers. They were highly educated people in adjacent or unrelated domains, using the internet for substantial time. This is exactly the kind of baseline needed for a benchmark that claims “Google-proof” status. OpenReview
Its third strength is clear automated scoring. Multiple-choice answers are crude, but they make repeated model comparison cheap and reproducible. This made GPQA Diamond useful during the rapid o1/o3/GPT-5 era because labs could evaluate many model variants under many inference settings.
Its fourth strength is contamination awareness. The canary string, the decision not to disclose examples from the core benchmark, and the explicit repository guidance were good practices for the 2023–2024 period. They did not solve public-benchmark contamination, but they made the problem visible rather than accidental. OpenReview
Its fifth strength is alignment with scalable oversight concerns. GPQA is not just a “science quiz.” It probes a regime where ordinary evaluators cannot easily tell whether an answer is correct without expert infrastructure. That makes it relevant to AI Evaluation, Scalable Oversight, and Elicitation vs Capability: the benchmark asks whether a model can solve problems whose answers are outside the easy supervision range of most users.
8. Methodological weaknesses
The most obvious weakness is small N. Diamond has 198 questions. That was the cost of quality filtering, but it makes frontier ranking fragile. Once top systems are above 90%, the remaining differences are often only a few items.
The second weakness is narrow domain coverage. GPQA Diamond is not “graduate school” in general. It is mainly hard natural-science Q&A in selected areas of physics, chemistry, and biology. It says little directly about proof search, engineering design, economics, medicine, law, fieldwork, or multi-month research.
The third weakness is closed-endedness. Multiple-choice scientific reasoning is not the same as scientific research. GPQA does not require a model to design an experiment, collect evidence, write code, run simulations, update beliefs over a literature corpus, or produce a paper-quality argument. Benchmarks such as SciCode and BixBench were partly motivated by this gap: SciCode evaluates code generation for real scientific research problems across natural-science subfields, while BixBench evaluates agents on real-world bioinformatics scenarios involving multi-step data analysis and interpretation. arXiv
The fourth weakness is public static exposure. GPQA was contamination-resistant in design, but not contamination-proof in deployment. Public datasets are eventually learned by the ecosystem, even if not deliberately inserted into pretraining data. This is especially important for systems trained with benchmark-aware post-training or reinforcement learning, where “scientific multiple-choice problem solving” can become an optimized behavioral niche.
The fifth weakness is expert validation does not imply perfect labels. The original expert accuracies were far below 100%, and the paper discusses objectivity, demonstrable mistakes, and explanation-based judgments. That does not invalidate the dataset; it means some questions are extremely specialized, some are hard even for experts, and the validation process itself is part of the evidence. OpenReview
The sixth weakness is measurement ambiguity under test-time compute. Reasoning models blur the boundary between model capability and inference procedure. If a system spends many samples, uses self-consistency, calls tools, or searches the web, the GPQA score measures an engineered solving loop. That may be exactly what users care about, but it should not be compared casually with older single-pass language-model scores.
9. The saturation question
GPQA Diamond is not useless because frontier models score above 90%. Saturated benchmarks can still detect regressions, evaluate smaller models, compare tool-free versus tool-using settings, and provide continuity across model generations. MMLU remains informative for broad sanity checks even though it stopped being a frontier discriminator; GPQA Diamond may now occupy a similar role for graduate-level science.
But as a frontier benchmark, Diamond is close to saturated. The issue is not merely that top scores are high. It is that the benchmark has limited remaining resolution. A public 198-item multiple-choice set cannot reliably distinguish a 93.6% system from a 94.3% system unless the evaluation is repeated, controlled, and statistically interpreted. At that point, leaderboard ordering risks becoming a proxy for harness choice.
This is the same pattern that drove the field beyond MMLU. MMLU originally tested multitask academic and professional knowledge across 57 subjects and required broad world knowledge and problem solving; by 2025, newer benchmarks explicitly described MMLU as above 90% for frontier models and no longer difficult enough for fine-grained frontier tracking. arXiv
The successor pattern is visible in MMLU-Pro, Humanity’s Last Exam, FrontierMath, SciCode, BixBench, SWE-bench, and ARC-AGI. MMLU-Pro expanded MMLU toward more reasoning-focused questions, ten answer options, and lower prompt sensitivity; HLE assembled thousands of expert-written, multimodal, closed-ended questions across many subjects; FrontierMath uses unpublished advanced math problems designed to take experts hours or days; SciCode converts scientific research problems into coding tasks; BixBench targets multi-step bioinformatics analysis; SWE-bench evaluates real software issue resolution; ARC-AGI tries to isolate fluid generalization rather than accumulated domain knowledge. ARC Prize+6arXiv+6arXiv+6
The direction is clear: as models saturate static academic Q&A, frontier evaluation moves toward fresh tasks, private held-out sets, tool-use workflows, code execution, long-horizon tasks, open-answer grading, and expert audits.
10. Reasoning capability or expert-domain pattern matching?
The central interpretive question is whether GPQA Diamond measures reasoning capability or expert-domain pattern matching. The honest answer is: both are plausible contributors, and GPQA alone cannot separate them.
The case for “reasoning” is strong enough to take seriously. GPQA questions were written so that non-experts with search would fail, and early retrieval/search baselines did not make GPT-4 suddenly solve the benchmark. The largest observed jumps came with systems explicitly trained and deployed as reasoning models, especially o1-style models whose performance improved with test-time computation and chain-of-thought-like deliberation. OpenReview
The domain-specific pattern-matching case is also strong. A model trained on huge scientific corpora can learn technical priors, formulaic reasoning templates, common distractor patterns, and answer-choice cues without possessing robust scientific agency. Multiple-choice questions make that easier than open derivations. A static public benchmark also creates a long-term risk that models, prompts, or post-training curricula become tailored to the benchmark’s distribution.
A useful formulation is: GPQA Diamond measures elicited scientific problem-solving performance on a curated closed-ended distribution. It is not a pure measure of abstract reasoning, and it is not mere recall. It sits between those poles. Its design filters out simple web retrieval and generic exam cramming better than most earlier benchmarks, but its format still allows models to exploit learned scientific patterns and benchmark-specific regularities.
This distinction matters for Self-Improving Systems. If a model improves from 40% to 90% on GPQA Diamond, that is evidence of a major capability shift in expert-style scientific Q&A. It is not, by itself, evidence that the model can autonomously generate new science, verify its own hypotheses, build instruments, or manage long research programs. To evaluate those claims, GPQA must be combined with benchmarks and audits that test multi-step research workflows, code execution, literature-grounded reasoning, experimental design, and expert review.
11. How to read a GPQA Diamond score
A good GPQA Diamond report should answer the following questions in the main text, not bury them in a footnote:
| Question | Why it matters |
|---|---|
| Was the score on Diamond, Main, or Extended? | These are different distributions with different quality filters. |
| Was the model evaluated with tools or without tools? | Tool use changes the task and can add search, computation, or file analysis. |
| Is the metric pass@1, majority vote, consensus@k, or another aggregation? | Sampling many attempts can raise scores and measures test-time compute. |
| How many trials were averaged? | At 198 items, single-run variance is nontrivial. |
| Were answer choices randomized? | Multiple-choice artifacts can bias results. |
| Was the benchmark public, private, or canary-checked relative to training data? | Static public benchmarks face contamination and overfitting risk. |
| Are subdomain results reported? | A single score can hide uneven biology, chemistry, and physics performance. |
| Is the result self-reported or independently reproduced? | Frontier leaderboard differences can be smaller than harness variance. |
The most defensible interpretation is comparative and coarse-grained. GPT-4-era 39%, o1-era 78%, and 2026-era 93–94% are meaningfully different regimes. A one-point difference between two 2026 systems is not a meaningful standalone claim unless the setup is identical and the uncertainty is reported.
12. Relation to broader benchmark evolution
GPQA Diamond belongs to a broader transition from general-knowledge benchmarks to expert-reasoning and work-simulation benchmarks.
MMLU represented the earlier broad-knowledge paradigm: 57 tasks spanning elementary mathematics, U.S. history, computer science, law, and other subjects, intended to test world knowledge and problem solving across many academic and professional domains. It became a standard because it was broad and easy to run, but that same broad, exam-like structure made it vulnerable to saturation as models improved. arXiv
MMLU-Pro is a direct response: it adds harder, more reasoning-focused questions, expands from four to ten choices, removes some trivial or noisy MMLU items, and reports that model accuracy drops 16–33 percentage points relative to MMLU while prompt sensitivity falls. arXiv
Humanity’s Last Exam generalizes the GPQA instinct across a wider academic frontier. It is multimodal, covers thousands of questions across many subjects, uses expert contributors, and explicitly positions itself against the saturation of popular benchmarks such as MMLU. Its own discussion warns that high HLE accuracy would show expert-level performance on closed-ended verifiable questions, not autonomous research capability or AGI. arXiv
FrontierMath pushes in another direction: hidden, original, expert-vetted mathematics problems that can require hours or days from specialists and use automated verification. Its authors explicitly motivate the benchmark by the saturation of earlier math benchmarks and by contamination concerns in training data. arXiv
SciCode and BixBench move from “answer the expert question” toward “perform pieces of expert work.” SciCode asks models to write code for scientific research problems and decomposes 80 main problems into 338 subproblems involving knowledge recall, reasoning, and code synthesis. BixBench evaluates agents on more than 50 real-world bioinformatics scenarios and nearly 300 open-answer questions requiring data exploration, multi-step analysis, and interpretation. arXiv
ARC-AGI is philosophically different. It explicitly tries to avoid crystallized knowledge and instead test skill-acquisition efficiency, generalization, and fluid reasoning on novel tasks. That makes it a useful counterpoint to GPQA: GPQA asks whether a model can use deep learned scientific expertise; ARC asks whether a system can generalize from minimal priors. ARC Prize
The pattern is not that one benchmark replaces all others. The pattern is that different benchmarks probe different failure modes: knowledge breadth, expert-domain depth, hidden-problem reasoning, tool-using agency, software maintenance, scientific coding, data analysis, and fluid abstraction. GPQA Diamond’s niche is still valuable, but it is now one instrument in a larger evaluation battery.
13. Bottom line
GPQA Diamond was a well-designed benchmark for a specific moment: the moment when frontier language models were beginning to threaten expert-level closed-ended science Q&A, but had not yet crossed that line. Its design choices—PhD-written questions, expert validation, non-expert web-access validation, and contamination awareness—made it one of the strongest public benchmarks for graduate-level scientific reasoning in 2023–2024. arXiv
By 2026, the benchmark’s role had changed. Scores in the 90s do not make GPQA Diamond irrelevant, but they do make it a poor fine-grained frontier leaderboard. The right reading is historical and diagnostic: GPQA Diamond documented a real capability transition from GPT-4-level systems to reasoning-model systems, and now exposes the limits of static, small-N, multiple-choice expert benchmarks.
The unresolved scientific question is not “which model is 0.7 points higher on GPQA Diamond?” It is whether the behaviors that GPQA Diamond rewards are the same behaviors needed for robust scientific reasoning outside the benchmark: choosing the right abstraction, deriving under uncertainty, using tools correctly, noticing when the problem framing is wrong, and carrying a research process across many steps. GPQA Diamond gives evidence about that question, but not an answer.
Selected references
Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” Primary source for dataset design, expert/non-expert validation, split definitions, and early GPT-4/Claude baselines. arXiv
GPQA GitHub repository. Primary source for dataset release, canary string, prompt formats, and implementation notes. GitHub
OpenAI, “Learning to reason with LLMs.” Primary source for o1 GPQA Diamond pass@1 and consensus scores, plus the framing of test-time reasoning. OpenAI
OpenAI o3/o4-mini system card and model comparisons. Primary source for o-series tool-enabled reasoning setup and later official GPQA Diamond comparison numbers. OpenAI
OpenAI GPT-5 and GPT-5.5 benchmark reports. Primary sources for GPT-5 Pro and GPT-5.5-era GPQA Diamond figures. OpenAI
Google Gemini 3.1 Pro model card. Primary source for Gemini 3.1 Pro and Gemini 3 Pro GPQA Diamond results. Google DeepMind
MMLU, MMLU-Pro, HLE, FrontierMath, SciCode, BixBench, SWE-bench, and ARC-AGI sources. Context for the broader shift from broad knowledge exams to expert, fresh, agentic, and fluid-reasoning evaluations. ARC Prize+7arXiv+7arXiv+7
Companion entries
Core theory: Scalable Oversight, Elicitation vs Capability, Reasoning Models, Test-Time Compute, Benchmark Saturation, Contamination-Resistant Evaluation
Benchmarks: GPQA, MMLU, MMLU-Pro, Humanity’s Last Exam, FrontierMath, SciCode, BixBench, SWE-bench, ARC-AGI
Evaluation practice: Evaluation Harness Design, Multiple-Choice Evaluation Artifacts, Confidence Intervals for Benchmarks, Tool-Use Evaluation, Self-Consistency Decoding, Expert Validation
Counterarguments: Pattern Matching vs Reasoning, Benchmark Overfitting, Static Benchmark Contamination, Closed-Ended Questions vs Scientific Research, Leaderboard Epistemology