Capability Evaluation: Distinct from Safety, Reliability, and Behavior Evaluation

Capability evaluation tries to estimate what a model or model-system can do near its frontier under a defined elicitation envelope. It is operationally distinct from safety, behavior, and reliability evaluation, but the distinction is an accounting boundary rather than an observation-free natural kind: every reported capability is mediated by prompting, scaffolding, tool access, sampling budget, refusal policy, and scorer design. The distinction is governance-relevant — it underwrites the threshold systems used by the major frontier labs — but present capability evaluation is not yet an engineering audit, and treating threshold classifications as audit-grade measurements is the most consequential failure mode in the literature.

Coverage note: verified through May 2026.

The four constructs

The literature on frontier-model evaluation routinely names four overlapping but distinct evaluation targets. Keeping them apart matters for governance because the deployment decisions they support are different.

Capability evaluation asks: under a serious attempt to elicit performance, what can this system do? It estimates an upper envelope of task performance for a specified threat model or skill domain. The unit of analysis is the system's elicitable competence on a task family — solving an exploit chain, answering graduate-level chemistry, abstracting a novel pattern from few examples, executing a multi-step research plan with tools.

Safety evaluation asks: does the system cause or enable harm under realistic operating conditions? It is downstream of capability and conditional on it. A model that lacks a capability cannot enable the corresponding harm; a model that has the capability may or may not, depending on safeguards, refusal policies, monitoring, deployment context, and adversary effort. Safety Cases for AI Systems formalize this conditional structure.

Behavior evaluation asks: what does the system tend to do, refuse to do, or comply with under instructions, system prompts, and policies? It is the domain of refusal-rate measurements, sycophancy probes, instruction-following benchmarks, and propensity-to-deceive tests. Behavior is a function of post-training, system prompts, and inference-time guardrails layered on top of capability.

Reliability evaluation asks: how consistent is the system's performance across repetitions, sessions, prompt variants, and operating conditions? It is the engineering-audit cousin of capability evaluation: it cares less about the upper envelope and more about the floor and the variance. A model that can solve a task one percent of the time is a different deployment object than one that solves it ninety-five percent of the time, even if the capability claim "the model can do it" is true of both. See Model Reliability Evaluation.

The canonical disaggregation is Shevlane et al. 2023, "Model evaluation for extreme risks", which separates evaluations of dangerous capabilities from evaluations of a model's propensity to apply those capabilities harmfully. That paper has shaped the structure of every frontier safety framework now in production. The capability/propensity split is the source of the modern four-construct vocabulary, with reliability and behavior evaluation evolving as separate concerns under deployment-engineering pressure.

The reason the distinction matters at the decision layer is that the same model can present very differently across the four lenses: highly capable but behaviorally constrained, capable and reliable but unsafe under adversarial elicitation, behaviorally aligned but unreliable, or low-capability and therefore not yet a safety question. Collapsing the four into a single "safety score" produces incoherent governance. A model and a deployed product behave very differently under each lens, and the four constructs supply the vocabulary for that difference.

Why the boundary is porous

The conceptual distinction is clean. The measurement is not.

Capability is never observed directly. It is inferred from task performance under a harness, and the harness is part of the measurement. A capability score is a joint product of:

• Latent model competence
• Prompt and prompt-format choice
• Scaffolding (chain-of-thought, retries, self-consistency, agent loops)
• Tool access (search, code execution, file system, APIs)
• Inference-time compute budget
• Refusal policy and post-training safeguards
• Scoring method and grader competence
• Benchmark version and contamination state
• Evaluator skill and prompt-engineering effort

Each of these is a free variable. Hold them constant and you get reproducibility; vary them in plausible ways and you get capability estimates that move by orders of magnitude on the same model.

This is not a hypothetical concern. Sclar et al. (ICLR 2024) demonstrated that meaning-preserving prompt-format perturbations — changing separators, capitalization, whitespace — can shift LLM accuracy by tens of percentage points on standard benchmarks. Mizrahi et al., "When Benchmarks are Targets" found that leaderboard rankings reorder under prompt perturbations that any reasonable user might apply. Polo et al., "PromptEval" argued that benchmark results should be reported as distributions over prompt variants rather than single numbers; on MMLU, the gap between worst-case and best-case prompts often exceeds the gap between models. Brittlebench (Cao et al. 2026) extended the same finding to semantics-preserving perturbations, showing performance degradation and rank instability under transformations users would not even notice. See Elicitation Sensitivity in Language Models for the broader pattern.

The asymmetric versions of this problem matter most for governance. Under-elicitation false negatives make a model appear below threshold under weak prompting; later users discover the capability was always there. Anthropic's Sabotage Evaluations for Frontier Models names this directly: under-elicitation during pre-deployment evaluation can produce a model that later appears more dangerous than the evaluation indicated. Over-elicitation artifacts run the other way: extreme scaffolding, expert-written prompts, or evaluator-in-the-loop iteration can produce results that reflect the elicitation system's capability rather than the model's. The distinction matters for deployment policy: a base-model capability transfers to anyone with API access; a capability that requires a specific agent scaffold transfers only to those who can build the scaffold. Agent Scaffolding and Capability Elicitation develops this further.

The "password-locked model" literature (Greenblatt et al., NeurIPS 2024; Hofstatter et al., 2025) stretches this further. They construct settings where a model demonstrably has a capability, but the capability is hidden behind a behavioral block — the model only exhibits the skill when prompted in a specific way. The fact that such constructions exist is a proof of concept that "the model cannot do X" is rarely a stable fact about the weights. It is usually a fact about the weights conjoined with whatever elicitation was tried.

These observations do not dissolve the capability/safety distinction. They specify what the distinction is. Capability evaluation measures upper-envelope task performance under a specified elicitation envelope. The envelope is part of the claim. A capability claim without its envelope is not a measurement; it is a vibe.

Canonical capability evaluations

The phrase "capability evaluation" in current usage refers to several different families of evidence, with different constructs, different reproducibility profiles, and different governance relevance. Treating them as one ladder is a category error. Public benchmarks for abstract reasoning, expert math, and graduate-level science are not interchangeable with dangerous-capability evaluations for cyber, CBRN, autonomy, or AI R&D risk. They are adjacent evidence classes, not members of a single measurement family.

General frontier reasoning benchmarks

These are public, peer-reviewed benchmarks designed to probe a specific reasoning construct. They are not designed as deployment-risk assessments, but they appear in system cards and policy debates as proxies for general capability.

Benchmark	Construct	Format	Notes
ARC-AGI	Few-shot abstraction and skill acquisition on novel grid puzzles	Hidden test set, curated tasks	Resistant to memorization by design; ARC Prize 2025 emphasized refinement loops and contamination controls
GPQA	Graduate-level, Google-resistant science QA across biology, chemistry, physics	Multiple-choice with expert-validated questions	Designed to be hard for non-experts even with web search; intended as expert-knowledge probe, not deployment risk test
FrontierMath	Expert-level mathematical reasoning across research-grade problems	Hidden problem set, expert-validated solutions	Methodology notes that scores can vary by internal scaffold, test-time compute, and subset; private holdout reduces but does not eliminate contamination concerns
MMLU / MMLU-Pro	Broad multi-domain knowledge	Multiple-choice across 57+ subjects	Saturated at the frontier; useful as a baseline rather than a frontier signal
HumanEval, SWE-bench, OSWorld	Code generation and agentic software tasks	Programmatic scoring	OSWorld and SWE-bench Verified expose substantial reliability gaps even at frontier capability

The 2026 AI Index reports that capability is outpacing benchmark design, that evaluations saturate in months, and that agent systems still fail roughly one in three attempts on structured OSWorld-style tasks. The empirical prior should be that a single impressive score is real evidence of possible frontier performance, but weak evidence of routine reliability or governance sufficiency.

Dangerous capability evaluations

These are evaluations specifically designed to probe whether a model materially enables harmful pathways: cyber operations, chemical/biological/radiological/nuclear (CBRN) uplift, autonomous replication and resource acquisition, large-scale persuasion or manipulation, and AI research-and-development uplift. They are the evidence base behind threshold frameworks. See Dangerous Capability Evaluation.

Phuong et al., "Evaluating Frontier Models for Dangerous Capabilities" is the closest thing to a canonical public protocol. It defines a program across persuasion/deception, cyber, self-proliferation, self-reasoning/self-modification, and biological/nuclear risk, and reports that the evaluated models showed early warning signs rather than strong dangerous capability. The paper is also a case study in the difficulty of public reproducibility: many of the most diagnostic tasks cannot be published in full without dual-use leakage.

WMDP (Weapons of Mass Destruction Proxy) is a public benchmark for hazardous biology, chemistry, and cybersecurity knowledge. It is intentionally structured as a proxy: it does not contain operational uplift content, only knowledge probes correlated with it. The proxy structure is the right design for a public benchmark, but it also means a high WMDP score is not a direct claim about real-world uplift.

Lab-internal dangerous-capability evaluations — Anthropic's bio uplift studies, OpenAI's CBRN evaluations, DeepMind's autonomy and self-proliferation evaluations — extend further but are not fully public. Their results appear in system cards and framework documents, but the underlying tasks, scaffolds, and scoring rubrics are typically withheld for dual-use reasons. This produces a structural transparency tension explored further below.

Lab-internal frontier eval suites

Each major lab maintains an internal suite that combines public benchmarks, dangerous-capability tasks, agentic tasks, and bespoke probes designed against the lab's threat model. These are the suites that actually feed threshold decisions. Their public traces are partial: system cards report selected results, framework documents describe categories, and engineering blogs occasionally publish methodology. Independent reproducibility is limited.

The structural fact worth naming: the suites that drive governance decisions are the least public, and the suites that are most public are the least decision-relevant. ARC-AGI scores are reproducible but do not directly determine deployment policy. CBRN uplift evaluations directly determine deployment policy but are not reproducible by outsiders.

Frontier safety frameworks as threshold systems

Each of the three frontier labs has published a framework that maps capability evidence to action. They differ in vocabulary and structure but share an architecture: define tracked capability domains, define threshold levels within each domain, commit to safeguards triggered by threshold crossing, and report selected results publicly. They are all explicit about being living frameworks rather than fixed standards.

Framework	Current version	Threshold structure	What triggers what
Anthropic Responsible Scaling Policy	v3.2, effective April 29, 2026	AI Safety Levels (ASL-N) tied to Capability Reports and Risk Reports	Crossing an ASL threshold triggers required deployment and security safeguards plus industry-wide recommendations; Risk Reports synthesize capability + threat model + mitigation
OpenAI Preparedness Framework	v2, last updated April 15, 2025	Tracked categories with High and Critical capability thresholds	Capability Reports measure pre-deployment capability; Safeguards Reports document mitigations; Critical thresholds trigger restricted deployment until safeguards meet bar
Google DeepMind Frontier Safety Framework	3.1, updated April 17, 2026	Critical Capability Levels (CCLs) and Tracked Capability Levels (TCLs)	TCLs added in 3.1 for earlier detection; CCL crossing triggers mitigations including security uplift and deployment restrictions

See Responsible Scaling Policy, Preparedness Framework, and Frontier Safety Framework for fuller treatments of each.

The frameworks share a critical feature that is easy to miss: they are evidence-to-action mappings, not measurement standards. They do not define how to measure a capability; they define what to do once a measurement crosses a threshold. The measurement is delegated to the lab's internal evaluation program, with the threshold acting as a tripwire.

This delegation is where the governance pipeline carries forward measurement uncertainty. A threshold crossing under one elicitation regime might not cross under another. The frameworks acknowledge this in different ways. Anthropic's RSP explicitly notes that capability evidence interacts with threat models and mitigations in the Risk Report, which goes beyond raw capability evidence to include the safety case. OpenAI's PF distinguishes Capability Reports from Safeguards Reports, allowing the safeguards layer to compensate for capability uncertainty. DeepMind's FSF 3.1 added Tracked Capability Levels specifically to provide earlier detection — an admission that waiting for Critical Capability Level evidence is too brittle a tripwire.

The frameworks are useful forcing functions even where their measurements are uncertain. They require labs to predefine threat models, capability categories, threshold levels, safeguard obligations, and escalation processes. They make commitments public, which creates external accountability even when the evaluations themselves are not externally reproducible. They exist because organizations need decision rules under uncertainty, and "wait for measurement to mature" is not a viable answer when capability is advancing on a months-long doubling cycle.

But the frameworks are not, on their own, evidence that threshold classifications are stable scientific facts. They are governance artifacts produced by an evaluation program, a threat model, and a safety case. A "High" or "Critical" classification is a decision under uncertainty, not a calibrated instrument reading. Reading the framework documents this way matters because the public discourse routinely flattens them into "labs have measured this model as below the dangerous threshold," which is a stronger claim than the framework documents themselves make.

The reproducibility critique

The strongest practical critique of capability evaluation as currently practiced is that results are non-reproducible in a strong sense. Three failure modes recur.

Elicitation sensitivity. Capability scores depend on prompt, scaffold, tool access, and inference budget. The literature cited above — Sclar et al., Mizrahi et al., Polo et al., Brittlebench — shows this is not an artifact of bad benchmark design but a property of how transformer models respond to inputs. Two evaluators running the same benchmark on the same model with different prompt templates can produce results that meaningfully disagree.

Contamination and saturation. Public benchmarks leak. ARC-AGI's hidden test set and FrontierMath's private holdout are designed against this, but most benchmarks were trained on by post-cutoff models inadvertently or by data-curation processes that included benchmark-adjacent text. Saturation compounds the problem: when scores cluster near the ceiling, small differences in test set, scoring rule, or prompt variant produce large differences in apparent ranking. The 2026 AI Index documents both effects. See Benchmark Contamination.

Opacity of dangerous-capability protocols. The evaluations that matter most for governance — CBRN uplift, autonomous replication, large-scale cyber operations — cannot be published in full. Their tasks, scoring rubrics, and high-performing scaffolds are dual-use information. This produces a structural transparency tension: the public evidence is reproducible but not decision-grade, and the decision-grade evidence is held by the developer making the deployment decision.

The naive fix — publish everything — is unsafe for the dangerous-capability case. A more defensible approach is tiered transparency (Tiered Transparency for Dual-Use Evaluations):

• Public construct definitions and threat models
• Public model and version metadata
• Public scoring methodology at the level needed to interpret results
• Public uncertainty reporting (intervals, distributions, sensitivity analyses)
• Independent access under controlled conditions (red-teaming partnerships, government access, structured safety case review)
• Withheld operational details where dual-use leakage is the controlling concern

The other naive fix — standardize elicitation across labs and evaluators — reduces variance but creates a new optimization target and may miss capabilities that emerge only under novel scaffolds. Standardization is the right move for comparability protocols (running the same protocol across models to track change over time). It is the wrong move for stress-test protocols intended to find the frontier of what a model can do, where the explicit goal is adversarial creativity in elicitation.

The cheapest experiment that would actually shift the reproducibility debate is a preregistered multi-elicitation replication packet in a domain where scoring is relatively objective. Cyber capture-the-flag tasks and advanced math are the strongest candidates. The packet would specify: a held-out task set, multiple elicitation regimes (baseline prompt, optimized prompt, tool scaffold, agent scaffold, higher sampling budget, expert human-in-the-loop), public prompts and scaffold code, scoring code, raw transcripts, confidence intervals, contamination analysis, and rank sensitivity. The output is not a single score; it is the distribution of elicited capability and the gap between median, upper-quartile, and best-known elicitation. Independent reruns by competent third parties would test whether the distribution is stable. See Multi-Elicitation Replication Packets.

This is the engineering-audit aspiration. Nothing currently published meets it across all conditions. Some components are in place — held-out test sets, public scoring code for some benchmarks, system-card transparency — but the multi-elicitation distributional reporting and independent-rerun infrastructure are not yet routine.

Where the distinction breaks down

The capability/safety/behavior/reliability boundary is operational, not metaphysical. There are cases where the boundary is genuinely contested.

Capability vs. behavior under refusal training. A model that refuses to help with a CBRN task may lack the capability or may possess the capability with a behavioral block. Distinguishing these cases requires either jailbreaking the model (which probes whether the block is robust) or fine-tuning to remove the safeguard (which probes the underlying weights). Both move the elicitation envelope. The framework documents handle this by treating refusal robustness as a separate evaluation rather than as evidence about underlying capability — but the operational decision of whether to deploy depends on both.

Model capability vs. system capability. A base model with no tools, no scaffold, and no system prompt may fail a task that the same model passes when wrapped in an agent loop with web search, code execution, and a planner. Is the capability "in" the model? It depends on the threat model. For deployment governance, what matters is what the deployed system enables; for understanding the underlying model, what matters is what the weights afford. Frontier safety frameworks operationalize this by evaluating both the base model and the deployed product surface, but the distinction is not always preserved in public reporting.

Capability vs. fine-tuning capability. Open-weights models can be fine-tuned to remove safeguards or add capabilities. A model that lacks a dangerous capability under standard inference may acquire it under fine-tuning by an adversary with modest resources. Whether this counts as "the model's capability" for governance purposes depends on the threat model and the accessibility of fine-tuning. The frameworks vary in how they handle this; Anthropic's RSP and DeepMind's FSF treat fine-tuning robustness as part of the evaluation envelope, while OpenAI's PF treats it more as a deployment-context question.

Capability vs. reliability for agentic systems. An agent that can complete a complex task once in a hundred attempts has the capability in some meaningful sense, but the reliability is so low that the deployment risk profile is qualitatively different from a 95% reliable agent. The literature has not fully resolved how to integrate reliability into capability claims — whether to report best-of-k, mean, median, or distribution — and the answer probably depends on the threat model. A bioweapon synthesis task that succeeds 1% of the time is governance-relevant; a customer-support task that succeeds 1% of the time is not deployable. The same statistic supports different conclusions in different contexts.

These are not edge cases. They are the routine cases that frontier safety frameworks must adjudicate. The right response is not to pretend the boundary is clean but to specify, for each governance decision, which elicitation envelope and which threat model the capability claim is conditioned on. The frameworks largely do this internally; the public discourse routinely loses the conditioning and treats threshold classifications as unconditional facts.

Audit theater and threshold laundering

The most consequential failure mode in current practice is what risk analysts call threshold laundering: the conversion of uncertain, elicitation-sensitive capability evidence into governance-grade threshold classifications without preserving the uncertainty. See Threshold Laundering and Audit Theater in AI Governance.

The mechanism is structural. A lab runs a capability evaluation; the result is a distribution conditioned on an elicitation envelope; the framework requires a binary or categorical threshold classification; the classification is published in a system card; downstream readers cite the classification as a fact about the model. Each step is reasonable in isolation. Together, they produce a confidence ratchet that can outrun the underlying evidence.

The frameworks themselves are aware of this. They use uncertain language internally, distinguish between Capability Reports and Safeguards Reports, and explicitly describe themselves as iterative and updateable. But the public-facing artifacts — system cards, blog posts, third-party summaries — tend to flatten this language into deployment claims. A model is described as "below the Critical threshold for autonomy" when the more accurate claim is "below the Critical threshold under the elicitation envelope evaluated in our internal program, with mitigations in place, as of the report date."

This matters because policymakers, enterprise buyers, and the public read the flattened claims. If a wiki entry on capability evaluation does anything useful, it is to keep the conditioning visible: every threshold classification is conditioned on an elicitation envelope, a threat model, a model snapshot, a date, and a safeguards configuration. Removing any of these conditions strengthens the claim beyond what the evidence supports.

The defensive measures are unglamorous:

• Pin model snapshots, framework versions, and report dates in every citation
• Require uncertainty intervals or distributional reporting on capability claims
• Distinguish base-model results from agent-system results from deployed-product results
• Treat threshold classifications as tripwires for safeguard activation, not as deployment permissions
• Require independent rerun under controlled access for the highest-stakes claims
• Use coverage notes on documents to make staleness visible

These are also the conditions under which capability evaluation could plausibly mature into an engineering audit.

Research project or engineering audit?

The unresolved question implicit in the topic is whether capability evaluation is a research activity or an engineering audit. The honest answer is that it is currently both, and the proportions vary by domain. See The Research vs Audit Question for Capability Evaluation.

In the engineering audit mode, capability evaluation looks like other compliance disciplines: fixed protocols, frozen harnesses, public scoring code, transcript retention, independent reruns, statistical uncertainty bounds. The evaluator's job is to apply a known protocol and report results within stable error bars. Components of current practice meet this bar. Public benchmarks with held-out test sets, frozen scoring code, and independent runs (ARC Prize, parts of HumanEval, FrontierMath replications) approach engineering-audit quality. So do some safety evaluations: jailbreak robustness measurements, refusal-rate audits, and structured red-team exercises with predefined success criteria.

In the research project mode, capability evaluation looks like adversarial science: the evaluator's job is to discover the elicitation protocol that reveals the capability in the first place. The threat model is provisional, the evaluation envelope is being explored, and the result is a hypothesis about what the system can do under elicitation conditions that have not yet been fully characterized. Most dangerous-capability evaluations sit closer to this end of the spectrum. The evaluator is doing science on the model, not running a fixed test.

The two modes have different epistemic standards. Engineering audits want high reproducibility, low variance, and stable protocols. Research projects want creative elicitation, sensitivity analyses, and explicit hypothesis testing. They can coexist, but a single evaluation can rarely satisfy both. A protocol stable enough for audit is usually too rigid to find novel capabilities; a protocol creative enough to find novel capabilities is usually too unstable to support compliance claims.

The constructive reading is that capability evaluation is moving from research toward audit unevenly. Domains with objective scoring and large public test sets (math, code, narrow scientific QA) are closer to audit. Domains with subjective or dual-use scoring (CBRN uplift, persuasion, autonomy) are closer to research. The frontier safety frameworks attempt to operationalize both modes simultaneously, treating threshold classifications as audit outputs while acknowledging that the underlying evaluations remain partly research.

The maturity question for governance is whether the audit components can be made independently reproducible while preserving the research components' adversarial creativity. The answer is probably yes for some domains and no for others, with the dual-use constraint forcing the most consequential evaluations to remain in a tiered-access regime rather than full public reproducibility. This is uncomfortable but not unprecedented; nuclear safeguards, biosecurity oversight, and certain financial-regulation regimes operate the same way.

Practical guidance

For readers interpreting capability claims:

1. Look for the elicitation envelope. A capability claim without prompts, scaffolds, tool access, inference budget, and model snapshot is a vibe, not a measurement.
2. Distinguish public benchmarks (reproducible but not directly governance-sufficient) from lab-internal evals (governance-relevant but not reproducible by outsiders).
3. Treat threshold classifications as conditional on the framework, the evaluation program, the threat model, and the safeguards configuration. They are not unconditional facts about the model.
4. Read system cards carefully for what is reported and what is not. Capability claims without distributional reporting, contamination analysis, or model-snapshot metadata are weaker than they look.

For builders running internal capability evaluations:

1. Specify the elicitation envelope before running the eval. Prompt distribution, scaffold variants, tool permissions, inference budget, sample counts, and refusal-policy treatment should be preregistered.
2. Report distributions, not single scores. Worst-case, median, and best-case elicitation tell different stories; collapsing them loses the information governance most needs.
3. Run sensitivity analyses on prompt format and scoring rule. If the result is unstable under perturbations users would plausibly apply, the result is not a stable capability claim.
4. Retain raw transcripts. Even when full publication is not safe, controlled-access replication requires transcript retention.
5. Distinguish base-model, system, agent, and product results explicitly. A capability that requires an agent scaffold transfers to anyone who can build the scaffold; a base-model capability transfers to anyone with API access.
6. Treat threshold crossings as triggers for safeguard activation and decision review, not as deployment permissions or prohibitions.

For policy readers:

1. The frameworks are evidence-to-action mappings, not measurement standards. They do not certify that capability claims are accurate; they certify that the lab has committed to specific actions when its own evaluations report specific results.
2. The strongest accountability lever is independent access, not full public disclosure. The transparency tension on dangerous-capability evaluations is real, and tiered access is a more defensible structure than either full disclosure or full opacity.
3. Coverage and version dates matter. Framework versions change, model snapshots change, and capability evaluations age fast. A claim about "the current state of frontier capability" without a date is already wrong.

Summary

Capability evaluation, safety evaluation, behavior evaluation, and reliability evaluation are operationally distinct because they support different deployment decisions and rely on different evidence. Capability evaluation estimates an elicited upper envelope under a specified protocol. Safety, behavior, and reliability evaluations test what the deployed system causes, tends to do, and consistently does under operating conditions. The boundary is porous at the observation layer — every capability claim is mediated by behavior, scaffolds, and reliability under the eval harness — but useful at the decision layer, where governance frameworks need a specific evidence type to trigger specific actions.

The major frontier safety frameworks (Anthropic RSP v3.2, OpenAI Preparedness Framework v2, DeepMind FSF 3.1) are governance-relevant but not measurement standards. They operationalize commitments under uncertainty rather than certifying that capability claims are stable scientific facts. Reading them as the latter is the most consequential failure mode in current discourse.

Capability evaluation is currently both research and audit, with the proportions varying by domain. Public benchmarks with frozen harnesses approach audit quality; dangerous-capability evaluations remain closer to research. The bridge is preregistered multi-elicitation reporting, tiered transparency for dual-use content, and independent reruns under controlled access — not a single canonical benchmark or a single canonical threshold. The discipline is real, useful, and incomplete.

Companion entries

Core theory:

• Frontier Model Evaluation
• Dangerous Capability Evaluation
• Capability vs Propensity in Model Evaluation
• Construct Validity in AI Evaluation
• Threat Models for Frontier AI

Practice:

• Elicitation Sensitivity in Language Models
• Benchmark Contamination
• Model Reliability Evaluation
• Agent Scaffolding and Capability Elicitation
• Multi-Elicitation Replication Packets
• Tiered Transparency for Dual-Use Evaluations

Governance frameworks:

• Responsible Scaling Policy
• Preparedness Framework
• Frontier Safety Framework
• Safety Cases for AI Systems
• Capability Thresholds and Tripwire Governance

Counterarguments:

• Audit Theater in AI Governance
• Threshold Laundering
• The Research vs Audit Question for Capability Evaluation
• Limits of Public Benchmarking for Frontier Models