Evaluation Harness: Tooling for Reproducible LLM Eval at Scale

An evaluation harness is the runtime apparatus that turns benchmark items into model calls, parses outputs, scores them, aggregates results, and records artifacts for comparison over time. This article argues that harnesses are measurement instruments rather than neutral plumbing — a benchmark score is interpretable only as a joint function of model + adapter + prompt + decoding + scorer + judge + runtime + data version + aggregation, and treating it as a portable property of model weights is the dominant failure mode of public LLM evaluation.

Coverage note: verified through May 2026. Harness ecosystems move quickly; APIs, default prompts, and supported backends drift between minor releases. Consult primary repositories for current behavior.

What an evaluation harness is, and why the framing matters

An evaluation harness has a deceptively simple job description. It takes a benchmark — a dataset of inputs, targets, and metadata — and a model, runs the inputs through the model under some prompting and decoding policy, extracts the model's answer, scores it, and aggregates per-sample scores into per-task metrics. EleutherAI's lm-evaluation-harness describes itself as "a unified framework to test generative language models on a large number of different evaluation tasks," with multiple model backends, vLLM and API support, public prompts, custom metrics, and use as the engine of the original Hugging Face Open LLM Leaderboard (EleutherAI lm-eval, HF Leaderboard archive). OpenAI Evals is presented as "a framework for evaluating LLMs and LLM systems," with explicit support for private, workflow-specific evals alongside a registry of common benchmarks (OpenAI Evals). Inspect AI formalizes the concept in three components: a Task composed of a Dataset, a Solver, and a Scorer, where solvers can do prompt engineering, multi-turn dialog, and full agent scaffolds, and scorers cover exact-match, pattern-based extraction, and model-graded evaluation (Inspect Tasks, Inspect Solvers, Inspect Scorers). LightEval describes itself as a lightweight, multi-backend evaluation suite that emphasizes saving detailed sample-level outputs (LightEval).

Reading those descriptions in sequence is misleading. Each tool sits at a different point in a layered stack, and "harness" is sometimes used to mean the runtime, sometimes the benchmark adapter, and sometimes the entire end-to-end measurement program. The first move of any honest treatment is to separate those layers and accept that a harness is not a neutral pipe. It is the experimental apparatus around the benchmark — the part that decides what the model actually sees, which API surface is used, whether answers are generated or ranked by log probability, how refusals and malformed outputs are handled, whether a chat template is applied, how stop strings are interpreted, and what gets counted as success. Two harnesses running the "same" benchmark can quietly disagree on every one of those decisions.

This is not a hypothetical concern. The Hugging Face team's own analysis of MMLU showed three implementations producing materially different scores on the same model and dataset: LLaMA-65B was reported at 0.637 under HELM, 0.488 under the EleutherAI Harness implementation in use at the time, and 0.636 under the original MMLU codebase (Hugging Face MMLU analysis). That is not noise around a true value. It is a measurement-design result: three teams asked the same conceptual question — "how well does this model do on MMLU?" — and got back three different numbers because they implemented different operational definitions of the question. Treating any one of those numbers as MMLU's "true" score is a category error.

The framing this article adopts, then, is that an evaluation harness is calibrated experimental apparatus. Its outputs are conditional measurements. A harness can be perfectly reproducible — given the same inputs and seeds, it returns the same outputs — and still produce numbers that are misleading for the question the user actually wanted to ask. Reproducibility of code is a necessary condition for trustworthy evaluation. It is not sufficient.

The layered architecture of a harness

Treating "harness" as a single object collapses too many design decisions. The layers below correspond roughly to the architectural seams in lm-evaluation-harness, OpenAI Evals, and Inspect AI, and they are where harness-induced variation enters.

Task and benchmark layer

This is the dataset itself: items, targets, splits, metadata, licensing, contamination history. The harness consumes this layer; it does not own it. Important consequences follow. First, the same benchmark can be implemented in multiple harnesses with different choices about which split is "the" eval split, how few-shot examples are drawn, whether held-out items are filtered, and how malformed items are handled. Second, contamination — the chance that benchmark items appeared in pretraining or fine-tuning data — is not solved by any harness; it is a property of the benchmark and the model. Third, benchmark suites such as BIG-Bench Hard are best understood as content, with the harness providing an adapter that maps suite items to the harness's runtime conventions. The lm-eval leaderboard tasks directory documents BBH, MUSR, MATH, GPQA, IFEval, and MMLU-Pro adapters that pin specific prompt templates and scoring rules for the second-generation Open LLM Leaderboard (lm-eval leaderboard tasks).

Sample format and prompt rendering layer

Once a benchmark item is loaded, the harness must turn it into something the model will actually receive. This involves a sample schema (the harness's internal representation), prompt rendering (system prompts, user turns, few-shot exemplars, chain-of-thought instructions, output format hints), and tokenizer-aware chat templating where the model uses one. The choices here are load-bearing. The BIG-Bench Hard paper found that few-shot prompting without chain-of-thought substantially underestimated reasoning capability on the hard subset of BIG-Bench, and that adding CoT instructions in the same harness changed measured ability across many tasks (Suzgun et al., 2022). That is not a harness bug; it is the harness exposing two different operationalizations of the same underlying capability.

A subtler version of the problem is that one harness may apply a model's tokenizer chat template by default while another sends a raw prompt string, or that one harness's few-shot ordering is deterministic while another's is randomized per run. These are mechanical differences, but they produce score deltas that look like model differences.

Model dispatch layer

The harness must talk to some inference engine: a local Hugging Face Transformers model, a vLLM or SGLang server, an OpenAI-compatible API, an Anthropic or Google endpoint, or a hosted evaluation service. Each adapter has its own defaults for batching, retries, rate-limit backoff, response caching, generation parameters, stop-token handling, and timeout behavior. Two harnesses configured to "evaluate the same model" can hit different endpoints, with different decoding defaults and different retry policies, and produce different results on stochastic tasks. For hosted APIs, the situation is worse: the provider's model identifier is a moving target, and a "GPT-4o" or "Claude Sonnet 4" reference in a paper from 2024 may not address the same underlying weights in 2026.

Cloze-style multiple-choice tasks have a special subtlety in this layer. Many academic benchmarks were designed to be scored by log-likelihood: the harness scores each candidate answer's probability under the model and selects the highest-probability candidate. Hosted chat APIs typically do not expose log-probabilities for arbitrary continuations, so a harness running MMLU against a chat API generally has to switch to a generative protocol (ask the model to output "A", "B", "C", or "D" and parse the response). The two protocols are not the same measurement. A model that ranks the correct answer highest under log-likelihood may still produce a different letter under generation, especially when the chat template introduces format pressure or the answer extractor is brittle.

Scoring and judge layer

Scoring is the layer where the harness decides what counts as right. The options form a ladder of complexity:

Exact match against a string target.
Log-likelihood of candidate continuations, sometimes with normalization for length or token-count effects.
Regex / structured extraction from generative output, before exact match.
Unit tests / executable scoring for code (pass@k and variants).
Model-graded scoring, where a separate "judge" model scores the candidate's answer against a rubric.
Human review, used as a calibration gold standard or in rare-event review.

The judge case is where harness design touches a deep methodological problem. An LLM judge has its own prompt sensitivity, its own preference biases, its own version drift, and — when the judge and the candidate share a model family — the risk of in-family circularity. The harness's job is to make the judge's prompt, model, version, decoding settings, and rubric explicit and reproducible, but it cannot eliminate judge drift across model versions. Inspect AI's scorer documentation makes this explicit by treating model-graded evaluation as a first-class scorer type alongside pattern-match and exact scorers (Inspect Scorers).

Aggregation and regression layer

Once per-sample scores exist, the harness aggregates: per-task metrics, grouped task metrics (e.g., MMLU subject buckets), bootstrap or seed-level variance, cross-task averages. This layer is where a single number is produced for a leaderboard or a dashboard, and where most reporting hides. Two harnesses can produce identical per-sample correctness on overlapping items and still publish different headline numbers because of differences in micro- versus macro-averaging, weighting of subtasks, treatment of refusals, or rounding conventions. Confidence intervals are inconsistently reported across the canonical harnesses. Some, including LightEval, save sample-level results that allow downstream variance estimation; others publish only aggregates (LightEval).

For regression detection, this layer must also decide what counts as a meaningful change. A 1-point delta on a 30-item benchmark is one item — 3.33 percentage points of resolution before any model stochasticity enters. Treating a sub-resolution delta as a regression is a frequent failure mode of CI-driven eval programs.

Provenance and policy layer

The final layer is not always considered part of the harness, but it should be: the manifest of what was run, the retention policy for raw outputs, and the rules by which a harness result is converted into a decision (publish a paper, ship a model version, hold a release). Conflating the harness with the policy is one of the more common architectural mistakes. The harness produces evidence; a separate eval policy interprets it. Pass/fail thresholds, regression tolerance, multi-comparison correction, manual review triggers, cost and latency budgets, and rollback rules belong in the policy. A harness that hard-codes thresholds blocks the policy from evolving without a code change.

The architectural payoff of this decomposition is that each layer has an audit surface. When two harnesses disagree on a benchmark, the disagreement can be localized: prompt rendering, dispatch defaults, scoring protocol, or aggregation. Without the decomposition, the disagreement is rolled up into a single mysterious score gap.

Canonical harnesses, by what they actually optimize for

The five tools the prompt names are not interchangeable, and the article would mislead by listing them as parallel options. They occupy overlapping but distinct positions in the eval stack.

Harness	Primary scope	Strongest at	Weaker at	Maintainer
`lm-evaluation-harness`	Public academic benchmarks; many backends	Standardized prompts, log-likelihood MCQ, broad task coverage, leaderboard reproducibility	Tool-using agents, custom production traces, judge-heavy free-response tasks	EleutherAI; HF leaderboard adopter
OpenAI Evals	LLM and LLM-system testing; custom workflow evals	Application-shaped evals, model-graded grading, JSON sample format, hosted Evals API	Cross-vendor parity outside OpenAI's API surface	OpenAI
BIG-Bench Hard (via adapter)	Reasoning-heavy benchmark suite	Stress-testing CoT reasoning, hard subtasks of BIG-Bench	Not itself a runtime; depends on host harness adapters	Suzgun et al., adopted by lm-eval and others
LightEval	HF-ecosystem multi-backend harness	Sample-level result retention, vLLM/Transformers/Inference Endpoints/TGI dispatch, ease of inspection	Newer; smaller task surface than lm-eval; fewer leaderboard-canonical configs	Hugging Face
Inspect AI	Tasks composed of dataset + solver + scorer; agent-capable	Tool-using and agentic evals, structured logs, model-graded scorers, sandboxed solvers	Less drop-in coverage of legacy academic MCQ benchmarks; smaller community of pre-built tasks	UK AI Security Institute

A few observations follow from this comparison. First, lm-evaluation-harness is the closest thing to a default for publishable academic-style benchmarks; the second-generation Open LLM Leaderboard uses it to standardize BBH, MMLU-Pro, MATH-Lvl-5, GPQA, MUSR, and IFEval evaluations (lm-eval leaderboard tasks). Second, OpenAI Evals is shaped by the assumption that the user is evaluating an LLM system, not a raw model, and its registry/private-eval split reflects that. Third, BIG-Bench Hard (Suzgun et al., 2022) is content rather than a runtime; the question for an engineering team is which harness's BBH adapter to trust. Fourth, LightEval is positioned where sample-level inspection is the priority and where the team is already on Hugging Face's stack (LightEval). Fifth, Inspect AI's solver/scorer abstraction makes it the most natural fit for tool-using and agent evaluations, where the "model" being evaluated includes scaffolding the older harnesses were not designed for (Inspect Solvers).

A team choosing among them is choosing where to take on risk. lm-evaluation-harness minimizes the risk of having an idiosyncratic interpretation of MMLU and maximizes leaderboard comparability. A bespoke OpenAI Evals or Inspect AI task minimizes the risk of evaluating something distant from the deployed system and maximizes representativeness. Neither choice eliminates the other risk.

Harness-induced result variation: the empirical core

The most consequential property of an evaluation harness is that it can change a result. The literature on this is no longer thin.

Biderman et al.'s "Lessons from the Trenches on Reproducible Evaluation of Language Models" frames the problem from inside the lm-eval team: setup sensitivity, comparison difficulty, and lack of reproducibility/transparency are persistent methodological issues, even with a shared codebase, and lm-evaluation-harness is offered as a partial mitigation rather than a solution (Biderman et al., 2024). The paper's value for an engineering reader is its enumeration of the specific moves that change scores: prompt formatting, in-context example selection, normalization choices, answer-extraction details, and decoding parameters.

OLMES is the strongest direct response to that diagnosis: a documented standard for LM evaluation that pins prompt formatting, in-context examples, probability normalization, and task formulation, on the explicit grounds that common evaluation choices vary enough to undermine reproducible claims (Gu et al., 2024). The interesting feature of OLMES is what it standardizes: not the runtime, but the protocol. Two harnesses can both implement OLMES-conformant evaluations and remain different code.

"State of What Art?" pushes the variance argument to the prompt level: single-prompt evaluations can be brittle enough that multi-prompt evaluation should be considered standard for robust claims, because per-prompt variance can swamp small inter-model differences (Mizrahi et al., 2024). This is a methodological challenge the canonical harnesses partially address (LightEval's emphasis on saved sample-level outputs makes multi-prompt analysis tractable; lm-eval supports prompt selection via task YAMLs) but do not enforce.

The most directly relevant paper to this article's central question — do different harnesses produce different scores on the same model and benchmark? — is the COLM 2025 "Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility" (Hochlehnert et al., 2025). The authors hold model, dataset, hardware, decoding parameters, and random seeds fixed, and compare LightEval against Evalchemy on a set of reasoning-heavy benchmarks. They observe that framework differences are generally small but ranking-relevant: in their reported table, one model is unchanged on AIME24, while another moves from 22.2 to 17.7 — a 4.5-point shift driven by prompt template, response extraction, and inference-engine differences. On a 30-item benchmark like AIME24, that delta is a handful of items, but it is enough to alter close model comparisons. The paper attributes the variance to prompt templates, inference engines, and response extraction, mapping cleanly onto the layered architecture above.

Two takeaways follow. First, the magnitude of harness-induced variation is not uniformly large; the LightEval-vs-Evalchemy comparison gives a calibration point in the low-single-digit range for held-fixed comparisons on reasoning benchmarks. Second, the magnitude is enough to flip ordering in close comparisons, which is exactly where leaderboard claims tend to live. Treating "harness-induced variation" as either a decorative caveat or a total-skepticism wrecking ball both misread the evidence. The honest claim is the conditional one: a harness score is a measurement of model × harness × protocol, and in close races the harness-and-protocol factor is comparable in size to plausible model-improvement deltas.

The HF MMLU comparison sits at the larger end of the variation spectrum: a 14-point gap between two implementations of the same nominal benchmark on the same model (Hugging Face MMLU analysis). That gap is not a routine result; it reflects a period where MMLU implementations had genuine protocol differences (log-likelihood scoring, choice normalization, prompt format) that were later partially standardized. Citing it as a typical magnitude would overstate the problem. Citing it as proof that the problem is real and large enough to change leaderboard claims understates it only by being too modest.

A third class of variation deserves separation. Some prompt-protocol differences are not accidents of harness design; they are deliberate changes to the task being measured. BIG-Bench Hard with chain-of-thought and BIG-Bench Hard without chain-of-thought are not the same construct, even if they share items and a scoring rubric — they elicit different cognitive paths and answer different empirical questions. A harness that lets the user choose between them is exposing a methodological choice, not introducing noise. The article's argument about harness-induced variation should not collapse intentional elicitation differences into accidental protocol drift, because the methodological responses are different. For accidental drift, the answer is pinning and provenance. For intentional elicitation differences, the answer is naming the protocol explicitly in the result label.

Standardized vs bespoke: a false dichotomy

The framing this article inherits — standardized harnesses versus bespoke harnesses — is the most common way the topic is discussed and one of the least useful. The framing implies a single axis with comparability at one end and representativeness at the other, where any choice is a tradeoff between the two. The empirical and architectural picture is more granular.

A standardized harness can be highly reproducible and badly mismatched to the deployed system. lm-evaluation-harness running MMLU against a base-model checkpoint via log-likelihood scoring tells the team almost nothing about how the chat-tuned, RAG-augmented, tool-using, refusal-policy-bound deployed product will behave on customer queries. The score is reproducible. The score is also operationally distant from the question the team actually has.

A bespoke harness can be representative and still reproducible if it is engineered with the same discipline as the standardized ones: pinned task definitions, versioned samples, deterministic prompt rendering, captured raw outputs, and CI-integrated regression detection. "Bespoke" does not mean "ad hoc." A team's product harness can be every bit as instrumented as lm-evaluation-harness if the team chooses to instrument it.

A bespoke harness can also be the worst of both worlds: representative-looking but encoding hidden bias in the sample selection, weak judges with undocumented prompts, scoring code without tests, no raw-output retention, and no review of its own coverage. A product team's "we evaluate on our real workload" claim is not credible without versioned samples, representative failure coverage, raw traces, human-review calibration of any LLM judges, and explicit regression thresholds.

A standardized harness can be the worst of both worlds in a different way: reproducibly measuring the wrong construct, where the protocol's defaults happen to favor the model whose tokenizer and chat template fit the task. The MMLU implementation differences are a benign version of this; the gaming pressure on a single canonical leaderboard is a malignant version.

The real axes are these:

Comparability: can this result be compared against another team's result on the same benchmark?
Representativeness: does the eval reflect the question the team actually wants to answer?
Transparency: are prompts, scoring code, model adapters, decoding parameters, and judge configurations inspectable by a third party?
Operational cost: how much does it cost to run, and how often can it be rerun on regressions?
Contamination exposure: how likely is the benchmark in pretraining data?
Change sensitivity: how does the harness respond to model upgrades, provider changes, and dataset patches?

Standardized and bespoke harnesses occupy different points along all of these axes, not a single comparability/representativeness axis. The defensible engineering pattern is to run both, with shared provenance, and to treat their disagreements as data rather than as failure modes of one side.

When this fails: failure modes worth naming

The harness model has a small set of recurring failure modes that engineering teams should be able to recognize.

Treating reproducibility as a measure of validity. A reproducible harness produces the same number when re-run; a valid harness measures what the team thinks it is measuring. Reproducibility is a property of the code; validity is a property of the operational definition. The most common failure mode in public LLM evaluation is publishing reproducibility (pinned commit, JSON output, leaderboard entry) and inheriting validity by association. The OLMES authors are explicit that the same dataset under different evaluation choices yields different scores, which is a validity argument dressed as a reproducibility one (Gu et al., 2024).

Hidden defaults. Stop sequences, max-token limits, temperature, top-p, retry policy on rate limits, batching, prompt cache reuse, and tokenizer chat-template application are all defaults that vary by harness and by adapter within a harness. None of these are decisions the user is consciously making when they run lm_eval --tasks mmlu. All of them can change the result. Inspect AI's structured logs and lm-eval's task YAMLs are partial mitigations: they expose the configuration, but only if someone reads it.

Judge drift. When a harness uses an LLM judge, the result depends on the judge's model, version, decoding parameters, and rubric prompt. A judge upgrade — gpt-4-turbo to gpt-4o, or claude-sonnet-3-5 to claude-sonnet-4 — can move the apparent quality of the candidate model under evaluation without the candidate model changing at all. Without judge-version pinning, judge prompt versioning, and periodic recalibration against human review, model-graded scores are not portable across time.

Provider model drift. Hosted model identifiers are not stable. "GPT-4" in a paper from 2023 is not the same set of weights as "GPT-4" in 2026; "Claude 3.5 Sonnet" was reissued multiple times under similar identifiers. A harness that records the API model name without recording the date and provider-specific version tag is recording a moving target. Anthropic's and OpenAI's documentation now expose dated model snapshots for exactly this reason; the harness has to capture them.

Cloze/generative protocol drift. As noted above, MMLU and similar log-likelihood benchmarks must be reformulated as generative tasks for hosted chat models, and the reformulation is a choice. A harness that silently switches between protocols depending on backend is a harness that publishes scores for two different operational definitions under one name.

Aggregation hiding the disagreement. A headline benchmark score averages over subtasks, items, and sometimes runs. A team comparing models on a single number can miss the real story: that model A wins on math and loses on commonsense, that model B improves on the easy MMLU subjects and degrades on the hard ones, that the harness's aggregate is favorable to one architecture by accident of weighting. Sample-level retention, supported by Inspect AI's logs and LightEval's saved outputs, is the corrective (Inspect AI tasks, LightEval).

Treating CI gates as deployment decisions. A harness that fires regression alerts on a single aggregate delta with no variance estimate, no correction for multiple comparisons across tasks, and no minimum effect size will produce a steady stream of false alarms that desensitize the team. The mitigation is to treat regression detection as a statistical procedure: bootstrapped confidence intervals, stratified comparisons, predeclared minimum effect sizes, and explicit human-in-the-loop review for borderline cases.

Bespoke evals that overfit to themselves. A team that improves its model by iterating on its bespoke harness's gold answers — without independent regeneration of test samples and without periodic sampling against held-out distributions — has built a harness that measures progress against its own history, not progress against the world. The same risk exists for standardized harnesses if they become a community's optimization target, but it is harder to spot in private code.

The consensus harness question

The article's prompt asks whether the field needs a single consensus harness or whether harness diversity is healthy. The honest answer is that the question is mis-posed, and the more useful question is at which layer consensus should live.

A single consensus runtime would have real benefits. Cross-paper comparisons would become tractable. Reviewers could insist that any benchmarked claim be reproducible by re-running the exact harness invocation. Methodological disputes about prompt format and scoring would be settled in code rather than in prose. Regression-detection infrastructure could be shared across organizations.

A single consensus runtime would also have real costs. Monoculture creates a single point of optimizer pressure: every model team learns to game the same prompts, the same scorer, the same answer-extraction regex. Bugs in a dominant runtime become field-wide score artifacts. The runtime's design freezes a particular set of evaluation assumptions — that benchmarks are static MCQ-and-extraction tasks, that judges are themselves LLMs, that "model" refers to a single API endpoint — at exactly the moment when deployed systems are moving toward retrieval, tools, agents, and structured outputs that those assumptions don't fit. Governance becomes political: who controls the runtime controls what counts as a valid measurement.

The HF Open LLM Leaderboard archive captures the symmetric problem in microcosm. The original leaderboard was created precisely because comparing model reports across paper-level scoring code was nearly impossible without the same questions and setup (HF Leaderboard archive). It improved comparability dramatically. It also made lm-evaluation-harness defaults into de facto rules, and the leaderboard's eventual second-generation revamp — replacing several MMLU-style tasks with harder, less-saturated benchmarks like MMLU-Pro and IFEval — was in part a response to the gaming and saturation that monoculture had enabled.

The empirical and architectural evidence supports a different target: consensus at the manifest layer, not the runtime layer. A consensus manifest specifies what must be reported alongside any benchmark score:

Harness name and version (commit hash where possible).
Task name, version, and item-set hash.
Prompt template, system prompt, and rendering rules; chat template if applicable.
Few-shot configuration: number of examples, selection rule, ordering.
Model identifier, including provider-specific dated snapshot.
Model adapter (local Transformers, vLLM, hosted API, Inspect solver, etc.) and its version.
Decoding parameters: temperature, top-p, max tokens, stop sequences, sampling vs. greedy.
Scoring code and version; for log-likelihood, the normalization rule.
Judge model, version, prompt, and decoding parameters where applicable.
Aggregation rule (micro vs. macro, weighting, refusal handling).
Variance information: bootstrap or seed-level estimates, confidence intervals.
Raw outputs and per-item scores, retained at least to the point of publication or release decision.
Run date and environment metadata (hardware, framework versions where they affect determinism).

A manifest of this shape is harness-agnostic. It can be produced by lm-evaluation-harness, by Inspect AI, by LightEval, by OpenAI Evals, by an internal product harness, or by some combination running cross-harness calibration. It makes results inspectable and makes disagreements localizable. OLMES is one concrete instance of this kind of standardization at the protocol-and-reporting layer (Gu et al., 2024).

This is a stronger position than "harness diversity is healthy" alone. Diversity without provenance norms produces incomparable private scripts and cherry-picked claims. Provenance norms without diversity reproduce monoculture. Diversity with shared manifests, raw outputs, and periodic cross-harness calibration is the engineering pattern that holds up under both methodological and operational pressure.

A practical pattern: paired standard-plus-bespoke evaluation with cross-harness calibration

The defensible workflow for an organization that takes evaluation seriously combines standardized and bespoke evals with explicit calibration.

The standardized track runs public benchmarks through a recognized harness — most commonly lm-evaluation-harness — under a pinned configuration. Its job is not to drive deployment decisions; it is to produce comparable baselines for cross-model and cross-vendor sanity checks, to catch gross regressions, and to provide an external audit trail. The output of the standardized track is a set of numbers that other teams can interpret, replicated against the same harness commit.

The bespoke track runs the deployed system — including retrieval, tools, routing, refusal policies, output contracts, and any scaffolding that is part of the product — against a representative sample of real or realistic queries. Inspect AI is well-suited to this track because solvers can wrap full agent scaffolds and scorers can be configured for application-specific output contracts (Inspect Solvers, Inspect Scorers). The bespoke track's job is to drive deployment decisions: regression gates, A/B-style comparisons of model versions, and qualitative review of failure cases.

The calibration layer ties the two tracks together. At a minimum, it includes:

Cross-harness spot checks: take a small slice of overlapping benchmarks (e.g., a multiple-choice knowledge slice, a reasoning slice, an extraction-sensitive slice) and run them through at least two harnesses with prompt and scorer harmonized as far as possible. Item-level disagreement analysis localizes variance to prompt rendering, dispatch defaults, scoring protocol, or aggregation. The COLM "Sober Look" methodology — holding model, dataset, hardware, decoding parameters, and seeds fixed across LightEval and Evalchemy — is a template (Hochlehnert et al., 2025).
Production-trace replays: replay 50–100 real or synthetic product traces through both the standardized harness's approximation of the deployed system and the bespoke harness, to quantify representativeness loss when public benchmarks are used as a proxy.
Human-review calibration of judges: where the bespoke harness uses an LLM judge, periodically sample judge decisions for human review to estimate judge bias and drift, and pin the judge's model, version, and prompt.

The output of this pattern is not a single number. It is a small set of numbers with explicit provenance, plus a stated relationship between them: "On the standardized harness's MMLU, model B is +1.2 over model A. On our bespoke production harness, model B is -0.4 on customer-realistic queries because of refusal rate changes. Human-review calibration of the judge shows it is currently unbiased between the two models on safety-sensitive items." A reader can interpret that. A reader cannot interpret a single leaderboard delta with the same confidence.

A provenance schema, in checklist form

If the manifest above seems heavy, that is the point. Reproducible LLM eval at scale is heavy, and a harness that hides the weight is not making the weight smaller — it is moving it somewhere the user cannot inspect. A practical checklist for any harness-produced result intended to be shared, archived, or used in a deployment decision:

Harness: name, version (commit hash if available), invocation command.
Task: name, version, dataset hash or pin, split, item count, contamination notes if known.
Prompt: rendered prompt for at least one item (full text), system prompt if any, few-shot count and selection rule, chat template name and version.
Model: identifier, provider, dated snapshot for hosted models, weights hash for local models, adapter (vLLM/Transformers/API/etc.) and adapter version.
Decoding: temperature, top-p, top-k, max tokens, stop sequences, sampling vs greedy, seed where applicable.
Scorer: scoring code or scorer name and version; for log-likelihood, the normalization rule; for extraction, the regex or parser version.
Judge (if any): model, dated snapshot, decoding parameters, full prompt template, rubric.
Aggregation: per-task and grouped metrics; macro vs micro; refusal handling; bootstrap or seed variance.
Raw outputs: per-item input, output, extracted answer, score, judge rationale where applicable.
Run metadata: date, hardware where relevant, retry policy, rate-limit handling.

If a harness cannot produce the items above for an arbitrary run, it is not reproducible in the sense the field needs; it is rerunnable, which is a weaker property. Inspect AI's structured log format and LightEval's sample-level result retention are concrete examples of harness designs that make this checklist tractable to populate (Inspect AI tasks, LightEval).

Limits of the harness model itself

Two limits are worth naming explicitly, because they bound what any harness — standardized, bespoke, or hybrid — can deliver.

Construct validity is not a runtime property. A perfectly instrumented harness can still measure the wrong construct. An MCQ-style MMLU evaluation does not measure "knowledge"; it measures "the ability to rank a correct option higher than three distractors under a particular prompt format and scoring rule." The two are correlated, sometimes strongly, but the leap from one to the other is a methodological argument, not a code path. The literature on prompt sensitivity (Mizrahi et al., 2024), benchmark sensitivity (Biderman et al., 2024), and reasoning-evaluation reproducibility (Hochlehnert et al., 2025) is, read collectively, an argument that the construct-validity step is where most public LLM evaluation is currently weakest. A harness improves the conditions under which the step can be made; it cannot make the step.

Saturation is a moving target. A benchmark's usefulness erodes as models saturate it, as the benchmark leaks into pretraining corpora, and as the methodological community shifts to harder tasks. The first-generation Open LLM Leaderboard's transition to MMLU-Pro, MUSR, MATH-Lvl-5, GPQA, IFEval, and BBH (lm-eval leaderboard tasks) reflects this dynamic. A harness whose canonical task set freezes is either evaluating the wrong thing or signaling that the field has stopped moving. Neither is sustainable. Engineering teams should expect to migrate eval suites every 12–24 months as model capability outruns the benchmarks.

A third limit deserves mention even though it is partly out of harness scope: data contamination. A harness cannot tell whether a benchmark item appeared in pretraining data; it can only tell whether the model gets the item right. Public benchmarks in wide circulation are particularly exposed. Any harness-based claim about a frontier model's capability on a long-public benchmark should be weighed against contamination risk, and contamination-aware reporting (held-out splits, paraphrased variants, dynamic generation) belongs in the methodology stack alongside the harness.

Where the evidence runs out

The honest answer to "what is the typical magnitude of harness-induced variation?" is that the evidence is patchy and benchmark-specific. The cleanest direct comparison currently in the literature — LightEval against Evalchemy on reasoning benchmarks, with model, dataset, hardware, decoding, and seeds held fixed — gives effect sizes in the low single digits, large enough to flip close comparisons but small enough that they do not invalidate large benchmark deltas (Hochlehnert et al., 2025). The HF MMLU comparison gives a much larger effect — 14 points — but in a setting where the implementations had genuine protocol differences that have since been narrowed (Hugging Face MMLU analysis). Between those two reference points, the field does not have broad multi-benchmark, multi-model audits that would let one quote a typical magnitude with confidence.

This matters for the consensus-harness question. If broad future audits show that variation is largely configuration-mediated — i.e., it disappears when prompts, scorers, and model adapters are fully harmonized — then the case for consensus manifests gets stronger and the case for a consensus runtime gets weaker, because manifests are sufficient to suppress the variance. If audits show that variation persists after harmonization, then runtime-level effects matter more than the manifest layer can capture, and the case for either consensus runtimes or routine cross-runtime calibration gets stronger. The current state of evidence is most consistent with the first picture, but not strongly enough to settle the question.

It also matters for product decisions. A team comparing two model versions on a public benchmark with a 0.5-point delta cannot conclude much; that delta is comfortably inside plausible harness-induced variance. The same team comparing two model versions on a 5-point delta on a single benchmark, with no cross-harness check, has a stronger but still partially harness-dependent claim. The conservative engineering posture is to require either (a) a delta that is large relative to documented harness variance for the benchmark family, (b) a cross-harness replication, or (c) a bespoke production-relevant eval as the actual decision driver.

The article's central recommendation, then, is closer to a discipline than a tool choice: treat every harness number as a conditional measurement, retain enough provenance to localize disagreements, run paired standardized-and-bespoke evaluation, and be skeptical of any leaderboard-style framing that omits the provenance.

Companion entries

Core theory:

Construct Validity in LLM Evaluation
Benchmark Saturation and the Half-Life of Public Tasks
Reproducibility, Rerunnability, and Auditability in ML
Operational Definitions: When the Measurement Becomes the Construct

Practice:

Bespoke vs Standardized Evaluation Suites
Production-Trace Evaluation for Deployed LLM Systems
CI Gates for Model Regression Detection
Provenance Manifests for ML Experiments
Cross-Harness Calibration for Reasoning Benchmarks
Sample-Level Result Retention as Evaluation Hygiene

Tools and infrastructure:

lm-evaluation-harness Architecture and Task System
Inspect AI: Tasks, Solvers, and Scorers
LightEval and the Hugging Face Evaluation Stack
OpenAI Evals: Registry and Custom Workflow Evals
BIG-Bench Hard and Reasoning Benchmark Adapters
OLMES: Standards for Reproducible LLM Evaluation

Methodological issues:

LLM-as-Judge: Calibration, Drift, and Circularity
Multi-Prompt Evaluation and Single-Prompt Brittleness
Cloze vs Generative Scoring of Multiple-Choice Benchmarks
Benchmark Contamination in Pretraining Corpora
Hosted Model Identifier Drift and Time-Stable Citations

Counterarguments:

The Case for a Consensus Evaluation Runtime
Limits of Cross-Harness Calibration in Agentic Evaluations
Bespoke Evaluation Overfitting and the Self-Comparison Trap
Why Reproducibility Is Necessary but Not Sufficient