Infrastructure-Noise
Abstract. Infrastructure noise is the error introduced when an LLM evaluation measures not just a model, but the serving stack, retry logic, prompt renderer, decoding defaults, harness version, grader, and runtime environment wrapped around it. The main claim of this article is that many “model score” deltas are better interpreted as measurements of a full Evaluation System, and that cross-paper comparisons are often invalid unless the evaluation pipeline is versioned, logged, and treated as part of the experimental treatment. Recent harness-variation studies, agentic-eval infrastructure experiments, and vendor reproducibility documentation show that this is not a hypothetical nuisance; in some settings it can move scores by enough to swamp leaderboard gaps. OpenAI Developers+3arXiv+3arXiv+3
Coverage note: verified through May 11, 2026.
Infrastructure Noise: When the Pipeline Fails Your Eval
Definition
Infrastructure noise is any variance or bias in an evaluation score caused by the evaluation pipeline rather than by the intended model capability. It includes Model Serving Stochasticity, retry and rate-limit handling, API backend drift, prompt-rendering differences, tokenizer and chat-template changes, sampling defaults, answer-extraction heuristics, dataset loading and shuffling, grader configuration, harness version changes, hardware/runtime resource limits, and concurrency effects.
This is broader than ordinary statistical uncertainty. A binomial confidence interval around an accuracy score answers: “Given this fixed evaluation procedure, how uncertain is the finite-sample estimate?” Infrastructure noise asks a prior question: “Was the evaluation procedure actually fixed, and was it measuring the same object across runs, papers, labs, and time?” Anthropic’s statistical-evals work frames evals as measurement under noise and recommends power analysis for hypotheses such as a three-percentage-point model difference, but infrastructure noise adds another layer: a nominally identical score can be shifted by hidden implementation choices before sampling error is even calculated. Anthropic
A compact model is:
observed score=g(M, D, P, R, S, H, E, G, T)+ϵ\text{observed score} = g( M,\ D,\ P,\ R,\ S,\ H,\ E,\ G,\ T ) + \epsilonobserved score=g(M, D, P, R, S, H, E, G, T)+ϵ where MMM is the model, DDD the dataset, PPP the rendered prompt, RRR the decoding and retry policy, SSS the serving backend, HHH the harness implementation, EEE the execution environment, GGG the grader, TTT the time-dependent state of external services, and ϵ\epsilonϵ ordinary sampling noise. In a clean experiment, only MMM varies. In many published LLM evals, several of these variables vary implicitly.
Why the pipeline is part of the treatment
The core methodological error is to treat the harness as neutral instrumentation. In traditional supervised ML, evaluation often reduces to running deterministic predictions over a fixed test set and computing a metric. LLM evaluation is less clean. The “prediction” may be sampled, chat-formatted, few-shot conditioned, tool-mediated, retried after transient failures, parsed with regex, or judged by another model. In agentic settings, the model may install packages, call tools, write files, run tests, and modify an external environment over many turns. Anthropic’s agent-eval guidance explicitly describes multi-turn agent evaluation as a pipeline involving tools, tasks, environments, an agent loop, and grading logic, not merely prompt-response scoring. Anthropic
EleutherAI’s reproducibility paper makes the same point for apparently simpler academic benchmarks: minor differences in prompts, formatting, and implementation details can significantly change results, and reimplementations without original evaluation code are likely to diverge in ways that undermine fair comparison even on the same benchmark. arXiv This is the methodological heart of Harness Divergence: a benchmark name such as “MMLU,” “ARC-Challenge,” or “SWE-bench” is not a complete experimental specification. It is a family name for many possible procedures.
A useful distinction:
| Layer | What people often report | What actually affects the score |
|---|---|---|
| Dataset | “MMLU 5-shot” | Dataset version, subject filtering, answer order, few-shot examples, split choice, shuffling, prompt style |
| Prompt | “standard prompt” | Exact text, whitespace, chat template, system/developer messages, separator tokens, rendered few-shot examples |
| Inference | “temperature 0” or not reported | Temperature, top-p, max tokens, stop sequences, seed, logprobs availability, backend nondeterminism |
| Serving | Model name | Exact model ID, alias resolution, backend fingerprint, tokenizer/chat serialization, provider route, rate limits |
| Harness | Tool name | Version, commit, task YAML, task version field, batching, collation, answer extraction, scoring implementation |
| Runtime | Usually omitted | CPU/RAM/GPU, sandbox provider, kill thresholds, network egress, package installation, time limits |
| Grader | “accuracy” or “LLM judge” | Parser, regex, rubric, judge model, judge prompt, judge sampling settings, aggregation |
The crucial implication is that “same benchmark” does not mean “same evaluation.” It means “possibly comparable if the hidden layers match.”
Main sources of infrastructure noise
Model serving stochasticity
Hosted LLM APIs are often nondeterministic even when the user sends the same prompt. OpenAI’s API documentation states that Chat Completions are nondeterministic by default and exposes seed and system_fingerprint as partial controls; it also notes that determinism can be affected by backend model-configuration changes. OpenAI Developers Microsoft’s Azure OpenAI reproducibility documentation similarly says repeated identical questions are likely to yield different responses by default, and warns that determinism is not guaranteed even with the same seed and system_fingerprint, especially for longer outputs. Microsoft Learn
For evals, this means that a run is not fully specified by a model name and prompt. A result should record model ID, serving endpoint, seed if supported, decoding parameters, backend/system fingerprint if exposed, timestamps, and ideally per-sample outputs. Otherwise, a later rerun may differ for reasons that cannot be reconstructed.
Retry policies and rate-limit handling
Rate limits are part of the serving environment, not an implementation detail outside the eval. OpenAI documents rate limits as restrictions on how often a user or client can access the API, motivated by abuse prevention, fair access, and infrastructure load management; rate limits can apply through requests-per-minute, tokens-per-minute, daily caps, and related metrics. OpenAI Developers OpenAI’s rate-limit cookbook discusses 429 and RateLimitError failures and recommends strategies such as throttling and exponential backoff. OpenAI Developers
A retry policy can change a score in several ways. If a harness retries transient failures, the sample may be scored; if it does not, the same sample may become a failure. If the retry changes random seeds, timeout budgets, request order, or concurrency, it may alter output distributions. If failed requests are silently dropped, the denominator changes. If a model gets more wall-clock time because of retries, an agentic task may become easier. The retry policy is therefore part of the Eval Runtime Contract.
Prompt rendering and chat-template differences
Prompt rendering is one of the most visible sources of harness divergence. EleutherAI’s reproducibility paper reports large prompt-style effects on ARC and MMLU: for example, Mistral-7B scores 50.1% versus 72.4% on ARC-Challenge under two prompt styles in their table, and Mixtral-8x7B scores 56.7% versus 81.3% under the same comparison. arXiv The paper’s broader recommendation is not merely “write down the prompt,” but make the actual evaluation code and implementation details available, because paper-rendered prompts are often stylized and do not exactly match code. arXiv
The problem becomes sharper for chat models. OpenAI’s token-counting documentation notes that the exact conversion of chat messages into tokens may vary from model to model, making future token-count estimates approximate. OpenAI Developers A harness that serializes a chat conversation as raw text, another that applies a Hugging Face chat template, and a third that uses a provider-native messages API may be asking related but non-identical questions.
Sampling-temperature defaults
Sampling parameters are experimental conditions. EleutherAI’s reproducibility paper states that temperature, top-k, top-p, and beam search can significantly affect generated outputs and task performance, and that these hyperparameters should be reported for generative evals. arXiv Amazon Bedrock’s inference-parameter documentation likewise says inference parameters influence the pool of possible outputs and that parameter defaults and ranges depend on the model. AWS Documentation
The danger is not just an explicit temperature mismatch. It is an implicit default mismatch: one harness defaults a backend to greedy decoding, another inherits a provider default, another sets temperature=0.5 in examples, and another changes defaults between releases. EleutherAI’s lm-eval v0.4.9 release notes, for example, include a default-temperature update for vLLM and SGLang backends. Astrophysics Data System This kind of versioned default change can make two “same benchmark, same model” claims differ even when neither author made a deliberate methodological choice.
Version drift and model aliases
Model names are not always immutable. Anthropic’s Claude Code model-configuration documentation distinguishes full model names from aliases such as sonnet, opus, and haiku; it states that aliases point to recommended provider versions and update over time, and recommends full model names for pinning. Claude The same documentation recommends pinning model versions before rollout through Bedrock, Vertex AI, or Foundry, because unpinned aliases resolve to the latest version. Claude
For evaluation, alias drift creates a simple but serious failure mode: a paper, blog post, or leaderboard may record a model alias that later resolves to a different model. Even if the benchmark, prompt, and harness are preserved, the evaluated model may no longer be recoverable.
Harness version drift
Harnesses are software, and software changes. EleutherAI’s lm-eval paper says the library reports a task version field that increments when a task is modified in a scoring-affecting way, and it still recommends rerunning baselines for robust comparison. arXiv The lm-evaluation-harness README also exposes utilities for inspecting rendered LM inputs and checking task data integrity, which are practical responses to prompt and dataset drift. GitHub
The important point is that a harness version is not metadata decoration. It is part of the measurement instrument. A paper that says “we used lm-eval” but omits version, commit, task version, model backend, and generated prompt artifacts is under-specified.
Runtime resources in agentic evals
Agentic evaluations turn infrastructure from a background variable into a visible part of the task. Anthropic’s infrastructure-noise study reports that infrastructure configuration can move agentic coding benchmarks by several percentage points and that its Terminal-Bench 2.0 experiments showed a six-percentage-point gap between most- and least-resourced setups. Anthropic In their calibration, as many as 6% of tasks failed because of pod errors unrelated to model problem-solving; success rates increased as resource headroom reduced infrastructure failures. Anthropic
The deeper result is not just that under-resourced containers cause spurious failures. Above roughly a stability threshold, extra resources can change what the benchmark measures: a model that solves by installing large dependencies and running heavy subprocesses is advantaged under generous limits, while a model that writes lean code is advantaged under tight limits. Anthropic Anthropic also reports a smaller but directionally similar SWE-bench effect when varying RAM up to 5× baseline across 227 problems with 10 samples each. Anthropic
This is the cleanest empirical example of Infrastructure as Treatment. The VM is not merely carrying the eval; it partially defines the task.
Documented patterns in harness-variation literature
Pattern 1: Prompt style can dominate model deltas
The strongest public evidence predating the 2026 infrastructure-noise discussion is prompt and implementation sensitivity. EleutherAI’s reproducibility paper documents large score changes across prompt styles on ARC and MMLU, and explicitly argues that comparisons become nonsensical when different papers silently choose different prompt formats. arXiv The “Efficient multi-prompt evaluation” paper generalizes this into a statistical framing: popular benchmarks often rely on a limited set of prompt templates, which can affect leaderboard reproducibility; it proposes estimating a performance distribution across many prompt variants instead of trusting one prompt. arXiv
This matters because prompt variation can masquerade as capability variation. If Model A is reported with a prompt template closer to its training or instruction format, while Model B is reported under another, the paper-to-paper comparison may say more about elicitation than about model competence.
Pattern 2: Multiple-choice benchmarks are sensitive to scoring and formatting perturbations
The “When Benchmarks are Targets” paper studies MMLU and ARC-Challenge under prompt, scoring, and answer-choice perturbations. It uses LM Evaluation Harness for baseline and modified MMLU runs, describes the harness’s log-likelihood scoring over answer tokens, and reports that shuffling choice order, changing symbols, and using alternative scoring methods can shift model rankings. arXiv
This is not “infrastructure” in the narrow Kubernetes sense, but it is infrastructure in the methodological sense: the scoring pipeline and prompt renderer alter the measured quantity. A model’s rank can change because the benchmark wrapper changed, not because the model changed.
Pattern 3: The same benchmark can score differently across lm-eval-harness versions
The multilingual LLM evaluation paper gives a concrete version-drift example. In its appendix on accuracy variations, the authors report notable differences when evaluating with LM Eval Harness v0.4.1, v0.4.3, and a 42dc24/main commit, focusing on French ARC-Challenge with Mistral-7B-v0.1 across batch sizes and floating-point formats. The differences were small but consistent: for example, normalized accuracy for batch size 30 was 46.59% on v0.4.1 and 46.50% on v0.4.3 in the table, with related variations across batch sizes. arXiv
Their code inspection attributed discrepancies partly to random-number-generator behavior in few-shot settings, task interactions within the same run, data-parallel few-shot-context reuse, batch reordering and collation changes, and a v0.4.3 shift from plain text task fields to Jinja2 templates that removed trailing newlines and slightly shifted MMLU. After reverse-patching the reordering and template changes, results matched in the tested setting. arXiv
The important lesson is proportionality. A <1% shift may sound negligible, but it is not negligible if a leaderboard ranks models separated by tenths of a point, or if a paper claims a small improvement from a training intervention.
Pattern 4: Framework fragmentation affects Inspect AI, lm-eval, HELM, and custom runners
The evidence for “Inspect AI versus custom runners” is thinner than for lm-eval version drift. Through the coverage date, there is no single canonical public study that systematically runs the same benchmark through Inspect AI and a custom runner and reports a generalizable effect-size table. What is well documented is framework fragmentation: the Every Eval Ever schema project describes evaluation results as siloed by framework and explicitly names HELM, EleutherAI, Inspect, and custom scripts as ecosystems that need a common interchange format for direct comparison. EvalEval Coalition
Inspect’s own documentation also makes clear why cross-runner equivalence cannot be assumed. Inspect separates solvers, scorers, tools, and logs; scorers may extract answers heuristically, apply text similarity, use another model as judge, or implement another rubric. Inspect The Inspect Evals MMLU-Pro page gives a concrete benchmark implementation with defaults such as fewshot=0, shuffle=True, an optional command-line temperature, prompts based on EleutherAI’s lm-evaluation-harness and MultipleChoiceTemplate.SINGLE_ANSWER, and a built-in choice scorer computing simple accuracy. ukgovernmentbeis.github.io
That does not prove Inspect is “better” or “worse” than a custom runner. It proves a weaker but more important point: unless solver, scorer, prompt template, shuffling, few-shot count, temperature, and logs are aligned, the framework boundary itself is a plausible source of divergence.
Pattern 5: Agentic evals amplify infrastructure noise
Agentic benchmarks make hidden infrastructure more causally active. Anthropic’s Terminal-Bench and SWE-bench experiments show resource limits affecting pass rates; the Holistic Agent Leaderboard paper similarly frames agent evaluation as requiring standardized infrastructure, reporting a harness that orchestrates parallel evaluations across hundreds of VMs to reduce evaluation time and eliminate common implementation bugs. Anthropic
The same pattern appears in vendor and platform documentation. Google’s Vertex AI Gen AI evaluation service treats evaluation as a workflow involving datasets, metrics, model response generation, and assessment; it also supports agent evaluation using traces and response quality. Google Cloud Documentation Amazon Bedrock’s custom-metric evaluation jobs require judge prompts, evaluator models, prompt datasets, inference sources, inference-parameter configuration, and output locations; Bedrock stores custom metric definitions as JSON artifacts in the specified S3 output bucket. AWS Documentation NVIDIA’s NeMo Microservices documentation exposes LM Evaluation Harness as an evaluation type over many standard benchmarks, but still requires an evaluation target, making the target and environment part of the configured run. NVIDIA Docs
Vendor docs do not usually call this “infrastructure noise.” But they document the moving parts that create it.
Implications for cross-paper comparability
The practical consequence is harsh: many scores in different papers are not directly comparable, even when they name the same benchmark.
This does not mean all published evals are useless. It means their scope is narrower than the headline number suggests. A score is comparable to another score only under an equivalence relation over model, dataset, prompt, harness, decoding, serving, runtime, and grader. If any of those fields are unknown, comparability becomes a claim, not a fact.
| Cross-paper claim | Safe interpretation | Required evidence |
|---|---|---|
| “Model A beats Model B on MMLU by 0.7 points” | Possibly true under one harness/prompt/scoring setup | Same dataset version, same prompt template, same few-shot construction, same scoring method, same harness/task version, same decoding setup |
| “Our model improves ARC-Challenge over prior work” | Not comparable if prior work used different prompt style or cloze-vs-MMLU-style scoring | Rerun prior baseline in the same code path or prove prompt/scoring equivalence |
| “This agent is better on SWE-bench” | May reflect model capability, scaffold, runtime limits, tool policy, or retry behavior | Same model-serving config, scaffold, resource limits, timeouts, package/network policy, sample count, grader |
| “Provider X’s latest model regressed” | Could be alias drift, backend drift, prompt serialization, or actual model behavior | Full model version, timestamp, system/backend fingerprint if exposed, identical prompt and decoding parameters |
| “Leaderboard rank changed” | Could be a real capability shift or a harness/environment artifact | Versioned leaderboard protocol, raw sample logs, uncertainty intervals, environment manifest |
EleutherAI’s paper directly recommends rerunning baselines rather than copying numbers across papers when setups differ, and its task-versioning system is designed to keep task changes reproducible rather than magically making all historical results comparable. arXiv Anthropic’s infrastructure study makes a similar point in agentic coding: small leaderboard differences deserve skepticism until resource methodology is documented and matched, because infrastructure confounders stack on top of ordinary binomial uncertainty. Anthropic
The cleanest rule: a benchmark name is not a measurement protocol. A benchmark name plus a full reproducibility manifest may be.
Methodology responses
1. Pin harness versions and task versions
At minimum, reports should include the harness name, package version, Git commit, task version, task YAML/config, and any local patches. EleutherAI’s lm-eval task-version field is a good pattern: when task implementations change in scoring-affecting ways, the version increments so future researchers can identify which task definition produced a result. arXiv
Version pinning should include dependencies that can affect results: tokenizers, model-serving libraries, vLLM/SGLang/Transformers versions, CUDA/PyTorch versions for local inference, dataset package versions, and any scorer dependencies. Release notes can include behavior-affecting changes such as chat-template handling, backend support, context-length fixes, MMLU dataset source changes, and default generation parameters. Astrophysics Data System
2. Log the rendered prompt, not just the template
A prompt template is not enough. The logged artifact should include the exact per-sample rendered prompt or messages payload, including system/developer messages, few-shot examples, whitespace, separators, answer labels, stop sequences, and chat-template serialization. The lm-evaluation-harness README’s write_out.py utility is a concrete example of writing task inputs for inspection. GitHub
For chat models, the artifact should distinguish between user-visible prompt text and provider-native message structures. If the provider applies hidden or model-specific serialization, record the API request body and provider documentation version where possible.
3. Record decoding and serving parameters
Every result should include temperature, top-p, top-k where applicable, max output tokens, stop sequences, seed, number of samples per item, logprobs mode, and answer-selection rule. OpenAI and Azure both expose partial reproducibility mechanisms but warn that backend changes and nondeterminism remain possible, so recording system_fingerprint or equivalent backend metadata matters when available. OpenAI Developers
For hosted APIs, also record exact model IDs rather than aliases. Where aliases are unavoidable, record timestamp, provider, region, deployment ID, and any backend fingerprint. Anthropic’s documentation explicitly recommends full model names or environment-variable pinning to avoid alias drift. Claude
4. Make retry, timeout, and rate-limit handling explicit
A reproducibility manifest should include concurrency, rate-limit throttle, retry count, retry backoff, timeout budget, failure classification, and whether failed samples are retried, marked incorrect, excluded, or separately reported. Rate limits exist to manage shared API infrastructure, and hitting them can produce 429 or RateLimitError failures; therefore, rate-limit handling is part of the measurement pipeline. OpenAI Developers
For agentic evals, retries are especially dangerous because they may grant extra time, reset state, or produce multiple attempts. A fair protocol should define whether a retry is a transport retry, a model retry, or a new attempt.
5. Report environment artifacts in results
Results should include a machine-readable environment artifact: container image digest, OS, CPU/GPU/RAM allocation, runtime limits, kill thresholds, sandbox provider, network policy, package cache policy, dependency lockfile, and evaluator hardware where relevant. Anthropic’s recommendation for agentic evals is to specify both guaranteed resource allocation and hard kill threshold per task, because a single pinned value can create spurious OOM instability. Anthropic
The broader ecosystem is moving in this direction. Every Eval Ever argues that a score without configuration is noise and proposes capturing prompt templates, inference parameters, and system states for provenance. EvalEval Coalition Bedrock stores custom metric definitions as JSON artifacts in result output locations, and Google’s Vertex AI evaluation workflow treats datasets, metrics, response generation, and assessment as explicit steps rather than hidden implementation details. AWS Documentation
6. Log per-sample outputs and grading traces
Aggregate scores are lossy. Per-sample logs allow future researchers to identify whether a score change came from model output, answer parsing, grader judgment, retry failure, prompt drift, or dataset difference. EleutherAI’s lm-eval paper notes per-sample logging for post-hoc score reproduction and error analysis. arXiv Inspect’s documentation similarly emphasizes logs after runs, and its scorer design makes clear that answer extraction and rubric choice are part of the eval definition. ukgovernmentbeis.github.io
For LLM-as-judge evaluations, log the judge prompt, judge model, judge sampling settings, rubric, raw judge outputs, parsed scores, and any adjudication rules. The Judge Reliability Harness paper reports that judge reliability varies across models, benchmarks, and perturbation types, with consistency issues caused by formatting, paraphrasing, verbosity, and label flips. arXiv
7. Rerun baselines in the same pipeline
The most defensible comparison is not “our score versus their reported score.” It is “our model and their baseline, rerun under the same current pipeline.” This is expensive, but it directly addresses harness divergence. EleutherAI explicitly advises rerunning baselines, even while offering task versioning to improve reproducibility. arXiv
When rerunning is impossible, a paper should downgrade the claim: “not directly comparable,” “reported for context,” or “approximate comparison under differing harnesses.” This is not pedantry. It prevents small deltas from being interpreted as real capability gains.
8. Separate statistical uncertainty from infrastructure uncertainty
A good eval report should not collapse all uncertainty into a single accuracy point. It should report at least three layers:
| Layer | Example question | Example response |
|---|---|---|
| Sampling uncertainty | Is the test set large enough to detect a 3-point difference? | Confidence intervals, bootstrap SE, power analysis |
| Run stochasticity | Does the same model/harness produce stable results across seeds or reruns? | Multiple seeds, repeated runs, per-sample variance |
| Infrastructure uncertainty | Does the score change under plausible serving/harness/runtime variants? | Harness-version sensitivity, prompt-template sensitivity, resource sweep, retry-policy audit |
Anthropic’s “Adding Error Bars to Evals” work addresses the first layer through power analysis and clearer statistical reporting. Anthropic The infrastructure-noise literature argues that the second and third layers must also be measured, because otherwise a confidence interval around an under-specified pipeline is falsely precise. Anthropic
Minimum reproducibility manifest
A practical Eval Reproducibility Manifest should look more like a build artifact than a methods paragraph.
| Field | Required contents |
|---|---|
| Benchmark identity | Name, dataset source, dataset version/hash, split, filtering, subject list |
| Harness identity | Framework, version, Git commit, task version, task config/YAML, local patches |
| Prompt artifacts | Exact rendered prompts/messages per sample, chat template, few-shot examples, whitespace, answer labels |
| Model identity | Full model ID, provider, endpoint/region, deployment ID, alias resolution timestamp |
| Serving metadata | Backend/system fingerprint if exposed, tokenizer/chat serialization assumptions |
| Decoding config | Temperature, top-p, top-k, max tokens, stop sequences, seed, number of samples |
| Runtime config | CPU/RAM/GPU, container image digest, sandbox provider, resource guarantees and kill thresholds, timeouts |
| API control | Concurrency, rate limits, retry/backoff policy, failure handling, request timestamps |
| Grading config | Parser, scorer, rubric, judge model, judge prompt, judge decoding settings, aggregation |
| Logs | Per-sample input, raw output, parsed answer, score, error state, retry trace |
| Statistics | Standard error/CI, number of samples, rerun variance, power analysis where relevant |
| Provenance | Code repository, lockfiles, artifact hashes, result schema version |
Not every field matters equally for every benchmark. A static log-likelihood multiple-choice task does not need a sandbox kill threshold. A coding-agent benchmark absolutely does. The methodological standard should be: report every variable that could plausibly change the score or prevent reproduction.
Open question: engineering bug or intrinsic feature?
There are two live positions.
Position A: infrastructure noise is a transient engineering problem
On this view, infrastructure noise is analogous to early reproducibility problems in any maturing experimental field. Better tooling, containerization, task versioning, prompt logging, model pinning, and common result schemas will reduce most of the problem. Evidence for this view includes lm-eval’s standardized task implementations and task-version fields, utilities for inspecting inputs, vendor-exposed reproducibility controls such as seeds and backend fingerprints, and standardized agent-eval infrastructure such as HAL. OpenReview+3arXiv+3GitHub+3
This view is partly correct. For many closed-form, static, multiple-choice evals, most infrastructure noise is avoidable engineering debt. Pin the harness, freeze the dataset, log rendered prompts, set deterministic decoding, fix the model version, publish outputs, and rerun baselines. The remaining variance becomes smaller and easier to reason about.
Position B: infrastructure noise is intrinsic to LLM evaluation
On this view, the pipeline is not a removable nuisance because LLM behavior is inherently interactional. The prompt, decoding policy, tool scaffold, runtime environment, and judge are part of the task definition. Multi-prompt evaluation work supports this view by treating prompt choice as a distribution to estimate over, not merely a nuisance to eliminate. arXiv Anthropic’s resource-sweep result also supports it: once resources move beyond reliability stabilization, they change which agent strategies succeed, meaning there is no single resource-neutral score for an agentic coding task. Anthropic
The strongest version of this view is emerging in causal-evaluation proposals. A 2026 OpenReview submission argues that measured LLM performance is produced by a full evaluation system including workloads, prompting, decoding parameters, and the software-hardware stack, and that causal attribution to the model is unreliable without a controlled specification of that system. Its larger empirical claims should be treated as contested until independently validated, but the causal framing matches the infrastructure-noise problem. OpenReview
Synthesis
For static academic benchmarks, infrastructure noise is often a solvable reproducibility defect. For generative, API-mediated, model-judged, and agentic evals, it is partly intrinsic. The goal should not be to pretend the pipeline disappears. The goal should be to choose which parts of the pipeline are controlled constants, which are experimental variables, and which are modeled as random effects.
A mature evaluation report might therefore say:
We estimate model performance under prompt distribution PPP, serving backend SSS, harness version HHH, and runtime regime EEE, with uncertainty decomposed into sample error, run stochasticity, and infrastructure sensitivity.
That is less leaderboard-friendly than a single score. It is also more honest.
Practical protocol
Before running the eval
Define the estimand: model-only capability, application-stack performance, or end-to-end agent performance. If the target is model-only capability, freeze and document as much infrastructure as possible. If the target is application-stack performance, do not hide the stack; describe it as part of what is being evaluated.
Pin the harness, task definitions, model version, dataset, prompt renderer, decoding parameters, dependency lockfile, and runtime environment. Decide how failures are classified before the run. Decide whether retries are transport retries or new attempts. Decide whether API latency, timeouts, and package-installation failures are part of the test or excluded infrastructure faults.
During the eval
Capture exact rendered prompts, raw responses, parsed answers, scores, errors, retries, timestamps, model metadata, and backend fingerprints where exposed. For agentic evals, capture tool calls, filesystem diffs, package installs, test logs, resource usage, wall-clock time, and sandbox failures. Avoid silently dropping failed samples.
Run a small validation batch and inspect inputs before the full run. EleutherAI’s write_out.py pattern is valuable because many evaluation bugs are visible only when the actual prompt, not the intended prompt, is inspected. GitHub
After the eval
Publish aggregate scores with uncertainty, but also publish enough artifacts for forensic reconstruction. Include standard error or confidence intervals, but do not let them imply more precision than the pipeline supports. For leaderboard-style claims, rerun baselines in the same harness. For agentic evals, report resource sensitivity or at least resource guarantees and kill thresholds.
If a score delta is smaller than plausible infrastructure variation, say so directly. Anthropic’s Terminal-Bench analysis suggests skepticism for small agentic leaderboard gaps until resource methodology is documented and matched; that principle generalizes beyond their specific benchmark. Anthropic
Source map
The core reproducibility source is EleutherAI’s “Lessons from the Trenches on Reproducible Evaluation of Language Models,” which documents prompt sensitivity, implementation sensitivity, task versioning, per-sample logging, and the recommendation to rerun baselines. arXiv+2arXiv+2
The clearest same-harness-version drift example is the multilingual evaluation paper’s appendix on LM Eval Harness v0.4.1, v0.4.3, and commit 42dc24/main, including batch-collation and Jinja2-template explanations for small ARC/MMLU shifts. arXiv
The clearest runtime-infrastructure evidence is Anthropic Engineering’s 2026 infrastructure-noise study on Terminal-Bench 2.0 and SWE-bench, which quantifies resource-configuration effects and recommends documenting resource methodology as a first-class experimental variable. Anthropic+3Anthropic+3Anthropic+3
The vendor-environment sources are OpenAI and Azure reproducibility documentation for nondeterminism, seed, and backend fingerprinting; OpenAI rate-limit documentation and cookbook guidance; Anthropic model-alias pinning guidance; AWS Bedrock inference-parameter and evaluation-job documentation; Google Vertex AI evaluation-service documentation; and NVIDIA NeMo LM Harness evaluation documentation. NVIDIA Docs+9OpenAI Developers+9Microsoft Learn+9
Companion entries
Core theory: Evaluation System, Infrastructure as Treatment, Construct Validity in LLM Evaluation, Measurement Error, Causal Attribution in AI Evaluation, Model Capability vs System Capability
Harnesses and benchmarks: LM Evaluation Harness, Inspect AI, HELM, SWE-bench, Terminal-Bench, MMLU, ARC-Challenge, Holistic Agent Leaderboard
Infrastructure: Model Serving Stochasticity, Rate Limit Handling, Retry Policy, Prompt Rendering, Chat Templates, Sampling Temperature, Model Version Drift, Sandbox Resource Limits, System Fingerprint
Practice: Eval Reproducibility Manifest, Prompt Template Logging, Per-Sample Eval Logs, Harness Version Pinning, Baseline Reruns, Environment Artifacts, LLM-as-Judge Audit
Statistical methods: Power Analysis for Evals, Bootstrap Standard Error, Multi-Prompt Evaluation, Random Effects Models for Evals, Infrastructure Sensitivity Analysis
Counterarguments: Leaderboard Pragmatism, End-to-End Evaluation as the Real Metric, Overfitting to Eval Protocols, Reproducibility Cost Tradeoffs