Inference Scaling Laws
Inference scaling laws Inference Scaling Laws describe how performance changes as a fixed model is given more computation after training: more samples, more reasoning tokens, more verifier calls, or more search. The strongest evidence is not a universal Chinchilla-style rule, but a cluster of regime-specific regularities: repeated sampling often follows predictable pass@k curves, compute-optimal allocation can materially outperform naïve best-of-N, and some search procedures admit theoretical guarantees. What is not yet established is a single portable law analogous to Chinchilla Scaling; today’s curves depend heavily on task family, verifier quality, search policy, and the unit of “compute” being measured. (arXiv)
Coverage note: verified through April 5, 2026.
This entry is the technical companion to Inference-Time Compute. The parent article argued that inference-time computation is a distinct scaling axis; this entry asks a narrower question: when people say “inference scaling laws,” what exactly has been shown, in what regimes, and with what limits? The short answer is that there are real laws in the weak and medium sense—regular, optimizable performance-budget curves inside particular regimes—but not yet a single strong law that plays the role Training Scaling Laws and Chinchilla Scaling play for pretraining. (arXiv)
What the term should mean
An inference scaling law is an empirical or theoretical regularity linking additional inference budget to improved task performance for a fixed trained model. Depending on the paper, the budget can be counted in samples, rollouts, reasoning tokens, verifier calls, search depth, wall-clock time, FLOPs, or dollars; the performance metric can be pass@k, coverage, final-answer accuracy, verifier score, or a problem-specific success rate. That is already one reason the field is less settled than classical pretraining scaling: the independent and dependent variables are not standardized. (OpenReview)
It helps to distinguish three claims that are often conflated. Weak form: more inference compute often helps. Medium form: within a defined regime—say repeated sampling on verifier-backed coding tasks—the improvement follows a stable enough curve to forecast or optimize. Strong form: there exists a portable, Chinchilla-like inference law that survives across models, tasks, and search procedures with only modest retuning. The literature strongly supports the weak form, increasingly supports the medium form, and does not yet support the strong form; that sentence is a synthesis rather than a direct quote, but it is the fairest reading of Brown, Snell, Wu, Levi, Chen, and related 2025–2026 work. (arXiv)
The analogy to pretraining is useful but limited. Kaplan et al.’s Scaling Laws for Neural Language Models found relatively smooth power-law relationships between training loss and model size, data size, and compute, while Hoffmann et al.’s Training Compute-Optimal Large Language Models reframed the question around compute-optimal allocation of parameters and tokens. Inference scaling work borrows that spirit—fit curves, find optima, trade off one budget against another—but the object being scaled is not a single loss surface. It is an inference policy layered on top of a trained model. (arXiv)
| Property | Classical Training Scaling Laws | Current Inference Scaling Laws |
|---|---|---|
| Fixed object | training procedure changes weights | weights fixed; inference policy changes |
| Typical x-axis | model size, data, training compute | samples, rollouts, reasoning tokens, verifier/search budget, time, dollars |
| Typical y-axis | loss or benchmark performance | pass@k, coverage, final accuracy, reward, success rate |
| Best-understood regime | pretraining loss and compute-optimal allocation | repeated sampling on math, coding, proofs, and other verifiable tasks |
| Main optimization question | how to spend training compute | how to spend per-prompt compute |
| Main failure mode | undertraining / overtraining / bad data allocation | poor selection, correlated samples, bad verifier, overthinking, latency collapse |
Sources for the comparison: Kaplan et al.; Hoffmann et al.; Brown et al.; Snell et al.; Wu et al.; Levi. (arXiv)
A second clarification is linguistic. In the training literature, “scaling law” usually implies a fairly stable parametric relationship over large ranges of scale. In the inference literature, the term is used more loosely. Sometimes it means a fitted curve for pass@k under repeated sampling; sometimes it means a compute-performance frontier for a particular search algorithm; sometimes it means a provable failure-decay result for a tournament procedure; sometimes it means a vendor’s observation that longer hidden thinking improves benchmark scores. Those are related phenomena, but they are not yet one theory. (arXiv)
The core metrics: Pass@k, coverage, and inference loss
The cleanest mature regime is repeated sampling. Brown et al.’s Large Language Monkeys studies coverage, effectively pass@k: the fraction of problems solved by at least one of k generated samples. Levi’s A Simple Model of Inference Scaling Laws formalizes the same object and treats pass@k as the main metric for reasoning tasks with repeated attempts. This is the sense in which inference scaling laws are presently most concrete: not “the model thinks better” in the abstract, but “the probability that at least one of k attempts solves the task follows a regular curve.” (arXiv)
For a single problem with independent attempts and single-sample success probability (s), pass@k is just (1-(1-s)^k). The interesting part begins when one moves from an individual problem to a dataset. Real benchmarks contain easy items, hard items, and hopeless items; the aggregate curve is therefore a mixture over a distribution of difficulties. Levi’s paper models this explicitly and shows how pass@k and a proposed “inference loss” can exhibit power-law-like decay with the number of trials and with total inference cost. (arXiv)
Levi also highlights a second complication: repeated samples are not truly independent. Similar prompts can generate correlated trajectories, so “100 samples” is not the same as “100 independent shots on goal.” The paper therefore introduces an effective number of independent trials, using power-law-decaying correlations to explain why idealized pass@k formulas overstate gains in practice. Inference scaling is partly about raw budget, but equally about diversity and de-correlation of the search process. (arXiv)
Coverage is not the same thing as deployed accuracy. If a coding problem or formal proof can be automatically checked, then “at least one correct sample exists” can often be converted directly into better final performance. If there is no reliable verifier, the system needs a selector—majority vote, reward model, judge model, or search policy—to pick the right sample. Brown et al. make this distinction explicit: in coding and formal proofs, increased coverage translates into better performance because answers can be automatically verified; in domains without automatic verifiers, common selection methods plateau after several hundred samples. That gap between generation and selection is the main reason many impressive pass@k curves do not translate cleanly into product behavior. (arXiv)
A second useful distinction is between budget units. Some papers count samples or rollouts; OpenAI counts hidden reasoning tokens and bills them as output while discarding them from ordinary context after the response; Google exposes thinkingBudget and charges for thinking tokens as part of response pricing; ARC Prize often reports samples, tokens, time, and cost per task together. An “inference scaling law” is therefore always conditional on the budget unit chosen. A curve fitted against samples is not immediately comparable to one fitted against hidden reasoning tokens or dollars. (OpenAI Developers)
| Budget unit | Public examples | What it captures well | What it hides |
|---|---|---|---|
| Samples / rollouts | repeated sampling, self-consistency, ARC sample counts | breadth of search | variable per-sample cost and trajectory length |
| Reasoning / thinking tokens | OpenAI reasoning tokens, Gemini thinkingBudget |
depth of internal deliberation | selection quality and branch diversity |
| Verifier calls / comparisons | generative verification, tournament or league search | selection effort | model generation cost |
| Total inference cost (FLOPs / dollars / latency) | Levi’s cost framing, ARC cost/task, production APIs | deployment reality | internal allocation details |
This table synthesizes Levi’s cost framing, Brown’s repeated-sampling regime, OpenAI and Google API semantics, and ARC’s cost reporting. (arXiv)
The major regimes currently grouped under the term
In practice, “inference scaling law” now covers at least four distinct families of behavior. Treating them as one thing obscures both the progress and the limitations of the field. The most useful way to read the literature is to ask which regime a result belongs to before comparing it to another paper’s curve. (arXiv)
Repeated sampling and Self-Consistency
This is the simplest and most mature regime. Generate multiple candidate reasoning trajectories, then either measure coverage with an oracle/verifier or pick a final answer by majority vote or some other selector. Brown et al. show law-like coverage curves in this setting; OpenAI’s o1 reporting uses the industrial version of the same idea via cons@64; ARC’s o3-preview result makes the same logic brutally concrete by scaling the sample count from 6 to 1024. If one wants the most robust public evidence that inference budget can buy more capability, this regime is still the baseline. (arXiv)
The reason this regime dominates the literature is methodological clarity. The knob—number of samples—is explicit. The performance metric—pass@k or coverage—is explicit. And in verifiable domains, the conversion from “one good sample exists” to “the system solved the task” is also explicit. That clarity is why repeated sampling currently supports the strongest “law-like” claims, even if it is not always the most compute-efficient deployment strategy. (arXiv)
Verifier and judge scaling
A second regime spends compute not only on generating more candidates, but on evaluating them. Snell et al. study search against dense process reward models; Singhi et al. compare self-consistency against generative reward-model verification under fixed budgets; DeepSeek’s Inference-Time Scaling for Generalist Reward Modeling argues that reward models themselves can be trained to scale better with extra inference compute via parallel sampling and a meta reward model. The common theme is that test-time scaling can happen on the judge side as well as on the generator side. (OpenReview)
This regime is where a lot of naïve optimism goes to die. Singhi et al. find that generative verification is usually less compute-efficient than classical self-consistency at practical budgets and can need up to 8× more compute just to match it. DeepSeek-GRM is promising precisely because it treats that limitation as a learning problem rather than assuming more verifier rollouts will automatically win. The right lesson is not that verifier scaling fails; it is that verifier scaling is itself a scaling problem with its own bottlenecks. (arXiv)
Search and tree-style scaling
A third regime scales computation by making the inference process more explicitly search-like. Wu et al. compare multiple strategies, including two tree-search algorithms, and show that the algorithmic choice can change the cost-performance frontier enough for a 7B model plus search to beat a 34B model. Chen et al. go further and give provable scaling results for knockout-style and league-style aggregation algorithms, where failure probability decays exponentially or by a power law under stated assumptions. Wang et al.’s DORA work then reframes the whole setting as resource allocation across reasoning directions rather than candidates. (OpenReview)
This regime is closest to classical Tree Search and Monte Carlo Tree Search intuitions, but it is still softer than old-school symbolic search. The “state” is often a partially formed reasoning trajectory or a family of candidate solutions rather than a clean formal object. That softness is not a defect; it is just a reminder that modern reasoning systems usually search in language-space, not in a hand-specified symbolic state space. Their inference scaling laws therefore live at the intersection of sampling, selection, and search. (OpenReview)
Hidden reasoning-budget scaling
A fourth regime is now prominent in products but thinner in the open literature: scaling hidden internal deliberation by allocating more reasoning tokens or thinking tokens. OpenAI says o1 performance improves with more time spent thinking at test time and that o3 continues to improve as it is allowed to think longer; Google exposes thinkingBudget for Gemini 2.5 and bills thought tokens as part of response cost. These are real public signals that hidden-thought budget is a scaling axis. But compared with repeated-sampling work, the public evidence is less standardized, less directly comparable across systems, and less backed by shared open benchmarks. (OpenAI)
The caution comes from the emerging negative results. Yang et al.’s Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning reports that excessively long chains of thought can impair performance in some domains. That does not falsify hidden-budget scaling; it sharpens it. The law-like question is no longer “does more internal thinking help?” but “what is the thinking-optimal allocation for this domain, model, and latency envelope?” OpenAI and Google product surfaces imply the same question operationally, even when they do not publish a clean academic curve. (OpenReview)
| Regime | Budget knob | Strongest current evidence | Main bottleneck |
|---|---|---|---|
| Repeated sampling | number of samples / rollouts | Brown et al.; ARC; cons@64-style reporting |
selector quality outside verifiable domains |
| Verifier scaling | judge/verifier calls, verifier CoT | Snell et al.; Singhi et al.; DeepSeek-GRM | verifier cost and judge quality |
| Search allocation | branches, comparisons, rollout allocation | Wu et al.; Chen et al.; DORA | search-policy design and allocation bias |
| Hidden reasoning budgets | reasoning tokens / thinking tokens | OpenAI o1/o3; Gemini thinkingBudget |
overthinking, opacity, poor cross-system comparability |
This regime map synthesizes the current public vendor evidence with the main open literature through April 2026. (arXiv)
What the empirical literature actually establishes
Brown et al.’s Large Language Monkeys remains the clearest single empirical starting point. Across multiple tasks and models, it finds that coverage scales with the number of samples over four orders of magnitude; the relationship is often approximately log-linear and can be modeled with an exponentiated power law. The paper also gives concrete domain-level numbers: on SWE-bench Lite, repeated sampling with DeepSeek-Coder-V2-Instruct raises the fraction of solved issues from 15.9% with one sample to 56% with 250 samples, beating the then single-sample state of the art. Just as important, it reports where the trick breaks: in non-verifiable domains, majority voting and reward-model selection plateau beyond several hundred samples. (arXiv)
Snell et al.’s ICLR 2025 oral, Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning, sharpens the question from “does more inference help?” to “how should a fixed inference budget be spent?” The paper studies two main routes—search against dense process reward models and adaptive updates to the response distribution—and finds that the effectiveness of each depends on prompt difficulty. Their headline result is not merely positive scaling but compute-optimal allocation: adaptive per-prompt allocation improves math-reasoning efficiency by more than 4× relative to a best-of-N baseline, and in FLOPs-matched settings test-time compute can let a smaller model outperform a 14× larger model when the smaller model already has a non-trivial chance of success. (OpenReview)
Wu et al.’s ICLR 2025 poster, Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving, broadens the algorithmic comparison. It evaluates greedy search, majority voting, best-of-n, weighted voting, and two tree-search variants across model sizes and budgets. Its main conclusion is that advanced inference strategies can be more compute-efficient than scaling parameters, and that smaller models combined with stronger inference algorithms can occupy Pareto-optimal regions of the cost-performance frontier. The paper’s most memorable specific claim is that Llemma-7B paired with its novel tree search consistently beats Llemma-34B across all tested inference strategies on MATH. (OpenReview)
Singhi et al.’s COLM 2025 paper When To Solve, When To Verify addresses a confusion that became common in 2025: if generation can be scaled, then perhaps verification can be scaled too, by letting a Generative Reward Model produce multiple verification chains of thought. Their answer is sobering. Under fixed inference budgets, classical Self-Consistency is more compute-efficient than generative verification for most practical regimes, and the generative-verifier approach can require up to 8× more inference compute before it merely matches self-consistency. The paper still finds scaling laws for the verification regime, but the compute-optimal policy favors spending more budget on solution generation than on extra verifications. (arXiv)
The 2025–2026 literature then shifts from merely observing curves to allocating and forecasting them. Wang et al.’s NeurIPS 2025 paper Every Rollout Counts formulates test-time search as a resource-allocation problem and argues that existing solution-level allocation overweights reasoning directions with many candidates; its DORA method instead allocates at the direction level and reports state-of-the-art accuracy at comparable cost. Lu et al.’s 2026 preprint Predicting and improving test-time scaling laws via reward tail-guided search argues that best-of-N lacks principled guidance and proposes estimating the tail distribution of rewards to predict scaling curves without exhaustive evaluation. The field, in other words, is already moving from raw scaling to Compute-Optimal Inference. (OpenReview)
A final domain-extension point matters. The early literature is math-heavy, and that can mislead readers into thinking inference scaling is only about contest problems. Brown et al. already include coding and formal proofs; CodeMonkeys extends the idea to real software engineering by combining serial scaling (more iterations per trajectory) with parallel scaling (more trajectories per issue), reaching 57.4% on SWE-bench Verified at an approximately $2300 budget and 66.2% when selecting over an ensemble of candidates. That does not make the phenomenon universal, but it does show that inference scaling is not confined to toy arithmetic. (arXiv)
| Work | Main scaling object | Strongest result | What it really establishes |
|---|---|---|---|
| Brown et al., Large Language Monkeys | repeated sampling / coverage | log-linear or exponentiated-power-law coverage over 4 orders of magnitude; SWE-bench Lite 15.9% → 56% at 250 samples | repeated sampling is law-like in verifiable domains |
| Snell et al., ICLR 2025 oral | adaptive test-time allocation | >4× efficiency gain over best-of-N; smaller model can beat 14× larger at matched FLOPs | compute allocation per prompt matters as much as raw budget |
| Wu et al., ICLR 2025 poster | strategy choice across budgets | Llemma-7B + tree search beats Llemma-34B on MATH | algorithm choice can dominate parameter scaling |
| Singhi et al., COLM 2025 | solve vs verify budget split | GenRM may need up to 8× more compute to match self-consistency | verifier scaling is real but often not the best use of budget |
| Chen et al., NeurIPS 2025 | tournament / league aggregation | provable exponential or power-law failure decay under assumptions | some test-time scaling laws can be proved, not just fit |
| Wang et al., NeurIPS 2025 | rollout allocation | DORA beats strong baselines at comparable cost | rollout allocation is an optimization problem |
| Yang et al., NeurIPS 2025 | CoT length | excessively long CoTs can hurt reasoning in some domains | scaling is not monotone in every regime |
The table condenses the main result of each paper rather than reproducing their full experimental scope. Sources: Brown; Snell; Wu; Singhi; Chen; Wang; Yang. (arXiv)
How to read an inference curve without fooling yourself
The first question is whether a paper reports coverage or final-answer accuracy. Coverage answers “did at least one generated sample solve the problem?” Final accuracy answers “did the system actually deliver the right answer after selection?” In verifiable domains the gap can be small; outside them it can be large. Brown’s plateau result and Singhi’s solve-vs-verify result are both reminders that many celebrated curves are fundamentally about candidate existence, not end-user-visible correctness. (arXiv)
The second question is what the x-axis really counts. A sample count curve hides variability in sample length and hidden reasoning cost. A thinking-token curve hides whether the system is exploring multiple branches or merely extending one. A dollar-cost curve is closest to deployment reality but bakes in vendor pricing, tool-call fees, and infrastructure assumptions. ARC’s results are unusually valuable because they report several of these at once: samples, tokens, minutes, and cost per task. (OpenAI Developers)
The third question is whether the evaluation preserves completeness. ARC’s public o3/o4-mini analysis shows why this matters: high-reasoning runs often failed to return enough outputs for leaderboard use, and the returned subset was biased toward easier problems. A scaling curve that excludes the hard cases because they timed out can look much better than the actual deployment curve. This is an underappreciated reason why productized inference laws are harsher than research curves. (ARC Prize)
The fourth question is whether the method is improving search breadth, selection quality, or internal deliberation. Those are not interchangeable. Brown-style repeated sampling mostly scales breadth. Singhi and DeepSeek-GRM scale selection. OpenAI and Gemini expose internal deliberation budgets. Comparing these curves without separating the mechanism is one of the easiest ways to confuse “more inference compute” with a single magic variable. (arXiv)
Why inference scaling laws are less universal than they sound
The single biggest limitation is verifiability. The strongest curves are found where the system can reliably tell whether a candidate is correct: code that passes tests, proofs that check, math with exact answers, or benchmarks with strong verifiers. Brown et al. say this directly, and much of the later literature inherits the same bias. Once automatic verification disappears, scaling the generator is no longer enough; the selector becomes the bottleneck, and plateau behavior appears much sooner. That is why pass@k is often a better scientific metric than “final deployed accuracy,” but also why pass@k can exaggerate practical capability. (arXiv)
The second limitation is algorithm dependence. A training scaling law is usually about a class of models trained in a fairly uniform way. An inference scaling law is about a whole orchestration layer: greedy decoding versus self-consistency, best-of-N versus verifier ranking, tree search versus tournament elimination, direction-level allocation versus candidate-level allocation. Wu, Singhi, Wang, and Chen all show that changing the search or selection algorithm changes the curve itself, not just the intercept. So when a paper claims an “inference scaling law,” the hidden qualifier is almost always “for this inference policy.” (OpenReview)
The third limitation is difficulty heterogeneity. Snell et al. show that the return on extra test-time compute varies strongly with prompt difficulty, which is exactly why adaptive per-prompt allocation helps so much. Easy problems saturate quickly; hopeless problems stay hopeless; the large gains come from the middle band where more search can uncover a correct trajectory. This is another way inference scaling differs from pretraining scaling: the economically meaningful object is often not one smooth global curve but a distribution of curves over prompts. (OpenReview)
The fourth limitation is negative scaling or Overthinking. Yang et al.’s NeurIPS 2025 paper Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning reports that excessively long chains of thought can impair reasoning performance in some domains, and proposes a thinking-optimal allocation strategy rather than “always think longer.” ARC Prize’s production o3/o4-mini analysis makes the same point operationally: high reasoning often failed to complete, and the subset of returned tasks was biased toward easier items, which would have inflated reported accuracy. More budget can help, but under real latency and completeness constraints it can also break the measurement. (OpenReview)
A fifth limitation is unit mismatch across systems. OpenAI’s public evidence for inference scaling is largely framed in hidden reasoning tokens and multi-sample consensus metrics; Google exposes token budgets directly; repeated-sampling papers often ignore hidden-thought tokens and count samples instead; ARC reports total task cost; search papers count rollouts or comparisons. There is no single canonical x-axis. This does not invalidate the curves, but it means that the word “law” still denotes a family resemblance more than a unified theory. (OpenAI)
Public system evidence: where the theory meets product reality
OpenAI’s o-series disclosures are the clearest public vendor evidence that inference scaling is not just an academic artifact. In Learning to reason with LLMs, OpenAI says o1 performance “consistently improves” with more reinforcement learning and with more time spent thinking at test time, and notes that unless otherwise specified o1 was evaluated at the maximal test-time compute setting. The same document reports both pass@1 and cons@64, with o1 on AIME 2024 moving from 74.4% pass@1 to 83.3% consensus@64, and GPQA Diamond from 77.3% pass@1 to 78.0% consensus@64. Those numbers are not a proof of a law, but they are strong evidence that frontier benchmark claims already depend on inference policy, not just model weights. (OpenAI)
OpenAI’s 2025 o3 launch note strengthens the claim. It says that during o3 development the company pushed another order of magnitude in both training compute and inference-time reasoning and still saw clear gains, explicitly validating that performance continues to improve the more the models are allowed to think. This is unusually direct vendor language: not merely “we made a better model,” but “the scaling trend with inference-time reasoning remains live.” OpenAI’s API docs then operationalize the point by defining hidden reasoning tokens, stating that they are billed as output tokens, and warning that they consume context even though they are not shown to the user. (OpenAI)
Google’s public materials show the same phenomenon in a more productized form. The Gemini 2.5 launch blog says Gemini 2.5 Pro leads math and science benchmarks without cost-increasing test-time techniques like majority voting, which only makes sense in a world where majority voting is a standard reasoning lever. The Gemini API docs then expose thinkingBudget directly for 2.5-series models, with 2.5 Pro using dynamic thinking by default and 2.5 Flash allowing thinking to be disabled or set anywhere from 0 to 24,576 tokens. Google also states that when thinking is enabled, response pricing includes thinking tokens, and surfaces thoughtsTokenCount in usage metadata. Google’s product surface is therefore an explicit public acknowledgement that inference compute is a tunable resource with quality-cost tradeoffs. (blog.google)
DeepSeek is important less for proving a law than for making the mechanism legible. Its deepseek-reasoner docs say the model outputs explicit reasoning_content, that max_tokens includes the chain-of-thought portion, and that prior reasoning content is not concatenated automatically into the next turn. Current pricing docs also note that the API aliases deepseek-chat and deepseek-reasoner now point to DeepSeek-V3.2, not the original R1 release, while the January 2025 R1 release note used the same API label for R1 at a much higher output price. The lesson is not just historical trivia. It is that any practical inference-scaling comparison across APIs must specify the exact model, alias, and billing semantics being measured. (DeepSeek API Docs)
ARC Prize is the best public case study because it makes the curve and the economics visible at once. In the December 2024 o3-preview evaluation, ARC reports testing at 6 samples versus 1024 samples, roughly 172× more compute, with semi-private ARC-AGI-1 accuracy rising from 75.7% to 87.5% and cost per task rising from $26 to $4,560. That is close to a canonical inference-scaling result: more rollout budget buys more capability, but at sharply diminishing economic attractiveness. (ARC Prize)
ARC’s later production-model analysis also shows why such curves do not automatically transfer to deployed systems. ARC says the public o3 model differs from the o3-preview system tested in December, that the preview-level test-time compute is not available in production, and that production o3/o4-mini high-reasoning runs frequently timed out or failed to return outputs. The same post reports much lower public scores—o3-low at 41% and o3-medium at 53% on ARC-AGI-1, with under-3% results on ARC-AGI-2—underscoring that a research system’s inference-scaling frontier is not the same thing as a product’s affordable frontier. (ARC Prize)
The economic meaning of a scaling law
The strongest practical implication of inference scaling laws is not philosophical but economic. If a smaller model with a better inference policy can beat a larger model under matched FLOPs or matched dollar budgets, then model size is no longer the only useful proxy for capability. Snell et al. make this point explicitly with their FLOPs-matched comparison; Wu et al. make it again via Pareto-optimal small-model-plus-search combinations. This is why compute-optimal inference is not a minor decoding tweak but a capital-allocation problem for any organization deploying reasoning systems at scale. (OpenReview)
Vendor APIs expose the same tension. OpenAI says hidden reasoning tokens are billed as output tokens and recommends leaving substantial output budget available for them; Google says thinking tokens are part of billable response tokens; DeepSeek bills total input and output tokens and exposes the reasoning trace directly. Under those semantics, “letting the model think” is literally an inference-budget decision, not just a style choice. That is why the economically relevant question is rarely “what is the maximum benchmark score?” and more often “what is the capability frontier under my latency and cost constraints?” (OpenAI Developers)
Current posted prices also explain why the field is obsessed with compute-optimality. OpenAI’s model pages list o1 at $60 per million output tokens and o3 at $8; Google lists Gemini 2.5 Pro output at $10 per million tokens including thinking and Gemini 2.5 Flash at $2.50; DeepSeek’s current deepseek-reasoner alias lists $0.42 per million output tokens, while the January 2025 R1 release note listed $2.19. These are not apples-to-apples quality comparisons, and DeepSeek’s alias drift makes historical comparison especially messy. But the spread itself is the point: the deployment value of an inference scaling law depends on the provider’s cost model as much as on the curve’s shape. (OpenAI Developers)
One of the most valuable 2026 connections is between inference laws and pretraining laws. Schaeffer et al.’s ICLR 2026 paper Pretraining Scaling Laws for Generative Evaluations of Language Models fits pass@k during pretraining, showing that generative evaluations introduce (k) itself as a scaling-law hyperparameter and proving that the compute scaling law can emerge as the compute-optimal envelope of a parameters-and-tokens law. That result is important because it connects the older pretraining literature to the newer inference literature rather than treating them as separate worlds. In effect, pass@k becomes a bridge variable between train-time and test-time thinking. (OpenReview)
An even more provocative link comes from Roberts et al.’s April 2026 preprint Test-Time Scaling Makes Overtraining Compute-Optimal. The paper proposes “Train-to-Test” scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. Its central claim is that once repeated sampling at deployment is taken seriously, the compute-optimal training prescription can shift toward smaller, more overtrained models. Because this is a fresh preprint rather than settled consensus, it should be treated as a research direction, not a canonical law. But it points toward the most likely next phase of the field: joint train–test optimization rather than separate scaling conversations. (arXiv)
Theory status: better than slogans, weaker than Chinchilla
There is now more theory here than many summaries admit. Levi’s A Simple Model of Inference Scaling Laws provides a statistical ansatz for pass@k and a corresponding inference loss that decays as the number of trials and total inference cost grow. The model is intentionally simple, but it does something valuable: it explains why repeated-sampling curves can look law-like even when single-shot task success is heterogeneous and samples are correlated. It turns a visible empirical regularity into a mathematically interpretable object. (arXiv)
Chen et al.’s Provable Scaling Laws for the Test-Time Compute of Large Language Models pushes further. Under assumptions that the model can generate correct answers with nonzero probability and can compare correct and incorrect candidates better than random, the paper proves that knockout-style and league-style aggregation algorithms drive failure probability down exponentially or by a power law as test-time compute grows. That is stronger than curve-fitting: it shows that certain inference-time scaling behaviors can be derived, not merely observed. The catch is that the guarantees are for specific algorithms under specific assumptions, not for generic “reasoning models.” (arXiv)
This is why it is premature to speak of a single universal law. In the training case, papers like Kaplan and Hoffmann made a relatively small set of variables feel canonical: model size, data, compute. In the inference case, the object of study is more plural: number of candidates, internal reasoning length, verifier strength, branching factor, selection rule, rollout allocation, prompt difficulty distribution, and deployment constraints. The field is making progress, but it is still a landscape of regime-specific laws rather than a single master curve. That is a synthesis, not a direct quotation, but it is the natural conclusion of the primary literature reviewed above. (arXiv)
A practical judgment
The most useful way to carry this concept into engineering is to be stricter about what is and is not established. It is established that repeated sampling can produce smooth, forecastable gains in pass@k on many verifiable tasks. It is established that smarter allocation of inference budget can dominate naïve best-of-N. It is established that some aggregation procedures admit provable error decay. It is not established that there exists a universal inference analogue of Chinchilla that tells you, across tasks and systems, the one correct balance between model size and per-query compute. (arXiv)
That does not make the field immature in a dismissive sense. It makes it closer to operations research than to folklore. The frontier question has already moved from “can test-time compute help?” to “how should a given deployment spend its limited search budget across prompts, branches, verifications, and tokens?” The papers by Snell, Wu, Singhi, Wang, and Schaeffer all point in that direction, and the OpenAI, Google, DeepSeek, and ARC materials show that product systems are now forced to confront the same optimization problem in public. Inference Scaling Laws therefore matter because they turn reasoning performance into a budgeted control problem. (OpenReview)
What remains open
The first open question is universality. The existing literature supports law-like behavior within regimes, but no single parametric form has yet shown the breadth and stability of classical pretraining laws. A universal inference law may exist only after the field converges on more standard budget units and more comparable task families—or it may turn out that inference is inherently too policy-dependent for that kind of universality. (OpenReview)
The second open question is how to count compute. Sample count, hidden reasoning tokens, verifier calls, rollouts, and dollars are all defensible budget units, but they are not interchangeable. The same system can look sample-efficient and dollar-inefficient, or vice versa. ARC’s public cost-per-task reporting is unusually useful here because it forces algorithmic gains to face retail economics. (OpenAI Developers)
The third open question is selection bottlenecks. Brown et al. and Singhi et al. both show, in different ways, that generating more candidates is only half of the problem. Without strong selectors, coverage gains stall before they become user-visible accuracy gains. That suggests the future of inference scaling may depend as much on Reward Models, Verifier-Guided Search, and allocation theory as on the generator itself. (arXiv)
The fourth open question is optimal stopping. Yang et al. show that more chain-of-thought can hurt, not just help, and ARC shows that high-reasoning configurations can fail to complete often enough to corrupt evaluation. Any mature inference scaling law will therefore need a stopping theory, not just a growth curve. A law without a stopping rule is only half a deployment policy. (OpenReview)
The fifth open question is mechanism. Inference scaling curves measure improved problem solving under larger budgets, but they do not settle whether the underlying gain is deeper reasoning, better search over memorized patterns, or some mixture of the two. That is a conceptual question more than a scaling-law question, but it matters because people often read law-like curves as evidence of a single underlying cognitive variable. The current literature does not justify that leap. It justifies a narrower claim: search at inference time has become measurable, optimizable, and often very effective. (arXiv)
Companion entries
Core theory: Inference-Time Compute, Training Scaling Laws, Chinchilla Scaling, Pass@k, Coverage, Compute-Optimal Inference
Methods: Self-Consistency, Best-of-N Sampling, Verifier-Guided Search, Process Reward Models, Tree Search, Monte Carlo Tree Search, Reasoning Tokens
Systems and case studies: OpenAI o1, OpenAI o3, Gemini Thinking Models, DeepSeek-R1, ARC-AGI, CodeMonkeys
Practice: Latency-Capability Tradeoffs, Cost-Aware Inference, Reasoning API Design, Prompt Difficulty Estimation, Adaptive Budget Allocation
Counterarguments: Overthinking, Brute Force vs Intelligence, Selection Bottlenecks, Universality of Scaling Laws, Reasoning vs Search