Modern neural language-model scaling laws began as a powerful empirical discovery about loss curves, not as a theory of intelligence. Kaplan et al. showed that next-token prediction loss in large transformers follows smooth power laws in model size, data, and training compute; Chinchilla corrected the dense compute-optimal recipe by showing that frontier models had been undertrained on tokens; and later DeepSeek work showed that once data quality and sparse routing matter, both the coefficients and even the choice of scaling variables stop looking universal. The upshot is that scaling laws remain indispensable engineering tools, but the frontier is now better described as a stack of scaling regimes—dense pretraining, sparse architectures, reinforcement learning, and test-time compute—rather than a single master curve in pretraining FLOPs.

Coverage note: verified through April 5, 2026.

Scaling Laws for Neural Language Models: From Kaplan to Chinchilla and Beyond

Why this topic matters

Kaplan et al. ([OpenAI/Kaplan 2020]) were not the first to observe power-law behavior in deep learning. Hestness et al. had already documented predictable scaling across machine translation, language modeling, image classification, and speech recognition. What Kaplan changed was the strategic meaning of those regularities: scaling became a practical planning instrument for frontier transformers, a way to fit small runs and forecast much larger ones.

That shift gave the field a compact language for Neural Scaling Laws and Compute-Optimal Training. It also hid a future complication. Different papers mean different things by “model size”: Kaplan centered non-embedding parameters, Chinchilla reasoned over dense-model compute budgets, DeepSeek argued for non-embedding FLOPs per token as the better scale variable, and sparse Mixture-of-Experts systems force yet another split between total parameters and active parameters. “Scale” turned out not to be one axis but a family of coordinate systems.

The central lesson from 2020 to 2026 is therefore narrower, but stronger. Scaling laws survived the Kaplan-to-Chinchilla transition. What died was the temptation to treat any one set of coefficients as universal. The best reading today is that scaling laws are robust empirical regularities inside specific regimes—defined by architecture, objective, data distribution, and compute accounting—not general laws of intelligence.

Kaplan et al. (2020): the modern scaling-law template

Kaplan et al. studied autoregressive transformer language models over broad ranges of parameters, dataset size, and training compute, and reported that language-model loss depends “strongly” and “smoothly” on scale, with power-law dependence spanning more than seven orders of magnitude. The paper’s abstract also emphasized two points that helped its influence: performance appeared only weakly dependent on many model-shape details, and larger models were more sample-efficient than smaller ones.

In the Kaplan regime, reducible loss followed approximate power laws such as (L(N)\propto N^{-0.076}) and (L(D)\propto D^{-0.095}), with a compute-limited frontier exponent around 0.05–0.06. Those exact numbers matter less than the structural claim: if one plots loss against log-scale increases in parameters, data, or compute, the curves are strikingly regular. That regularity is what made small-model extrapolation credible enough to guide large-model budgeting.

The most consequential result, however, was not the existence of power laws but the compute-optimal recipe they implied. Kaplan concluded that, under a fixed training-compute budget, optimal model size scales roughly as (N_{\mathrm{opt}}\propto C^{0.73}) while optimal data usage grows much more slowly, roughly (D_{\mathrm{opt}}\propto C^{0.27}). They further reported that compute-efficient training should favor very large models, relatively modest token budgets, larger batch sizes, only slowly increasing numbers of optimization steps, and stopping significantly short of full convergence.

That recommendation felt counterintuitive at the time because it implied deliberate undertraining on data. But it fit a clear intuition: bigger models extract more value from each token. In Kaplan’s framing, if compute is scarce, it is often better to buy more capacity than to exhaustively train a smaller network to convergence. This is the moment when “bigger models seem better” became a quantitative engineering doctrine.

Kaplan’s paper also smuggled in a philosophical temptation that later discourse often overextended. Because the curves were so smooth, some readers began treating them as evidence that intelligence itself must scale smoothly with compute. The paper itself made no such sweeping claim. It reported empirical regularities for next-token loss in a concrete model family and training setup. The stronger metaphysical reading came later, mostly from the culture around the result rather than from the result alone.

Chinchilla (2022): the compute-optimal correction

Hoffmann et al. ([DeepMind/Chinchilla 2022]) did not reject smooth scaling. They re-estimated the dense-transformer compute frontier by training more than 400 language models with varying parameter counts and token budgets, and concluded that many large language models of the time had been significantly undertrained. In their dense autoregressive regime, the compute-optimal answer was much closer to “scale parameters and tokens together” than Kaplan’s original prescription had suggested.

Their fitted exponents are the famous correction. Depending on method, Chinchilla reported recipes near (N_{\mathrm{opt}}\propto C^{0.50}), (D_{\mathrm{opt}}\propto C^{0.50}), with nearby fits such as 0.49/0.51 and 0.46/0.54. Compared with Kaplan’s 0.73/0.27 split, this is not a cosmetic revision. It moves the dense optimum toward much smaller models trained on much more data.

The flagship demonstration made the practical consequences hard to ignore. Chinchilla, a 70B-parameter model trained on 1.4T tokens using the same pretraining compute as Gopher 280B, outperformed Gopher, GPT-3, Jurassic-1, and MT-NLG on a wide range of downstream evaluations. Because the model itself was smaller, the paper also emphasized lower fine-tuning and inference cost at comparable or better quality.

This was the real force of the Chinchilla correction. It did not say scaling laws had failed. It said the dense frontier had been mislocated. For anyone running expensive models in production, that mattered twice: once during training, and again at inference, where smaller dense models are cheaper to serve. Chinchilla Scaling was therefore both a scientific update and an economic one.

Kaplan, Chinchilla, and the shrinking contradiction

The literature after 2022 substantially narrowed the apparent contradiction between Kaplan and Chinchilla. Pearce and Song argued that the discrepancy arose largely from using non-embedding parameters, from small-scale analysis choices, and from the extrapolation regime; when they replicated Chinchilla while constraining the setup to match Kaplan, they reported results compatible with Kaplan’s original findings.

Porian et al. pushed that reconciliation further. They identified methodological differences including last-layer compute accounting, warmup duration, and scale-dependent optimizer tuning, and reported that after correcting these factors their results aligned with Chinchilla’s conclusions; they also argued that careful learning-rate decay was not essential to recover the modern dense optimum.

The best synthesis is therefore not “Kaplan was wrong, Chinchilla was right,” nor the reverse. Kaplan was right that loss is smooth enough to forecast. Chinchilla was closer to the dense compute-optimal operating point for the regimes that later frontier labs cared about. The post-2024 replication literature mostly recasts the disagreement as one of accounting choices and regime definition rather than a deep clash of principles.

Regime	Representative source label	What scales smoothly	Compute-optimal takeaway	What later proved conditional
Kaplan dense regime	[OpenAI/Kaplan 2020]	Next-token loss vs parameters, data, compute	Under fixed compute, grow model size faster than token budget	Parameter proxy, optimizer setup, and extrapolation regime
Chinchilla dense regime	[DeepMind/Chinchilla 2022]	Same dense-LM loss frontier	Scale parameters and tokens roughly equally	Assumes dense architecture and broadly fixed data regime
DeepSeek dense revisit	[DeepSeek LLM 2024]	Loss under revised scale proxy	Best split depends on data quality; use non-embedding FLOPs/token	Coefficients are not universal constants
DeepSeekMoE and later sparse laws	[DeepSeekMoE 2024] plus later MoE work	Loss/efficiency under sparse routing	Active params, expert count, granularity, and memory matter	One-number “model size” breaks down

Table sources synthesize Kaplan, Chinchilla, DeepSeek LLM, DeepSeekMoE, and later MoE-scaling work.

DeepSeek: from revised dense scaling to sparse scaling laws

DeepSeek’s contribution comes in two layers. The first is a dense scaling revisit. In DeepSeek LLM, the authors pretrained 7B and 67B dense models on a 2T-token corpus, explicitly used scaling-law analysis to design the runs, and reported that the final 67B base model outperformed LLaMA-2 70B Base on several benchmarks, especially in coding, mathematics, and reasoning-heavy evaluations.

The second layer is more conceptually important. DeepSeek argued that parameter count is often the wrong x-axis. Their paper represented model scale with non-embedding FLOPs per token, derived power-law rules for hyperparameters such as learning rate and batch size, and reported that small-scale runs predicted the larger 7B and 67B systems well. This is a methodological extension of scaling-law practice, not merely a bigger benchmark run.

Its sharpest claim concerned data quality. DeepSeek reported materially different model-vs-data compute-optimal exponents across datasets: about 0.450/0.550 for earlier in-house data, 0.524/0.476 for improved current data, and 0.578/0.422 for OpenWebText2. Their interpretation was direct: as data quality rises, the optimal policy shifts toward scaling model size more aggressively and leaning less on token volume. That is a serious qualification of any simple “always 20 tokens per parameter” folklore derived from Chinchilla-style summaries.

The right conclusion is not that Chinchilla became obsolete. It is that the Chinchilla optimum is regime-specific. Data distribution, data quality, and the choice of scale proxy all matter. Compute-optimal training is real, but the optimum is conditional. This is one of the most important intellectual upgrades between 2022 and 2026.

DeepSeek’s extension to sparse architectures came through DeepSeekMoE. That work proposed finer-grained expert segmentation and shared experts to increase specialization while reducing redundant routed parameters. The paper reported that a 2B DeepSeekMoE matched GShard 2.9B, that a 16B DeepSeekMoE was comparable to LLaMA2 7B while using about 40% of the computations, and that a preliminary 145B configuration reached performance comparable with DeepSeek 67B dense using about 28.5% of the computations.

DeepSeek-V2 and DeepSeek-V3 then showed what that architectural path enables at larger scale. V2 used 236B total parameters with 21B activated per token, reported 42.5% lower training cost than DeepSeek 67B and sharply reduced KV-cache cost, and pretrained on 8.1T tokens. V3 scaled to 671B total and 37B active parameters, pretrained on 14.8T tokens, and reported full training in 2.788M H800 GPU hours. These are not scaling laws by themselves, but they show why sparse routing changes the feasible frontier.

Intellectual honesty matters here. DeepSeek itself did not publish a single universal MoE law equivalent to Kaplan or Chinchilla. Its MoE papers are primarily architectural and empirical. The more explicit sparse-law formalizations came afterward: fine-grained laws adding expert granularity, laws over active parameters plus dataset size and expert count, and analyses showing optimal sparsity levels under different parameter, FLOP, and memory constraints.

That is the deepest sense in which DeepSeek goes “beyond” Chinchilla. Once conditional computation enters, “model size” fractures into total parameters, active parameters, expert count, routing quality, activation ratio, and memory pressure. The law changes because the geometry of scale changes. Sparse Mixture-of-Experts models do not merely sit on the old curve more cheaply; they alter which curve matters.

What scaling laws did—and did not—predict about GPT-4

GPT-4 is the clearest public example of a frontier lab using scaling laws prospectively rather than merely narrating them retrospectively. In the GPT-4 technical report, OpenAI said a major focus was creating infrastructure and optimization methods that “behave predictably across a wide range of scales,” and reported that, from models using 1,000× to 10,000× less compute, it could accurately predict GPT-4’s final loss on an internal codebase and fit its expected HumanEval pass-rate trend with a power law.

That is an unusually strong public validation of the Kaplan-style worldview. It shows that small-scale fits were operationally useful for planning a much larger run. If one asks whether scaling laws ever became real production engineering rather than a nice-looking set of plots, GPT-4 is the strongest public “yes.”

But GPT-4 also illustrates the limits of what such laws predicted. The technical report explicitly withholds architecture, model size, hardware, training compute, dataset construction, and core training-method details. It also notes remaining weaknesses such as hallucinations and a limited context window, and describes substantial safety and post-training work. The public scaling-law result therefore covers selected aggregate quantities, not the full behavioral or product surface.

So what did scaling laws predict about GPT-4? They predicted that certain pretraining loss metrics and at least one coding benchmark proxy would extrapolate smoothly. What did they not predict? They did not predict the full architecture, the exact multimodal implementation, the post-RLHF behavior, the deployment safety envelope, the jailbreak surface, or the social consequences of the product. A smooth loss curve is useful, but it is not a complete theory of a deployed model.

What scaling laws did—and did not—predict about GPT-5

GPT-5 is revealing precisely because the public framing is different. OpenAI’s system card describes GPT-5 as a unified system composed of a fast response model, a deeper reasoning model, and a real-time router. The same materials say ChatGPT offers GPT-5-thinking-pro, which uses scaled but efficient parallel test-time compute, while OpenAI’s developer materials expose a reasoning_effort control and a 400k total context window with up to 128k reasoning-and-output tokens in the API.

That public description already signals a shift away from the classic “one dense pretraining run, one scaling curve” story. GPT-5 is presented as a system-level assembly involving routing, reasoning depth, and configurable inference effort. Those are scaling axes, but they are not the same axes that Kaplan and Chinchilla studied.

Just as importantly, the public GPT-5 materials I reviewed do not provide a Kaplan/Chinchilla-style pretraining report with model size, token budget, or compute-optimal extrapolation plots. That absence matters. It means any strong public claim that classic dense scaling laws “predicted GPT-5” would be speculation rather than citation-grounded analysis. What the public evidence supports is a narrower statement: by the time of GPT-5, frontier progress was clearly being described in system terms—routing, long context, reasoning effort, and parallel Test-Time Compute—not only in terms of pretraining FLOPs.

System	What public scaling evidence showed	What it did not show	Best reading
GPT-4	Accurate extrapolation of final loss and a HumanEval-related capability fit from much smaller runs	Architecture, compute, token budget, post-training behavior, deployment properties	Strong public validation of prospective scaling-law forecasting on selected metrics
GPT-5	Public docs emphasize unified routing, reasoning modes, long context, and parallel test-time compute	No public dense pretraining scaling report or Chinchilla-style recipe	Frontier scaling has shifted toward system-level and inference-level axes

Table sources: GPT-4 technical report and OpenAI’s GPT-5 system/developer materials.

Training-compute scaling and inference-compute scaling

The cleanest way to avoid confusion is to separate Training Compute from Test-Time Compute. Kaplan and Chinchilla are training laws: they relate average next-token loss to model size, data, and pretraining FLOPs. Test-time-compute work asks a different question: once a base model already exists, how should extra inference budget be spent on a hard prompt—more reasoning tokens, more samples, better search, verifier calls, or tool use—to maximize accuracy or utility?

OpenAI’s o1 report is unusually explicit about the distinction. It says o1’s performance improves both with more reinforcement learning at train time and with more time spent thinking at test time, while also noting that the constraints on scaling this approach differ substantially from those of standard LLM pretraining. That is a direct statement that the frontier no longer lives on pretraining compute alone.

The post-2024 research literature supports this split. Snell et al. analyze verifier-based search and adaptive test-time updates, finding that compute-optimal allocation depends on prompt difficulty and can be more than 4× as efficient as best-of-N; in some FLOPs-matched settings, a smaller model plus extra inference compute beats a model 14× larger. Wu et al. similarly report Pareto-optimal trade-offs where smaller models paired with better inference algorithms outperform larger models under fixed inference budgets. Chen et al. go further and prove settings where failure probability can decay exponentially or by power law as test-time compute increases.

DeepSeek-R1 and GPT-5 pro make the same idea concrete in frontier systems. DeepSeek-R1-Zero reported steadily improving reasoning behaviors as reinforcement learning increased test-time reasoning, with the model spontaneously allocating long chains of thought to harder problems. GPT-5 pro is described by OpenAI as using scaled but efficient parallel test-time compute for hard tasks. In both cases, capability gains are coming partly from how compute is spent after the model is trained.

That said, inference scaling is not a free substitute for pretraining. Extra search and verification help most when the base model already places meaningful probability mass near good answers. If the pretrained model is fundamentally ignorant, more sampling mostly explores bad trajectories. Pretraining shapes the prior; test-time compute spends extra budget to search, deliberate, and verify within that prior. The two forms of scaling interact, but they are not the same law.

Dimension	Training-compute scaling	Test-time-compute scaling
Main unit of compute	Pretraining/update FLOPs	Reasoning tokens, samples, searches, verifier calls, tool invocations
Typical target	Average loss or broad capability frontier	Per-instance accuracy or utility
Canonical literature	Kaplan, Chinchilla	o1, verifier/search work, R1-style reasoning
Regularity style	Smooth and relatively stable within a model family	More task-dependent and policy-dependent
Failure mode	Undertraining, bad compute allocation	Wasted search on weak base models

Table sources: Kaplan, Chinchilla, OpenAI o1, Snell et al., and related inference-scaling work.

Do scaling laws transfer across modalities?

Broadly, yes. Identically, no. Hestness et al. documented power-law generalization scaling years before the GPT boom across machine translation, language modeling, image classification, and speech recognition. The existence of smooth scale-sensitive error curves is therefore not a text-only artifact.

Henighan et al. extended that picture to autoregressive generative modeling in images, video, multimodal image↔text, and mathematical problem solving, finding smooth power-law-plus-constant behavior and near-universal optimal-size exponents across those autoregressive domains. This is one reason many researchers began treating language-model scaling laws as part of a broader generative-modeling pattern.

Vision quickly showed, however, that transfer is not literal identity. Google’s Scaling Vision Transformers study characterized the performance-compute frontier for models from 5M to 2B parameters and datasets up to 3B images, found that the frontier is fit well by a saturating power law, and emphasized that larger models are more sample-efficient. But the surrounding training and transfer setup differs from text autoregression, which already undermines any notion of one universal recipe.

Diffusion models supply another version of the same story. Recent work on diffusion transformers reports explicit scaling laws for pretraining loss versus compute, model size, and data, and argues that those fits can predict text-to-image generation loss and correlate with synthesis quality metrics such as FID. The functional form remains power-law-like; the objective and metric mapping are not the same as next-token language modeling.

Speech appears to scale predictably as well. The OWLS multilingual speech work, trained on up to 360K hours across 150 languages, reports that final performance can be reliably predicted under scaling and that larger models improve automatic speech recognition across both low-resource and high-resource languages. Again, the regularity transfers, but the regime specifics do not collapse to a single text-derived law.

The newest multimodal evidence sharpens the point. Apple’s native multimodal scaling study trained 457 models and found no inherent advantage for late fusion over early fusion when models are trained from scratch; early fusion was more efficient at smaller scales, the compute-loss slopes were similar across early and late fusion, parameters and training tokens should be scaled roughly equally for optimal performance, and sparse MoE variants benefited from modality-specific specialization. Multimodal models clearly have scaling laws, but not simply the text-only Chinchilla law repeated verbatim.

So the correct answer is neither “scaling laws are language-only” nor “one law governs every modality.” What transfers is the empirical fact that error or loss often scales smoothly over large ranges. What does not transfer automatically are the exact coefficients, the best scale proxy, the right architecture family, or the compute-optimal allocation between model size and data. Multimodal Pretraining expands the phenomenon while simultaneously weakening claims of universality.

Data walls, synthetic data, and new scaling substrates

The data-wall debate is the natural stress test for Chinchilla-style reasoning. Villalobos et al. estimate that, if current trends continue, frontier language models will be trained on datasets roughly equal to the available stock of public human-generated text sometime between 2026 and 2032, and they discuss synthetic data, transfer from data-rich domains, and improved data efficiency as the main escape routes. A text-data wall is therefore not a slogan. It is a quantified concern.

But “text wall” is not the same as “frontier wall.” The strongest version of the wall claim applies to dense next-token pretraining on finite stocks of public human text. It does not by itself show that progress must stall once text saturates. It shows that the old substrate becomes constrained.

One response is better data rather than merely more data. DeepSeek’s scaling analysis reports that compute-optimal exponents move with data quality, while Sorscher et al. show that in vision one can even beat standard power-law behavior by pruning the data distribution toward more learnable examples. Fixed-distribution laws can sometimes be improved by changing the distribution. Data Pruning is therefore part of the scaling conversation, not a side issue.

Another response is synthetic data. Qin et al. find that synthetic corpora can obey predictable scaling and can serve as a viable training source, but they also report plateaus around 300B tokens in their setup and show that the optimal synthetic token budget depends on model size. Synthetic data looks promising, but the evidence does not support the fantasy of an infinite free lunch.

A third response is new modalities and new architectures. Sparse MoE systems reduce per-token active compute and let frontier runs absorb far larger token budgets, while native multimodal models increase the available training substrate beyond public web text alone. DeepSeek-V2/V3 and Apple’s multimodal study both support the broader point: if text-only dense scaling gets squeezed, progress may continue by changing the substrate rather than by denying the wall.

A fourth response is reinforcement learning and inference-time reasoning. OpenAI’s o1 report, DeepSeek-R1, and GPT-5’s public system framing all point toward a regime where capability gains depend partly on train-time RL and partly on test-time allocation of compute, not just on more internet tokens. This does not eliminate the pretraining problem. It changes the stack of ingredients needed for frontier progress.

The most honest summary is therefore conditional. If “scaling” means Chinchilla-style dense pretraining on public human text, then a wall is plausible and increasingly well-motivated. If “scaling” means effective compute allocated across higher-quality data, synthetic corpora, multimodal inputs, sparse architectures, reinforcement learning, and Test-Time Compute, then the frontier looks less like a hard stop and more like a change of substrate. The current debate is real because both statements contain truth.

Scaling laws as empirical regularities, not a theory of intelligence

At their strongest, scaling laws are empirical regularities. They say that for a fixed family of models, objectives, and data processes, loss or error often follows smooth power laws over large ranges. That claim is extremely useful for forecasting. It is also much weaker than the claim that intelligence itself is a smooth scale-invariant quantity.

Theoretical work does offer partial explanations for why these regularities appear. Bahri et al. describe four learning regimes—variance-limited and resolution-limited phases for both data and model size—and derive power-law behavior from approximation and generalization structure. That is meaningful theory. But it is a theory of why particular learning curves emerge in particular statistical regimes, not proof that “intelligence” as such is governed by one universal power law.

The “emergence” debate sharpened this distinction. Wei et al. argued that some large-language-model abilities appear unpredictably only beyond certain scales. Schaeffer et al. responded that many supposed emergent abilities can disappear under different metrics or improved statistical treatment, making them more consistent with smooth underlying changes than with genuine phase transitions. The takeaway is not that emergence is fake; it is that benchmark-level jumps can be artifacts of measurement thresholds, even when underlying competence is changing continuously. Emergent Abilities should not be confused with raw loss scaling.

This matters philosophically. Cross-entropy is not a direct measure of reasoning, agency, understanding, or usefulness. The path from lower pretraining loss to “more intelligence” is mediated by architecture, objective choice, post-training, tool use, long-context behavior, routing, and inference policy. GPT-4 and GPT-5 together make that point especially clear: one publicly validated pretraining extrapolation, the other publicly foregrounded system routing and test-time reasoning. Neither licenses the claim that intelligence reduces to a single smooth law in training compute.

The strongest defensible claim today is therefore narrower and more durable: for many model families and many training or inference regimes, capability-relevant quantities improve predictably with effective compute. The strongest indefensible claim is the old sloganized version: that intelligence itself is “just scale.” The evidence is far better for the first sentence than for the second.

What “beyond Chinchilla” really means

“Beyond Chinchilla” does not mean the end of scaling laws. It means the end of treating one dense pretraining law as the whole story. Kaplan established that loss is smooth enough to forecast. Chinchilla corrected the dense compute-optimal frontier. DeepSeek showed that data quality changes the coefficients and sparse architectures change the relevant variables. GPT-4 publicly validated prospective extrapolation on selected metrics. GPT-5’s public framing suggests that frontier capability is now being scaled at the system level through routing and test-time compute as well as through pretraining.

The practical consequence is that modern scaling analysis has become more plural. Dense pretraining laws still matter. So do sparse laws, multimodal laws, synthetic-data laws, reinforcement-learning laws, and inference-scaling laws. The field is not leaving scaling behind. It is decomposing “scale” into a more realistic set of substrates.

Companion entries

Core theory: Neural Scaling Laws, Compute-Optimal Training, Chinchilla Scaling, Irreducible Loss, Emergent Abilities

Practice: Mixture-of-Experts, Test-Time Compute, Reasoning Models, Multimodal Pretraining, Synthetic Data, RLHF

Case studies: GPT-4, GPT-5, DeepSeek LLM, DeepSeekMoE, DeepSeek-V3

Counterarguments: Data Wall, Data Pruning, Scaling Laws and Intelligence, Benchmark Threshold Effects