In-Context Learning: The Surprise That Shaped LLM Practice

In-context learning (ICL) is the inference-time behavior by which sufficiently capable autoregressive language models adapt their outputs to tasks specified in their prompt — by instruction, by demonstration, or by both — without any change to their parameters. It was the empirical observation around which much of post-2020 LLM practice organized itself, and it remains one of the most theoretically contested behaviors in the field: not because the phenomenon is in doubt, but because the word "learning" papers over a family of distinct sub-behaviors that competing formal accounts each capture only partially.

Coverage note: verified through May 2026.

1. The empirical surprise

The canonical reference is Brown et al. 2020, the paper introducing GPT-3. The methodological move that made ICL legible as a research object was structural: Brown and co-authors evaluated GPT-3 in three conditions they called zero-shot, one-shot, and few-shot, in which a natural-language task description (and, in the latter two conditions, one or more input/output exemplars) was supplied entirely inside the prompt. No gradients were computed against the demonstrations. No optimizer state was updated. The model was used in pure forward-pass inference, and yet performance on dozens of NLP benchmarks rose monotonically — sometimes dramatically — as more exemplars were added.

Three things about this result deserve isolation, because subsequent discussion frequently conflates them.

First, the behavioral claim: a large pretrained autoregressive model, presented with a prompt of the form "Translate English to French: sea otter → loutre de mer; cheese → __", will often emit a plausible French translation. The model's parameters at inference time are bit-identical to its parameters before it ever saw the prompt.

Second, the scaling claim: this behavior was weak or absent in smaller models (GPT-2-scale and below) and improved with model scale, sometimes sharply. Brown et al. presented this as one of the headline findings of the paper, and it became one of the load-bearing observations in the early scaling-laws literature.

Third, the interface claim: because tasks can be specified in the prompt rather than encoded in weights, the prompt becomes the principal control surface for a pretrained model. This is the observation that turned prompt engineering from a curiosity into a discipline and, downstream, into entire product surfaces.

These three claims are independent. The behavioral claim is essentially uncontested. The scaling claim survives most subsequent work but has been refined: ICL ability does emerge with scale on many tasks, but the curves are jagged, regime-dependent, and sensitive to evaluation metric choice (Schaeffer et al. 2023 on emergent abilities as partial metric artifacts). The interface claim is the one that proved most consequential for practice — but, as later sections argue, the interface has been substantially reshaped by instruction tuning, retrieval, tool use, and long-context engineering.

What Brown et al. did not show, despite the popular reading of the paper, was that GPT-3 "learns from examples" in any sense that maps cleanly onto supervised learning. The training-set analogy was a useful piece of marketing copy and a poor piece of mechanism. The years since have been spent figuring out what ICL actually is.

2. Decomposition: what "in-context learning" names

The single most important conceptual move for understanding the literature is to stop treating ICL as one phenomenon. The label covers at least six distinguishable sub-behaviors, which different prompting setups recruit in different mixtures.

Sub-behavior	What it is	Evidence anchor
Task recognition	The prompt identifies a latent task or pretraining distribution; the model conditions on that distribution.	Xie et al. 2021; Pan et al. 2023
Format conditioning	Exemplars communicate output shape (JSON, label vocabulary, delimiter pattern, brevity, register).	Min et al. 2022
Label-space narrowing	Demonstrations restrict the answer distribution to a small set of valid tokens.	Min et al. 2022; Lu et al. 2022
Semantic-prior use	The model uses pretraining priors over what labels mean (e.g., "positive"/"negative" sentiment) rather than the demonstrated mapping.	Wei et al. 2023
Pattern completion	Attention circuits perform match-and-copy operations on repeated structure in the context.	Olsson et al. 2022
Input-label mapping	The model genuinely uses the demonstrated input-to-label correspondence to predict new inputs.	Wei et al. 2023; Garg et al. 2022

Min et al.'s 2022 result — that randomly shuffled labels often preserve much of the benefit of demonstrations on classification and multiple-choice tasks — has been widely misread as "demonstrations don't matter." It does not say that. It says the correctness of input-label pairings is often less causally important than the label space, input distribution, and format communicated by the demonstrations. Subsequent work qualified the claim further: Wei et al. 2023 ("Larger language models do in-context learning differently") showed that larger models, particularly when given flipped-label or semantically-unrelated-label conditions, can override their semantic priors and follow the demonstrated mapping. That is, the bigger the model, the more of ICL's behavior looks like genuine input-label conditioning rather than format/prior arbitration.

So the picture by 2023 was already not "few-shot examples teach the model the task." The picture was: examples select among a set of pretrained behaviors, and the mixture of selection mechanisms varies by model scale, post-training, and task structure. Calling all of this "learning" is defensible only under a sufficiently elastic definition of the word — one that conflates parameter updates, Bayesian conditioning, latent-task inference, and prompt-conditioned circuit selection.

This decomposition is not pedantry. The downstream consequences for prompt engineering, evaluation, and claims about emergent capabilities all hinge on which sub-behavior a given experiment is measuring.

3. Candidate mechanisms

Four families of mechanistic account have been advanced. None is dominant; each captures part of the behavior in regimes where it is most cleanly testable.

Implicit Bayesian inference

Xie et al. 2021 ("An Explanation of In-context Learning as Implicit Bayesian Inference") proposed that ICL emerges when the pretraining distribution contains coherent documents that can be modeled as samples conditioned on latent concepts. Under this view, the prompt's exemplars serve as evidence that lets the model perform approximate posterior inference over latent tasks, and the next-token prediction reflects marginalization over compatible tasks. Their theoretical analysis used a Hidden Markov Model construction in which ICL emerges provably, and they validated qualitative predictions on small synthetic and natural settings.

The strength of the Bayesian account is that it explains why a next-token predictor trained on a sufficiently structured distribution would acquire ICL: prompt-conditioned posterior inference is the natural Bayes-optimal behavior for such a predictor. The weakness is that it is a model of emergence, not a mechanistic readout of what GPT-4-scale models actually compute. The claim "ICL is Bayesian inference" should be read as "ICL is consistent with a Bayesian account in pretraining distributions of certain kinds," not as a direct algorithmic identification.

Gradient descent and meta-optimization

A second family of accounts asks whether transformers implement, in their forward pass, computations that are equivalent to gradient descent on the demonstrations. von Oswald et al. 2023 ("Transformers learn in-context by gradient descent") and Dai et al. 2022 ("Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers") show constructive equivalences between transformer attention and gradient steps on linear regression and related simplified objectives. Akyürek et al. 2022 ("What learning algorithm is in-context learning?") extends the analysis, showing that transformers trained on linear regression in-context implement algorithms competitive with or matching ridge regression and gradient descent.

These results are mathematically clean and conceptually important: they show that the forward pass of a transformer is expressive enough to implement learning algorithms on in-context data. They do not show that frontier LLMs, on natural tasks, actually do implement gradient descent during inference. The gap between "the architecture can simulate SGD on synthetic regression" and "the production model is running SGD on your few-shot exemplars" is large, and the claim sometimes leaks across that gap in summary articles. Gradient-descent accounts are best read as existence proofs of one possible algorithm in a controlled regime, not as descriptions of how Claude or GPT-4 handle a sentiment-classification prompt.

Mechanistic interpretability: induction heads and task vectors

The most direct causal evidence about what specific circuits implement ICL comes from mechanistic interpretability work. Olsson et al. 2022 identified induction heads — pairs of attention heads that perform a match-and-copy operation, attending to previous occurrences of the current token and then copying the token that followed — and presented evidence that their formation during training coincides with a phase change in the model's ICL ability. The argument is strong on sequence-continuation and repeated-pattern tasks: induction heads provide a concrete mechanism for the simplest forms of context-driven prediction.

Hendel et al. 2023 ("In-Context Learning Creates Task Vectors") and related work show that few-shot prompts induce compact internal representations — "task vectors" — that can be extracted from one prompt and injected into another to transfer task behavior. This points toward a picture in which ICL involves the construction, somewhere in the network's middle layers, of a compressed task representation that subsequent layers consume.

Pan et al. 2023 decomposed ICL into task recognition (using priors to identify the task) and task learning (using the examples to infer the input-label map), and showed that task recognition is largely scale-independent while task learning is the component that scales.

These results are the most epistemically secure within the mechanistic-interpretability tradition because they involve causal interventions — ablations, activation patching, vector injection — rather than only behavioral correlation. They are also the most narrowly scoped: induction heads have been characterized at small to mid scale, and the extrapolation to frontier models is partially evidenced but not closed. The honest reading is that attention-mediated pattern completion and compressed task representations are part of the story, especially for simpler ICL behaviors, but they have not been shown to explain all of ICL across all model scales.

Comparison

Account	Strongest evidence	Scope	Where it stops
Bayesian inference (Xie et al. 2021)	Theoretical derivation; emergence on synthetic HMM-structured pretraining	Explains why ICL would emerge from next-token pretraining on coherent documents	Not a direct mechanistic claim about frontier-model computation
Gradient descent / meta-optimizer (von Oswald et al. 2023; Dai et al. 2022; Akyürek et al. 2022)	Constructive equivalences; matching synthetic-regression performance	Shows architectural expressiveness for learning algorithms	Demonstrated mainly on synthetic / linearized tasks; weak evidence that natural-task ICL implements these algorithms
Induction heads / pattern completion (Olsson et al. 2022)	Causal interventions; phase-change correlation; circuit-level identification	Explains sequence-continuation, repeated-token, and copy-style behaviors	Characterized mainly at small-to-mid scale; not a complete account of high-level ICL
Task vectors / task recognition (Hendel et al. 2023; Pan et al. 2023)	Transferable internal representations; ablation of recognition vs. learning components	Connects mechanism to compressed internal task encodings	Internal-representation level; does not specify the algorithm by which the task vector is constructed

These are not competitors in the way the popular framing sometimes implies. Bayesian inference and gradient descent are mathematical lenses on what an optimal predictor would compute; pattern completion and task vectors are claims about what a particular network does compute. They live at different levels of analysis. A transformer can simultaneously implement attention-based pattern completion at the circuit level and approximate Bayes-optimal latent-task inference at the input-output level, the same way a Newton's-method solver can simultaneously be doing arithmetic at the circuit level and quadratic optimization at the algorithmic level.

The article-level takeaway: no single mechanism has been empirically established as the universal explanation for what frontier LLMs do when they "learn in context." The honest summary is that ICL is plural at the mechanism level, with strong partial evidence for several accounts in their natural regimes and weak evidence for any of them as a universal claim.

4. Scaling: jagged, not monotone

The original GPT-3 result included a clean scaling observation: ICL performance improved with model size on many tasks. This observation was load-bearing for the early scaling-laws literature and for the broader argument that compute-driven scaling was the engine of capability gains. Subsequent work has substantially complicated the picture.

The emergence critique

Schaeffer, Miranda, and Koyejo 2023 ("Are Emergent Abilities of Large Language Models a Mirage?") argued that many reported emergent abilities — including some classic few-shot ICL results — disappear under continuous or partial-credit evaluation metrics and reappear only under harsh thresholded metrics (exact-match accuracy, multiple-choice top-1). On smoother metrics, the same tasks show steady, log-linear improvement with scale. The paper's claim is not that emergence is non-existent, but that the sharpness of many emergence curves is partly a metric artifact.

This matters for the ICL scaling story because it disciplines the language. "ICL emerges at scale" is true under harsh metrics on certain tasks; "ICL gradually improves with scale on a wide range of tasks" is the broader and more defensible claim.

Many-shot ICL

The arrival of million-plus-token context windows around 2024 changed the empirical question. With short contexts, "few-shot" meant 1-32 examples. With long contexts, hundreds or thousands of examples can fit. Agarwal et al. 2024 ("Many-Shot In-Context Learning") at Google DeepMind showed substantial gains from many-shot prompting across mathematical reasoning, machine translation, summarization, and code tasks — in some cases closing or reversing the gap to supervised fine-tuning, particularly on tasks with large label spaces.

Bertsch et al. 2025 ("In-Context Learning with Long-Context Models: An In-Depth Exploration") provided a controlled study across model families, tasks, and context lengths. The headline findings: many-shot ICL often continues to improve at thousands of examples; gains are largest where the label space is large; and beyond a task-dependent saturation point, additional examples plateau or — in some configurations — actively hurt.

Golchin et al. 2025 ("Memorization vs. Generalization: Quantifying Data Leakage in LLM Personalization") and related selection/caching work showed that which examples are used and how they are ordered matters increasingly as N grows, and that compute-efficient many-shot ICL is a substantive engineering problem rather than a "just put more examples in" optimization.

Plateau and degradation regimes

Not all tasks scale with examples. Oskooei et al. 2025 ("In-Context Learning for Code Translation") found that functional correctness on code-translation tasks peaks in the low-tens of examples and degrades with very large prompts, attributing the curve to demonstration interference and context-noise effects. Low-resource translation studies, including the late-2025 / early-2026 multilingual work, show mixed scaling: language pairs with abundant pretraining coverage benefit from few examples and plateau quickly; pairs with sparse coverage sometimes benefit from many-shot prompting up to large N and sometimes degrade as the prompt's distribution drift exceeds the model's ability to anchor on it.

The order- and format-sensitivity literature — Zhao et al. 2021 ("Calibrate Before Use") and Lu et al. 2022 ("Fantastically Ordered Prompts and Where to Find Them") — shows that within a single model, accuracy can swing significantly based on demonstration order alone. Random reorderings of the same few-shot prompt have been shown to produce accuracy spreads of 30+ percentage points in extreme cases. Calibration techniques (output-distribution adjustments to counteract majority-label bias) partially mitigate this, but the underlying instability is not a metric artifact: it is a property of how the model arbitrates between demonstration content and pretraining priors.

The honest scaling summary

The defensible scaling claim is therefore not "more scale → more ICL → more examples → more performance" as a monotone law. It is something more like:

• ICL ability scales with model size on most tasks, with smoother curves under finer-grained metrics than the original "emergence" framing suggested.
• Many-shot prompting in long contexts produces substantial additional gains on large-label-space tasks and on tasks where the demonstrations meaningfully constrain the model's latent task distribution.
• These gains plateau and sometimes reverse on tasks where additional demonstrations introduce noise, conflict with the model's priors, or exceed the model's ability to integrate over long contexts.
• Within any given (model, task) combination, exemplar choice, order, format, and label-space specification continue to matter, sometimes dominating the effect of N.

That curve is jagged enough to undercut any version of the article's question that asks "does ICL scale?" as a yes/no proposition.

5. Post-training: how instruction tuning changes the picture

The original ICL literature was about base models. By 2026, almost all deployed models are heavily post-trained: instruction-tuned, RLHF'd, DPO'd, or some combination. The question of how post-training affects ICL is both practically central and conceptually slippery.

The popular framing — instruction-tuned models have weaker ICL — is not quite right. It originates in a real observation but compresses too aggressively.

What the evidence actually supports is a tradeoff. Instruction tuning, in nearly all its forms, increases the model's tendency to follow natural-language instructions and apply strong semantic priors to label vocabularies, and decreases its willingness to follow context-supplied mappings that conflict with those priors. On a diagnostic like flipped-label classification, instruction-tuned models often underperform their base counterparts: they refuse to follow the flipped mapping because their post-training has reinforced "positive means positive." On a diagnostic like ordinary semantic classification with natural labels, instruction-tuned models often outperform: they recognize the task more reliably from a brief description.

Wei et al. 2023 ("Larger language models do in-context learning differently") is the clearest study of this tradeoff. It shows that larger models, particularly post-trained ones, can override their semantic priors and follow flipped or semantically-unrelated label mappings when prompted with enough examples — but smaller post-trained models often cannot. The gradient is: small base models do format/prior arbitration; small instruction-tuned models do mostly prior-driven prediction; large base models can override priors with moderate evidence; large instruction-tuned models can override priors with sufficient evidence, but the threshold is higher.

Irsoy et al. 2025 ("Partial Instruction Tuning") reported that deliberately reducing the strength of instruction tuning — applying it to fewer parameters or for fewer steps — measurably improved few-shot ICL performance while sacrificing AlpacaEval and similar instruction-following benchmarks. This is the cleanest empirical demonstration of the tradeoff as a tradeoff: you cannot have both maximum instruction following and maximum context-override ICL simultaneously, at least not with current post-training pipelines.

Munjal et al. 2026 examined zero-shot chain-of-thought performance on out-of-distribution math problems and found that base models sometimes outperform their instruction-tuned counterparts when the task is sufficiently dissimilar from the instruction-tuning distribution. This is consistent with the priors-vs-override picture: instruction tuning narrows the distribution of contexts the model handles well, which is excellent for the intended use cases and worse for everything else.

The implication for the article-level question is that the "weaker pure-ICL" framing should be replaced with something like:

Post-training reweights the balance among ICL's sub-behaviors. It strengthens semantic-prior use and instruction-following sub-behaviors; it weakens context-override sub-behaviors, particularly for arbitrary or counter-prior mappings; and it has heterogeneous effects on task recognition depending on whether the task is in or out of the post-training distribution.

This is the right way to read the system-card observations that instruction-tuned models often "feel" less responsive to careful few-shot prompts. They are not failing to learn from examples; they are following a different policy about how to weight examples against instructions and priors.

The product-level consequence is that pure exemplar-based prompting has receded from many deployed user surfaces — not because the underlying capability has disappeared, but because instruction-following is a more legible user interface and because retrieval, tool use, and structured output schemas have absorbed many of the workloads few-shot prompting used to handle.

6. Chain-of-thought as structured in-context demonstration

Chain-of-thought prompting is often categorized as separate from ICL, but the relationship is closer than the separation suggests.

Wei et al. 2022 ("Chain-of-Thought Prompting Elicits Reasoning in Large Language Models") introduced few-shot CoT as a variant of standard few-shot prompting in which each demonstration includes not only an input and a final answer, but an intermediate reasoning trace. The reported effect was substantial on arithmetic, symbolic, and multi-step reasoning benchmarks: models that failed standard few-shot prompting succeeded with CoT exemplars.

Read as ICL, few-shot CoT is doing several things at once:

1. It is still a few-shot demonstration — the model is conditioning on input-output pairs in the prompt, in the standard ICL sense.
2. It is conditioning on a richer output shape — the demonstrated output is not just an answer but a trajectory, which shifts the model's generation distribution toward longer, more structured outputs.
3. It is allocating test-time compute — by generating a reasoning trace before committing to an answer, the model amortizes computation across more forward-pass tokens, which is increasingly understood as the dominant effect for many tasks.

Kojima et al. 2022 ("Large Language Models are Zero-Shot Reasoners") complicated the picture by showing that simply prepending "Let's think step by step" to a prompt — with no demonstrations at all — recovers much of the CoT benefit on many tasks. This is hard to reconcile with a pure ICL reading of CoT: if zero-shot CoT works, then the gains cannot be primarily about input-label mapping from demonstrations. They are about output-trajectory shaping and test-time-compute allocation, and the few-shot CoT prompt is partly serving as an instruction (produce reasoning then an answer) rather than as a training-set analogue.

This is one of the cleaner cases in which the ICL-as-learning framing actively misleads. Few-shot CoT is structured demonstration; zero-shot CoT is instruction-tuned trajectory shaping; both work; their mechanisms only partially overlap.

The subsequent CoT literature — Wang et al. 2022 on self-consistency, the test-time-compute scaling laws from 2024-2025, and the o1/o3-style reasoning-model training regimes — has shifted the conceptual center of gravity. CoT is now best understood as one technique in a broader family of test-time compute methods, in which the model is induced (by demonstration, instruction, or training) to spend more forward-pass tokens on intermediate computation before producing a final answer. Few-shot CoT remains a particular implementation of that idea using ICL as the conditioning mechanism.

A related question — are CoT traces faithful representations of the model's reasoning? — is mostly orthogonal to the ICL story but worth flagging. Turpin et al. 2023 and subsequent work have shown that CoT explanations can be systematically unfaithful: the model can produce a confident trace that does not reflect the actual causal computation behind its final answer. This does not undermine CoT's performance benefits, but it disciplines claims about what CoT reveals about model reasoning.

7. Engineering implications

The empirical literature on what makes prompts work has accumulated enough to support a small set of robust generalizations, plus a much larger set of context-dependent heuristics that should be treated as heuristics.

Robust findings

• Format matters. Min et al. 2022, Zhao et al. 2021, and many follow-ups establish that prompt format — delimiter choice, label vocabulary, output shape, presence/absence of natural-language description — has effects that often exceed those of exemplar correctness on classification-style tasks.
• Order matters. Lu et al. 2022 and replications show that the order of demonstrations can move accuracy by tens of percentage points, especially on smaller models and on tasks with imbalanced label distributions.
• Calibration matters. Zhao et al. 2021 showed that majority-label bias and recency bias in few-shot prompts can be partially corrected by adjusting the output distribution. Calibration is a real technique, not a curiosity.
• Label-space specification dominates label correctness on some tasks. Min et al. 2022's central finding, properly scoped, is that demonstrations primarily communicate the set of valid answers and the output format. This is robust on classification with constrained label spaces and weaker on open-ended generation.
• Exemplar similarity to the test input helps. Retrieval-based selection of demonstrations — picking demonstrations that are semantically similar to the query — usually outperforms random selection, particularly for tasks with high input variance.

Context-dependent heuristics

• More examples are better — true on average, false in detail. Plateau and degradation regimes are real (Bertsch et al. 2025; Oskooei et al. 2025).
• Diverse examples are better than redundant ones — sometimes true, sometimes the opposite when redundancy reinforces format.
• Put the most important example last — folklore, weakly supported, model-dependent.
• Add a one-line task description before examples — generally helpful for instruction-tuned models, less so for base models.

When ICL wins

The decision frame for when to use ICL versus alternatives — fine-tuning, retrieval-augmented generation, tool use, or full structured-output schemas — is now reasonably stable:

Approach	Wins when...	Loses when...
ICL (few-shot or many-shot)	Adaptation must be fast, reversible, cheap to prototype, or session-local; the task fits in context; deployment doesn't justify training cost	Task is high-volume and stable; demonstrations are long or expensive; behavior must be guaranteed across distribution shift
Fine-tuning	Task is stable, high-volume, and worth amortizing; the desired behavior is well-defined enough to encode in weights	Task evolves frequently; behavior must be reversible per-session; data is too sparse to fine-tune robustly
RAG / retrieval	Adaptation requires factual knowledge not in pretraining; updates are frequent; provenance matters	The bottleneck is procedural rather than factual; retrieved context dominates token budget
Tool use	The task decomposes into operations a tool can perform exactly (computation, search, code execution); the model's role is orchestration	The task is irreducibly linguistic; tool latency or cost dominates
Structured output schemas (JSON mode, constrained decoding)	Output format is the principal constraint; correctness is verifiable post-hoc	The schema cannot capture the relevant variability

These boundaries are not crisp. A modern production system typically uses several of these in combination — RAG to retrieve grounding documents, a structured output schema to constrain format, instruction-following for the high-level task, and few-shot demonstrations for edge cases. In that pipeline, ICL has not disappeared; it has become one technique among several, often invoked for the parts of the task that resist other forms of specification.

This is the strongest case against any reading in which ICL is "fading." At the user-interface level, pure exemplar prompting has receded — most users of ChatGPT or Claude do not write few-shot prompts. At the system-builder level, ICL remains foundational, and the line between "in-context examples" and "retrieved documents" and "tool results" is essentially a layout decision about how to populate the same conditioning surface.

8. The open question: fundamental property or transient capability?

The question the original prompt frames — is ICL a fundamental property of attention-based models, or a transient capability that recedes as models become more instruction-following? — is, on close inspection, several questions in a trench coat.

There is an architectural version of the question: does attention, as a primitive, naturally support context-conditioned computation in a way that other architectures (recurrent networks, state-space models) do not? The mechanistic interpretability work on induction heads gives a partial answer: yes, attention supports certain kinds of context-driven computation cleanly, and several of ICL's sub-behaviors are well-explained by attention circuits. But more recent state-space and hybrid architectures (Mamba, Jamba) also exhibit ICL, which weakens the strong-attention-specificity reading. The architectural reading is probably: transformers acquire ICL most cleanly and most studied, but it is not exclusive to attention.

There is a pretraining version of the question: is ICL a property of any sufficiently large autoregressive model trained on coherent text, or is it specific to certain training distributions? Xie et al.'s analysis suggests the former: any next-token predictor trained on documents with coherent latent structure should acquire something ICL-shaped. Empirically, this seems to hold across pretraining corpora and architectures, with quantitative differences. The pretraining reading is probably: ICL is a robust consequence of scale-plus-coherent-text, not a one-off discovery.

There is a post-training version of the question: as models are post-trained more heavily — RLHF, DPO, constitutional AI, reasoning-model training — does the pure ICL surface (responsiveness to demonstrations, override of priors by exemplars) shrink? The evidence above (Wei 2023, Irsoy 2025, Munjal 2026) suggests yes: post-training rebalances the sub-behaviors away from exemplar-override and toward instruction-following and prior-driven prediction. But this is a policy shift, not a capability loss. Partial instruction tuning recovers exemplar responsiveness at the cost of instruction-following polish, which means the capability is still there in the weights.

There is a user-interface version of the question: do users and product designers still need to write few-shot prompts? The honest answer is: less than they used to. Instruction-tuned chat is the default interface; few-shot prompting is now an advanced technique that production engineers use when other approaches under-perform. This is the strongest version of the "ICL is receding" reading, but it is a claim about user-facing patterns, not about model capability.

There is a system-architecture version of the question: is the conditioning-on-context primitive that ICL exemplifies central to how LLM-based systems are built? The answer here is unambiguous: yes, and increasingly so. Long context windows, RAG, agent scratchpads, tool-use traces, memory systems, and chain-of-thought reasoning are all variations on put structured information in the context window and let the model condition on it. ICL is the primitive of which these are all specializations.

So the four-way decomposition gives roughly:

Level of analysis	Is ICL fundamental?
Architectural	Strong yes for transformers; somewhat-yes for other competent sequence models
Pretraining-level	Yes — robust consequence of scale + coherent pretraining
Post-training-level	Reshaped but not removed; tradeoff with instruction-following
User-interface-level	Receding — fewer end-users write few-shot prompts
System-architecture-level	Increasingly central — conditioning-on-context is the dominant primitive

The single yes/no answer the original question demands is therefore underdetermined: it depends on which level of analysis the questioner has in mind. The most defensible synthesis is:

ICL is a robust architectural and pretraining-level capability that has been substantially absorbed into instruction-following, retrieval, tool use, and structured output at the user-interface and product levels — without ceasing to be foundational at the system-architecture level. The "pure" version of ICL (handcrafted few-shot prompts overriding semantic priors) has receded as a primary user-facing technique; the deeper primitive (forward-pass adaptation to structured context) has become more central, not less.

That synthesis is consistent with the empirical record. It is also untidy enough to resist the heroic-discovery narrative that ICL is sometimes given. The honest history is messier: GPT-3 surfaced a behavior that turned out to be a family of behaviors; a decade of theoretical and mechanistic work has clarified parts of the family without producing a single unifying account; the engineering practice that grew around ICL has matured and partly outgrown its origin; and the underlying primitive has quietly become the most important interface in deployed AI systems.

9. What would change this picture

The strongest pieces of evidence that would shift the picture meaningfully in one direction or the other are:

• Toward "ICL is one mechanism": A causal mechanistic study at frontier scale that isolated a stable circuit (or small family of circuits) explaining the majority of ICL behavior across natural-task classes, both few-shot and many-shot, and predicting scaling curves better than format/prior/retrieval baselines. As of mid-2026, no such study exists.
• Toward "ICL is receding": Matched-backbone studies showing that progressively stronger post-training consistently flattens exemplar-response curves across natural and synthetic tasks, without compensating gains elsewhere. Partial evidence exists (Irsoy 2025; Munjal 2026); a comprehensive longitudinal study across model generations does not.
• Toward "the scaling story is monotone": A broad, peer-reviewed synthesis of 2024-2026 many-shot and long-context evaluations across model families and task classes showing a stable monotonic ICL scaling law. The current literature does not support this; the curves remain jagged.
• Toward "the training-set analogy is right": Controlled benchmarks where ICL gains survive adversarial balancing of format, label space, and semantic priors — i.e., where the model is demonstrably learning the input-label mapping rather than performing format/prior arbitration. Partial evidence in flipped-label studies; not yet a settled benchmark.

The article-level honest position is that none of these decisive experiments has been run at sufficient breadth and quality, and that the field's best current understanding is plural, partial, and conditional. ICL is real, it is consequential, and it is not yet fully theorized. Most of what we know about it comes from carefully scoped studies whose conclusions do not extend automatically to frontier-scale models on natural tasks; most of the strong public claims about its mechanism overstate the evidence in one direction or the other; and most of the engineering practice that surrounds it works for reasons that the formal accounts have not fully captured.

That state of affairs is uncomfortable for a field that prefers clean narratives. It is also probably the correct epistemic posture to maintain until the relevant experiments are done.

Companion entries

Core theory:

• Implicit Bayesian Inference
• Gradient Descent as a Meta-Optimizer
• Induction Heads
• Task Vectors
• Task Recognition vs Task Learning

Empirical phenomenon:

• GPT-3 and the Few-Shot Surprise
• Many-Shot In-Context Learning
• Demonstration Order Sensitivity
• Prompt Calibration
• Emergent Abilities and Metric Artifacts

Mechanistic interpretability:

• Attention Circuits
• Activation Patching
• Mechanistic Interpretability of Transformers
• Circuit-Level Phase Changes During Training

Practice:

• Prompt Engineering
• Few-Shot Exemplar Selection
• Retrieval-Augmented Demonstrations
• Chain-of-Thought Prompting
• Zero-Shot Chain-of-Thought
• Self-Consistency Decoding
• Test-Time Compute
• Structured Output Schemas

Post-training:

• Instruction Tuning
• RLHF and Behavioral Priors
• Partial Instruction Tuning
• Semantic Priors vs Context Override

Scaling:

• Scaling Laws
• Long-Context Models
• Context-Window Economics

System architecture:

• Retrieval-Augmented Generation
• Agent Scratchpads
• Tool Use in LLM Systems
• Multi-Model Agent Orchestration Patterns

Counterarguments and limits:

• The Training-Set Analogy as Misnomer
• Unfaithfulness of Chain-of-Thought Traces
• Benchmark Contamination
• Format-Driven Apparent Learning