Process Supervision vs Outcome Supervision: A Training-Signal Distinction

Process supervision rewards the intermediate steps of a model's reasoning; outcome supervision rewards only the final answer or trajectory return. The distinction is narrow as a definition but consequential in practice: it changes credit-assignment density, label-acquisition cost, the shape of the reward model that consumes the signal, and — contested — what kind of oversight the resulting system supports. This article describes the distinction, anchors it in the Lightman et al. 2023 math result, separates it from the adjacent objects it is often conflated with (process reward models, verifiers, chain-of-thought monitorability), traces the post-2023 evidence on whether process signals generalize beyond verifier-rich domains, and treats the safety question as open rather than settled.

Coverage note: verified through May 2026.

1. The core distinction

In a reinforcement-learning or reward-model framing, "supervision signal" is the function the model is trained to maximize or that a learned evaluator is trained to predict. The granularity of that signal is the central design choice this article concerns.

Signal type	What it scores	Typical label unit	Typical use
Outcome supervision	The final answer, completed program, or trajectory return	One label per attempt	Train a policy by RL on terminal reward; train an outcome reward model (ORM) for selection or RLHF
Process supervision	Intermediate reasoning steps as the trace unfolds	One label per step or per segment	Train a process reward model (PRM); provide dense reward for RL; rerank candidates step-by-step at inference

A worked example clarifies why the granularity matters. Consider a multi-step math solution that performs four correct algebraic manipulations and then makes an arithmetic slip on the fifth, producing the wrong final answer. An outcome-supervised reward model sees only the final number and assigns negative reward to the entire trajectory, including the four correct steps. A process-supervised reward model can assign positive reward to the four correct steps and negative reward to the fifth. The credit-assignment problem — which token, which decision, which subgoal caused the eventual outcome — is what process supervision attempts to make local.

This is the entire formal distinction. Everything else — what model consumes the labels, how the labels are produced, whether the resulting evaluator is used for training or only for inference-time selection, what the supervision implies for safety — is downstream and should be treated separately.

1.1 Adjacent objects that are not the same thing

Public discussion frequently collapses several distinct artifacts under "process supervision." The article uses the following definitions throughout, and treats conflation between them as the most common analytical error.

Term	Definition	What it is not
Outcome supervision	A signal granularity: reward only at trajectory end	Not a claim about whether the trajectory is short or long
Process supervision	A signal granularity: reward at intermediate steps	Not specific to any particular evaluator or training algorithm
Outcome reward model (ORM)	A model trained to score completed solutions	Not the only consumer of outcome signals; RL on rule-based final-answer checks also counts
Process reward model (PRM)	A model trained to score partial traces or steps	Not identical to "trained with process supervision" — a PRM can be derived from automated rollouts, not human step labels, and a process-supervised dataset can be used without training a PRM
Verifier	A function — rule-based, executable, or learned — that judges correctness	Not necessarily process or outcome; depends on what it is asked to judge
Chain-of-thought (CoT)	The visible intermediate text a model emits while reasoning	Not the same as process supervision; CoT can exist without any per-step training signal
Chain-of-thought monitorability	The deployment-time property that visible reasoning is informative about model intent	Not implied by process-supervised training; faithfulness must be argued separately

Several common claim-shapes become defensible only after these are kept distinct: "this PRM was trained on automated step labels and then used to rerank candidates at inference" is a precise sentence; "this model was trained with process supervision" is, on its own, ambiguous about the granularity of labels, their provenance, and their use phase.

2. The Lightman et al. 2023 result

The empirical anchor for this topic is Lightman et al. 2023, "Let's Verify Step by Step". The paper compares outcome- and process-supervised reward models on the MATH benchmark, releases the PRM800K dataset of approximately 800,000 step-level human labels on generated solutions, and reports that the process-supervised reward model substantially outperformed the outcome-supervised baseline at solution selection — the headline figure was 78% on a representative MATH subset, drawn by best-of-N selection over a generator. At fixed compute budgets for the generator and selector, process supervision was the better signal for the selection task on MATH.

Three details of the experiment matter for any general conclusion drawn from it.

The labels were expensive and human-produced. PRM800K's step-level annotations were produced by hired contractors with mathematical training. The cost is not a footnote: it is the central reason later work focused on automating step labels.

The comparison was at fixed compute, not at fixed cost. The compute budget for training the reward model and the inference budget for best-of-N selection were held constant. Annotation cost was not, and could not have been, equalized — process labels are intrinsically denser per problem.

The domain was math. MATH is a difficult benchmark, but it is one in which final answers are easy to verify, intermediate steps are syntactically explicit, and "step" has a meaningful unit. None of these properties are guaranteed in code, agentic planning, scientific reasoning, ethical deliberation, or open-ended writing.

A useful boundary condition is Uesato et al. 2022, "Solving math word problems with process- and outcome-based feedback", an earlier DeepMind paper on GSM8K. Uesato et al. found that outcome-based supervision could match process-based supervision on final-answer accuracy, but that process supervision (or learned feedback that approximated it) was needed to reduce reasoning errors among solutions that happened to reach the correct final answer. That nuance — outcome supervision can be enough when only the final answer matters; process supervision becomes important when the trace itself is the object of interest — is the cleanest pre-2023 statement of the trade-off and remains a useful frame.

The headline that propagated through the field after Lightman et al. — "process supervision beats outcome supervision" — should therefore be read with three scope conditions: on MATH, at fixed compute, with expensive human step labels.

3. Process supervision vs process reward models

Lightman et al. do two things at once: they construct a process-supervised dataset (PRM800K) and they train an evaluator on it (a PRM). The two are separable, and most of the follow-up literature targets one or the other rather than both.

A process-supervised dataset is a collection of (trace, step-level labels) pairs. It can be used to:

train a PRM that scores partial traces, used downstream for selection, search guidance, or as a dense reward signal during RL;
directly fine-tune a generator via supervised learning on annotated traces, biasing the generator toward step patterns that received positive labels;
feed an RL training loop in which the per-step labels become per-step rewards (or, more often, are aggregated into discounted returns).

A PRM, conversely, can be trained on:

direct human step labels (the Lightman et al. recipe);
automatically generated step labels derived from rollouts that succeed or fail;
model-generated critiques in which a separate model judges each step;
progress signals derived from advantage functions that compare a partial trace to expected continuations.

The categorical question — "is this work an instance of process supervision?" — depends on which axis you are asking about. A paper that trains a PRM from rollout-derived labels and then uses it only for inference-time best-of-N selection has not "trained the model with process supervision" in the policy sense; it has used process-style information at evaluation time. A paper that uses a learned PRM as the dense reward signal in an RL fine-tune of the policy has done something closer to process-supervised policy training, but the labels are still automated rather than human, and the policy is supervised by a learned evaluator rather than directly by step labels.

The article's working convention is to describe each system along four axes — signal granularity, label provenance, target model, use phase — rather than relying on the umbrella term.

4. Label economics

The labeling cost is the single largest practical constraint on process supervision in its original form, and is the proximate reason most post-2023 work moved to automated label sources.

PRM800K's roughly 800,000 step labels were produced over months of contractor work and required mathematically literate annotators. Per-step costs are difficult to extract from the paper, but the economics imply that scaling the same approach to (a) larger problem sets, (b) longer or more complex traces, or (c) domains requiring higher annotator expertise (theorem proving, medicine, law) is prohibitive at frontier-lab scale. Even in math, the dataset is small relative to the volume of generated traces a frontier RL pipeline consumes per training run.

Three secondary costs are easy to underestimate.

Annotator disagreement. Step labels are not always crisp. In a long math derivation, a step that is locally valid may be strategically wrong (committing to an unproductive line). Annotators disagree on whether such steps deserve negative labels, and the disagreement is often unrecoverable without much longer instructions. Lightman et al. acknowledge calibration work on label conventions; later work that scales labeling further generally reports growing inter-annotator variance.

Domain expertise. Math is unusually friendly because most graduate-level mathematicians can label most steps in MATH problems. A code dataset requires programmers; a medical-reasoning dataset requires clinicians; a legal-reasoning dataset requires lawyers. The labor pool shrinks rapidly as expertise demands rise, and the cost rises with it.

"Step" is not always a defined unit. In a math solution, a step is roughly a line of derivation. In a tool-using agent, "step" might mean a tool call, a thought, or a sub-plan, and step boundaries are often arbitrary. In free-form natural language, step segmentation requires its own annotation policy. Where step boundaries are arbitrary, per-step labels are a less coherent object than the headline word "process supervision" suggests.

5. Automated process supervision

The dominant post-2023 research direction is to replace human step labels with labels derived automatically — from rollouts, from auxiliary models, or from formal verifiers. This is the path frontier-scale work has taken because human labeling does not scale; it is also the path where the epistemic status of the labels changes and must be tracked.

5.1 Rollout-derived labels (Math-Shepherd, OmegaPRM)

Math-Shepherd (Wang et al. 2024) trains a PRM by generating many continuations from each partial trace and labeling steps according to the empirical success rate of completions that pass through them. A step that, on average, leads to correct final answers is labeled positively; a step that consistently leads nowhere is labeled negatively. The labels are entirely automatic given a final-answer verifier and a rollout budget.

OmegaPRM (Luo et al. 2024) generalizes this with Monte Carlo Tree Search over partial traces, generating roughly 1.5 million automatically annotated step labels for math PRM training. The cost shifts from human annotation to inference compute for rollouts, and the label distribution becomes a function of the generator's behavior rather than of human judgment.

The trade-off is direct. Automated rollout labels can be produced at scale and updated as generators improve, but they confound several things: the quality of the step (would a different generator have recovered?), the strength of the verifier (does success here mean the final answer was actually right, or merely matched?), and the search procedure (steps in shallow expansions are labeled differently from steps in deeply explored branches). A PRM trained on rollout-derived labels learns a reward function that is approximately "this partial trace tends to succeed under this generator on this distribution." That is useful, but it is not equivalent to "this partial trace contains correct reasoning."

5.2 Model-generated step judgments (AutoPSV, generative PRMs)

AutoPSV (NeurIPS 2024) and related work use an answer-trained verifier or a separate critic model to infer step-level labels without rollouts: the critic reads the step and judges its correctness directly. This generalizes beyond domains with cheap rollout-based final-answer checks, including commonsense tasks where answers can be verified but rollouts are expensive.

GenPRM (Zhao et al. 2025) reframes the PRM itself as a generative reasoner that produces a critique of each step, rather than as a scalar regressor. The generative-PRM approach narrows the gap between "evaluator" and "policy" — the PRM is now a model that reasons about reasoning — but introduces correlated-error risk: if the critic and the generator share a base model, their blind spots will overlap, and supervision can amplify shared failure modes rather than correcting them.

5.3 Progress-based and advantage-based process rewards

Setlur et al. 2024, "Rewarding Progress" reframes process rewards as a measure of progress toward the eventual answer — closer in spirit to advantage estimation than to step-correctness classification. Under this framing, a step that increases the expected return relative to the previous state receives positive reward; a step that decreases it receives negative reward. The connection to RL theory is direct, and Jia, Rakhlin, and Xie 2025 argue formally that, under standard coverage assumptions, advantage-derived process rewards are theoretically optimal and that outcome-supervised RL is not necessarily statistically harder than process-supervised RL when verifier or rollout access is available.

This theoretical result is important because it weakens any general claim that process labels are intrinsically more informative than outcome labels. The empirical advantage that Lightman et al. demonstrated at fixed compute may, on this view, reflect algorithmic and engineering limitations of outcome-only training rather than a deep statistical asymmetry. Post-2023 progress in outcome-verifier RL (see §7) is consistent with this reading.

5.4 PRM benchmarks and the limits of current process evaluators

The empirical question for automated PRMs is whether they actually identify step errors competently. ProcessBench (Zheng et al. 2024) and PRMBench (Song et al. 2025) test PRMs on the task of localizing the first error in a math solution. Both find that current PRMs — including those from frontier labs — exhibit significant weaknesses: they over-trust plausible-looking steps, disagree on what counts as the first error, and can be led astray by surface fluency. Socratic-PRMBench (2025) extends the evaluation to dialogic step verification and reaches similar conclusions.

Qwen's "Lessons of Developing Process Reward Models in Mathematical Reasoning" (Zhang et al. 2025) is unusually candid about the difficulty: the team reports that naive PRM training pipelines suffer from severe label noise when scaled with rollout-based methods, and that the PRMs they ship do not always match outcome-only methods on downstream task performance after careful tuning. The paper is required reading for anyone treating "we trained a PRM" as a clean signal of empirical strength.

6. Generalization beyond math

The cleanest evidence for process supervision is concentrated in math. The honest summary of the post-2023 literature is that "promising extensions exist in adjacent domains, with the strongest evidence still in domains where intermediate or final correctness can be checked."

Code. Wang et al. 2025, "From Mathematical Reasoning to Code" and related work transfer rollout-style PRM training to programming. Code is well-suited to automation: execution is the verifier, intermediate compilation or test-pass states give natural step boundaries, and rollout costs are tractable. Reported gains are real but modest, and most pipelines combine PRMs with execution-based outcome rewards rather than replacing them.

Multimodal reasoning. Multimodal PRM work (e.g., on math problems with diagrams, visual question answering with reasoning chains) inherits both the benefits and the weaknesses of textual PRM work, with the additional complication that step boundaries in multimodal traces are even more arbitrary.

Agents. Agent-task work — web navigation, tool use, multi-step API calls — has produced a cluster of "agent PRMs" that score subtask completion or action plausibility. The challenge is that agent traces are long, branching, and full of recoverable mistakes; whether step labels generalize across task distributions remains poorly studied. AgentPRM-style systems have demonstrated improvements on specific benchmarks but have not produced a general result.

Open-ended domains. The article's load-bearing limit is here. Process supervision in policy reasoning, scientific writing, philosophical argument, ethical deliberation, and most safety-relevant deployment settings has not been demonstrated to provide robust gains over outcome-only training, because (a) "correctness" of intermediate steps is contested, (b) annotation requires expensive expert judgment, and (c) automated judges in these domains are themselves unreliable. Claims that process supervision generalizes broadly should be received skeptically until supported by matched-cost ablations in non-verifiable domains.

7. Reasoning models: what their reports actually say

A common shorthand asserts that "most current reasoning models are trained on process-style signals." This formulation collapses several distinct things — long chain-of-thought generation, RL on verifiable rewards, learned PRMs, distillation from reasoning traces, and process-style cold-start data — and overstates what public technical reports actually disclose.

The defensible summary is: many current reasoning systems are trained to generate and improve extended reasoning traces, and the supervision signal in published descriptions is some combination of human process labels, synthetic process labels, outcome-verifiable rewards, model-generated critiques, and undisclosed mixtures. The proportions vary by lab and are partially or wholly redacted in most reports.

7.1 OpenAI o-series and the "thinking" family

OpenAI's o1 system card and follow-up reports describe large-scale RL that teaches models to reason through chains of thought, with the chain-of-thought treated as a first-class object subject to optimization. OpenAI has stated publicly that the chain-of-thought is part of the training target and that the company does not directly optimize for the rendered chain-of-thought to remain monitorable, on the explicit grounds that doing so could teach the model to hide intent (OpenAI: "Chain-of-thought monitorability"). The reports do not disclose enough detail to determine whether the supervision is best characterized as process-supervised, outcome-supervised, or hybrid; the language is process-adjacent without committing to any particular formal recipe.

7.2 DeepSeek-R1: outcome-supervised RL as a counterweight

DeepSeek-R1 (Guo et al., Nature 2025) is the most transparent counter-evidence to a strong "process supervision is necessary" claim. The paper reports that R1 was trained on math, coding, and logic tasks using rule-based accuracy rewards and format rewards, and explicitly says the team avoided neural reward models — outcome-based or process-based — for the reasoning RL stage, because of reward-hacking concerns. The recipe relied on hard reasoning problems, reliable rule-based verifiers, and large RL compute, and produced a model whose reasoning behavior emerged without any neural PRM and without large-scale human step annotation.

This does not refute Lightman et al. and it does not show that outcome supervision is sufficient in general. The R1 reasoning RL stage relied on domains with cheap, reliable rule-based verifiers. The paper also documents a mixed pipeline (R1-Zero is the pure outcome-RL model; the released R1 includes cold-start data and additional supervised stages). What R1 does refute is the stronger claim that human process labels or learned PRMs are necessary for strong frontier reasoning in verifiable domains. It is the cleanest existing existence proof that an outcome-verifier RL pipeline can reach frontier reasoning performance without process-supervised reward modeling.

7.3 Kimi k1.5: long-context RL without explicit PRMs

Kimi k1.5 (Moonshot AI 2025) reports a different recipe: long-context RL with a focus on policy optimization, without MCTS, without value functions, and without process reward models in the conventional sense. The report is informative as evidence that competitive reasoning can be reached without the PRM stack, but it is more partial than R1 about its full data and reward composition. Kimi should be read as evidence that "no PRM" is a viable architectural choice, not as a clean falsification of process supervision.

7.4 Anthropic, Qwen, and others

Anthropic's published work on chain-of-thought faithfulness (Turpin et al. 2023; Lanham et al. 2023) is methodologically adjacent rather than a process-supervision recipe. The faithfulness work bears on the safety question (§9) but does not describe Anthropic's reasoning-model training in process-supervision terms. Qwen's PRM work (cited above) is unusually candid about PRM difficulties but does not disclose the production training pipelines for its frontier models.

The honest conclusion from the reasoning-model literature is therefore: the field is converging on extended reasoning traces as a training and inference object, the supervision signals used to elicit and shape those traces vary substantially across labs, and the public record is too partial to support claims about which signal type is dominant in deployed frontier systems.

8. The theoretical picture: is process supervision intrinsically better?

The empirical advantage Lightman et al. report invites the inference that dense credit assignment is intrinsically superior to sparse outcome feedback. Theoretical work suggests this inference is not safe.

Jia, Rakhlin, and Xie 2025 prove that, under standard coverage assumptions, outcome-supervised RL is not statistically harder than process-supervised RL, and that advantage functions can serve as optimal process rewards when verifier or rollout access is available. The result reframes the empirical gap as algorithmic rather than informational: the issue is not that outcome labels carry less information than process labels, but that the algorithms which consume sparse feedback may be less efficient than those which consume dense feedback under current engineering constraints.

This matters for two reasons. First, it predicts that improvements in outcome-RL algorithms (better exploration, better critic estimation, better use of rollouts) should narrow the empirical gap — and the post-2023 success of rule-reward RL pipelines like DeepSeek-R1's is consistent with that prediction. Second, it suggests that what looks like a debate between "process supervision" and "outcome supervision" is partly a debate about how cheaply and reliably we can derive dense signals from sparse outcomes, which is closer to a question about verifier and rollout quality than about supervision philosophy.

The theoretical result does not mean process supervision is useless. It means the case for process supervision rests on engineering, cost, and oversight grounds rather than on a deep informational asymmetry. The strongest pro-process arguments are now: (a) per-step credit assignment is easier to implement than dense reward derivation from terminal signals when terminal verifiers are unreliable; (b) per-step evaluators expose where in a trajectory things go wrong, which has debugging and interpretability value; (c) per-step training pressure may shape reasoning quality in ways that outcome-only training does not.

9. The safety question

The framing that "process supervision is safer than outcome supervision because it makes reasoning visible" is widespread and partly defensible, but the article treats it as unsettled. The evidence supports a narrower claim: process supervision can change which oversight strategies are available, but it does not by itself produce monitorable, faithful, or deception-resistant reasoning.

9.1 What process supervision plausibly helps

Process supervision can localize errors. An evaluator that scores each step can flag the first place a trajectory goes wrong, which is informative for both debugging and runtime monitoring. Outcome-only signals cannot do this directly.

Process supervision can train policies whose intermediate reasoning is more legible — not because labels enforce legibility, but because reward pressure on intermediate steps tends to produce intermediate steps that look like the kind of reasoning the labelers approved. This is useful where intermediate legibility is desired.

Process supervision provides finer-grained handles for Scalable Oversight research. A PRM that scores steps gives oversight systems a per-step interface rather than a per-trajectory one, and several proposals for amplified oversight assume such an interface exists.

9.2 What process supervision does not, by itself, provide

It does not guarantee chain-of-thought faithfulness. Turpin et al. 2023 demonstrate that models can produce post-hoc rationalizations whose reported reasoning does not reflect the actual basis of their answer; Lanham et al. 2023 find that ablating chains of thought has variable effect on final answers, suggesting the chain is not always causally load-bearing. Process supervision optimizes for traces that look correct under the labeling scheme; whether those traces are causally responsible for the model's answer is a separate question that process supervision does not answer.

It does not eliminate reward hacking. Process labels can be hacked the same way outcome labels can, often more subtly: a model can learn to produce traces that pattern-match on what the PRM rewards while pursuing a different actual computation. The DeepSeek-R1 team cited reward-hacking concerns as their explicit reason for avoiding neural PRMs in the R1 reasoning RL stage; the concern is not hypothetical.

It can damage monitorability when applied naively. OpenAI's chain-of-thought monitorability statement is explicit on this point: directly optimizing the rendered chain-of-thought (as opposed to optimizing the final answer and treating the chain-of-thought as a side effect) risks teaching the model to hide intent in the chain-of-thought itself. Process supervision sits awkwardly across this distinction. If the PRM rewards "good-looking reasoning," the policy is being optimized for the rendered trace, and the monitorability concern applies.

9.3 Decomposing "safe reasoning"

The phrase "safe reasoning" packages several properties that should be assessed separately:

Safety property	Does process supervision plausibly help?
Final-answer accuracy in verifiable domains	Modest — outcome supervision with reliable verifiers can match it
First-error localization	Yes — this is a direct affordance
Auditability of completed reasoning	Yes, conditional on faithfulness
Faithfulness of chain-of-thought	No direct effect; potentially harmful if PRM optimization is naive
Resistance to reward hacking	No — process labels are hackable too
Corrigibility under distribution shift	Unstudied
Deception resistance	Unstudied; potentially harmful if visible reasoning is optimized

Articles and reports that issue a single safety verdict on process supervision typically conflate these. The defensible position is that process supervision provides useful affordances for some safety properties (localization, auditability) but is at best neutral and possibly counterproductive for others (faithfulness, deception resistance), and that the choice should be made per-property and per-domain rather than as a blanket recommendation.

10. A decision framework

The article's working recommendation, given the evidence as of May 2026, is that the supervision signal should be chosen based on four questions, in order:

Is the task finally verifiable, locally verifiable, both, or neither? If the final answer is cheaply and reliably checkable (math, code with tests, formal proofs, structured data extraction), outcome-verifier RL is a serious option and the burden of proof for adding process supervision falls on the additional value it provides. If intermediate steps are checkable but final outcomes are delayed or noisy (multi-step agents with intermediate observations, scientific reasoning with stage-wise evidence), process supervision is more attractive. If neither is reliably checkable (open-ended planning, ethical deliberation, policy analysis), process supervision based on automated judges should be approached with significant caution because the labels themselves are weak.

What is being trained? A policy via RL benefits from dense feedback when terminal credit assignment is hard; a reward model trained for selection or reranking benefits from process signals only insofar as the downstream selection task uses partial-trace information; a verifier for monitoring benefits from process signals if first-error localization is a deployment goal.

What is the label source? Human step labels are expensive and only feasible at limited scale, but their epistemic status is comparatively clear. Rollout-derived labels are cheap and scale with compute, but conflate step quality with generator behavior and verifier reliability. Model-generated critiques are cheap and scale with model capability, but introduce correlated-error risk between the generator and the critic. Each source produces a different reward function with different failure modes.

What is the safety claim? If the claim is "first-error localization for debugging," process supervision is well-matched. If the claim is "faithful, monitorable reasoning under deployment pressure," process supervision is not sufficient and may not be necessary; faithfulness must be argued and evaluated on its own terms.

Scenario	Default recommendation
Math or code at training time, reliable rule-based verifier	Outcome-RL with rule rewards as baseline; consider PRM if first-error localization is a goal or if rule rewards saturate
Long-horizon agent task with intermediate observable state	Hybrid: outcome reward at episode end, process signals from observable substates (not from a learned PRM unless validated)
Open-ended reasoning with no reliable verifier	Neither pure approach is well-supported; treat published claims with skepticism and prefer human evaluation over PRM-mediated training
Reranking at inference time	PRM-based reranking is competitive with ORM-based reranking on math; on other domains the evidence is thin
Safety monitoring at deployment	Process signals for first-error localization; do not rely on visible reasoning being faithful without separate evaluation

This framework will date. The post-2023 literature is moving fast in two directions at once — toward better automated process supervision and toward better outcome-verifier RL — and the relative balance may shift substantially within months. The framework's structure (decompose by verifiability, target, label source, and safety property) should outlast the specific recommendations.

11. Open questions

Several questions are genuinely unsettled and the article flags them rather than resolving them.

Does process supervision generalize to non-verifiable domains under matched cost? The cleanest test would be a controlled ablation: one base model, one inference budget, three reward variants (ORM on final-answer labels, PRM on a small expert-labeled step subset, automated PRM on rollout-derived labels), evaluated on math, code, and one non-verifiable reasoning domain with expert-labeled process errors. This study has not, to public knowledge, been run.

Are current PRMs reliable enough to trust as training signals? ProcessBench, PRMBench, Socratic-PRMBench, and Qwen's lessons paper all suggest the answer is "not robustly." A PRM that mislocates the first error in 30% of math solutions is not yet a reliable training signal, and using it as one risks training a policy to optimize the PRM's mistakes.

What is the actual supervision recipe in deployed frontier reasoning models? This is the question the article cannot answer. OpenAI's o-series, GPT-5 thinking variants, Anthropic's reasoning models, and most others publish high-level descriptions that are consistent with multiple supervision regimes. Until disclosure improves or until labs publish ablations, claims about "what frontier reasoning models use" should be confidence-tiered and source-attributed.

Is process supervision necessary for safe reasoning, or do outcome-verifier systems suffice? This is the question the article explicitly leaves open. The strongest pro-process argument is that intermediate feedback supports interpretability and oversight affordances that outcome-only training does not. The strongest pro-outcome argument is that reliable verifiers and large RL compute can elicit strong reasoning without neural PRMs (DeepSeek-R1) and that process labels introduce their own reward-hacking and faithfulness risks. The honest current state is that hybrid approaches dominate engineering practice, and that no single signal type has been shown to dominate on the full set of properties that "safe reasoning" actually packages together.

Does optimizing visible chains of thought damage monitorability? OpenAI's stated concern is that it does, which is the argument for keeping training pressure off the rendered trace. Process supervision that rewards step quality is, by construction, training pressure on the rendered trace. Whether the resulting damage to monitorability is significant in practice is an empirical question that has not been adequately studied in the open literature.

12. Synthesis

The honest summary is narrower than the prompting headline suggests. Process supervision is a real and useful training-signal granularity, with strong evidence in math at fixed compute (Lightman et al. 2023), an active and partially successful program in automated label generation (Math-Shepherd, OmegaPRM, AutoPSV, GenPRM, progress-based methods), measurable but limited reliability of current PRMs (ProcessBench, PRMBench, Qwen lessons), and a credible counter-thesis that outcome-verifier RL can reach competitive reasoning performance without neural PRMs (DeepSeek-R1). The theoretical case for an intrinsic informational advantage of process labels is weaker than the empirical headline implies (Jia, Rakhlin, Xie 2025), and the safety case for process supervision is more conditional than the framing of "monitorable reasoning" usually conveys.

Process supervision is neither obsolete nor proven necessary. It is one path through a design space in which the relevant axes — signal granularity, label provenance, target model, use phase, safety property — are partly orthogonal, and the right path depends on which combination of those axes a particular system targets. Articles and reports that issue a universal verdict on process versus outcome supervision are almost always overclaiming.

Companion entries

Core theory:

Reward Modeling
Reinforcement Learning from Human Feedback
Verifiable Rewards in Reinforcement Learning
Credit Assignment in Long-Horizon Tasks
Advantage Estimation and Process Rewards

Training methods:

Process Reward Models
Outcome Reward Models
Best-of-N Selection
Monte Carlo Tree Search for Reasoning
Rejection Sampling Fine-Tuning
Long-Context Reinforcement Learning

Reasoning models:

OpenAI o-Series Reasoning Models
DeepSeek-R1 Training Recipe
Kimi k1.5 Long-Context RL
Chain-of-Thought Generation
Inference-Time Reasoning Compute

Evaluation:

ProcessBench and PRMBench
MATH Benchmark
GSM8K and Word-Problem Reasoning
PRM800K Dataset

Scalable oversight and safety:

Counterarguments and limits:

Outcome-Only Reinforcement Learning with Verifiers
Statistical Equivalence of Process and Outcome Supervision
The Generalization Gap of Math-Trained Reasoners
Correlated-Error Risk in Self-Generated Supervision
Domain Verifiability and Reward Engineering