RLAIF: Reinforcement Learning from AI Feedback

Reinforcement Learning from AI Feedback (RLAIF) is a family of preference-optimization techniques in which the pairwise preference labels used to train a language model's reward signal — or its policy directly — are generated by another AI system rather than by human annotators. The strongest current evidence supports RLAIF as a method that can match Reinforcement Learning from Human Feedback (RLHF) on bounded instruction-following and safety tasks while reducing direct annotation labor; the weaker, contested claim is that AI-generated feedback constitutes a general substitute for human oversight, especially when judge and policy share training distributions or when target behavior begins to exceed human audit capacity.

Coverage note: verified through May 2026.

Definition and scope

RLAIF names a substitution at one specific stage of the preference-learning pipeline: the source of the preference signal. In the canonical RLHF pipeline (Ouyang et al. 2022), prompts are sampled, candidate completions are generated, human annotators compare pairs and pick the preferred completion, a preference model or reward model is fit to those comparisons, and a policy is optimized against that reward. RLAIF replaces — or augments — the human-annotation step with an AI evaluator that produces the pairwise comparisons. Everything else in the pipeline (the reward-model architecture, the optimizer, the evaluation suite, the deployment policy) is left undetermined by the term.

This narrowness matters because the literature regularly conflates four distinct concepts. Feedback source (human, AI, or hybrid) is orthogonal to preference-model architecture (explicit RM versus implicit DPO-style objective), to optimization method (PPO, DPO, rejection sampling, expert iteration), and to constitutional implementation (whether the AI judge consults an explicit written specification when producing labels). RLAIF is a claim about the first axis only.

The most common conflation is between RLAIF and Constitutional AI. Constitutional AI (Bai et al. 2022) is a specific RLAIF design in which the AI feedback model judges outputs against a written constitution and the training loop also includes a supervised self-critique-and-revision phase. Constitutional AI is the most influential RLAIF implementation, but the category is broader: any pipeline that derives pairwise preferences from an AI evaluator — with or without an explicit constitution, with or without revision — falls under RLAIF.

The pipeline RLAIF modifies

The clearest way to understand RLAIF is to start from RLHF and identify exactly what changes.

Step 1 — Prompt sampling. Prompts are drawn from a target distribution (instruction-following, dialogue, summarization, safety-relevant adversarial prompts). RLAIF does not change this step. Prompt distribution remains a human design choice and a significant lever on the resulting policy.

Step 2 — Candidate generation. The base policy generates two or more completions per prompt, typically by sampling at non-zero temperature. RLAIF does not change this step either.

Step 3 — Preference labeling. This is the step RLAIF replaces. In RLHF, human annotators are shown the prompt and the candidate completions and asked which is better, often with a rubric. In RLAIF, a separate language model (sometimes the same family as the policy, sometimes a stronger external model) is given the prompt, the candidates, and an instruction — possibly augmented by an explicit constitution or set of principles — and asked to produce a preference label, sometimes with a chain-of-thought explanation. The resulting (prompt, chosen, rejected) triples are functionally identical in format to RLHF data.

Step 4 — Preference-model or reward-model training. A model is fit to the preference labels. This step is invariant to feedback source. Whether labels came from humans or AI, a Bradley-Terry-style preference model or a scalar reward model is trained to score completions. Direct preference optimization techniques skip this step by folding the preference signal into the policy objective, but DPO is feedback-source-agnostic — it can consume human or AI preferences.

Step 5 — Policy optimization. The policy is trained against the reward model, typically with PPO in the original InstructGPT lineage, or via DPO in more recent stacks. This step is also invariant to feedback source.

Step 6 — Evaluation and deployment. Held-out evaluation uses human raters, AI judges, automated benchmarks, or adversarial probes. This step is where many published RLAIF results are most fragile: if the evaluation judge is drawn from the same model family as the training-time AI labeler, "comparable to RLHF" claims may reflect shared bias rather than independent validation.

The substitution is therefore localized but consequential. It moves preference annotation from a slow, expensive, human-bottlenecked process into a fast, cheap, model-bottlenecked process. The question is what that move costs in evidentiary independence and in correlated error.

Constitutional AI as the canonical implementation

Constitutional AI (Bai et al. 2022; Anthropic 2022) is the most cited RLAIF implementation and the source of much of the field's vocabulary. The Anthropic paper describes a two-phase training process for harmlessness.

The first phase is a supervised stage: the base model is shown its own outputs to red-team prompts, then prompted with a constitutional principle and asked to critique and revise the output. The revised outputs become the supervised fine-tuning data. This step does not by itself use RL; it uses self-critique to produce a less harmful training corpus.

The second phase is the RLAIF step proper. The post-SFT model generates pairs of responses to prompts. An AI feedback model — given a randomly selected constitutional principle — is asked which of the two responses better satisfies that principle. The resulting AI-generated preferences train a preference model. The policy is then optimized against that preference model using RL. Anthropic calls this phase "RL from AI Feedback."

Two design choices in CAI are easy to miss but load-bearing.

First, the constitution is human-written. It is a short set of principles — extracted from sources including the Universal Declaration of Human Rights, Apple's terms of service, and Anthropic's internal research norms — that specifies what kind of behavior is preferred. The constitution does not eliminate human normative input; it concentrates and surfaces it. This is a methodological virtue (the values are inspectable in a way that aggregated rater behavior is not) and a methodological hazard (the values are now a small, fixed artifact whose authors and selection process determine downstream behavior).

Second, CAI in the original Anthropic stack uses human preference data for helpfulness training and AI preference data for harmlessness training. The hybrid is not incidental. Helpfulness preferences are diverse, taste-driven, and benefit from broad human sampling; harmlessness preferences are policy-driven, benefit from consistency, and expose human annotators to harmful content during collection. Routing only the harmlessness signal through AI feedback is partly an efficiency move and partly a worker-protection and consistency move.

The category-versus-instance distinction matters when reading the literature. Statements like "Claude is trained with RLAIF" or "RLAIF replaced human feedback" frequently elide the fact that CAI is one RLAIF recipe among many possible ones, that it composes with human helpfulness data, and that subsequent production training pipelines at Anthropic and elsewhere are not fully described in public sources. RLAIF as a category is broader than CAI; what is publicly demonstrated as part of a frontier deployment is narrower than the category.

Empirical evidence: what is established, what is not

The most useful way to read the RLAIF literature is along three axes: what comparison is being run, what evaluator is judging it, and what cost dimensions are being measured.

Bai et al. 2022: Constitutional AI on harmlessness

The CAI paper (Bai et al. 2022) does not aim to prove that AI feedback dominates human feedback in general. It demonstrates that an AI-feedback-trained policy can be more harmless than an RLHF-trained baseline on Anthropic's red-team prompt suite while remaining helpful, and that the resulting model can refuse harmful requests while explaining its refusal rather than evasively dodging. The headline claim is that AI feedback plus a written constitution produces non-evasive harmlessness — a behavior shape that purely human-feedback training found difficult.

The bounded scope is significant. The comparison is harmlessness-specific, the evaluation involves Anthropic's own crowdworkers and red-team prompts, and the policy is from the Claude-precursor model family. Generalizing this to "AI feedback works as well as human feedback" requires a leap; what the paper actually shows is that AI feedback against an explicit constitution can produce a particular kind of trained behavior more effectively than human-feedback training without that constitution.

Lee et al. 2023: the direct head-to-head

Lee et al. 2023, "RLAIF vs. RLHF" is the cleanest published head-to-head comparison. The authors train policies with RLHF and RLAIF on three tasks: summarization, helpful dialogue, and harmless dialogue. They evaluate with human raters under blind comparison.

The reported results, in summary:

• On summarization and helpful dialogue, RLAIF and RLHF achieve roughly equivalent human-rated win rates against the supervised fine-tuning baseline.
• On harmless dialogue, RLAIF outperforms RLHF in their setup.
• "Direct RLAIF," in which a single off-the-shelf LLM produces a numerical preference score used directly as reward (skipping the reward-model training step), can match or exceed standard RLAIF.

The strongest defensible reading is: in the tasks studied, with the model families used, AI-generated preference labels produced policies that human raters judged comparable to or better than RLHF-trained policies. This is not nothing. It is the cleanest available evidence that AI feedback is not a degenerate substitute. But it is also bounded: three tasks, particular model families, blind human evaluation but no large-scale adversarial probe, and no explicit accounting of correlated-error rates between the AI labeler and the trained policy.

Sharma et al. 2024: the critical evaluation

The most important counterweight in the literature is Sharma et al., "A Critical Evaluation of AI Feedback for Aligning Large Language Models". The authors argue that a substantial fraction of reported gains from LLM-AI-Feedback (LAIF) pipelines can be explained by the strength of the teacher model used during the supervised fine-tuning stage, rather than by the AI-feedback step itself. In several of their experiments, simple supervised fine-tuning on completions from a strong teacher matches or exceeds more elaborate AI-feedback pipelines.

This is not a refutation of RLAIF; it is a methodological warning about isolation of effect. If a CAI-style or RLAIF-style pipeline uses a strong teacher for SFT and then layers AI feedback on top, attributing the final policy's quality to the AI-feedback step requires controls on the SFT teacher that many published comparisons do not run.

The Sharma critique reframes the open empirical question: not "does AI feedback work?" but "how much of the alignment gain attributed to AI feedback is actually attributable to teacher strength, and how much to the preference-optimization step?"

Comparison summary

Study	What was compared	Evaluator	What it supports	What it does not isolate
Bai et al. 2022	CAI (RLAIF + constitution) vs. RLHF on harmlessness	Crowd workers + red-team prompts	Non-evasive harmlessness via AI feedback	Generalization beyond harmlessness; teacher-strength control
Lee et al. 2023	RLAIF vs. RLHF on summarization, helpful, harmless	Blind human raters	Comparable human-rated quality in three text-generation tasks	Adversarial OOD performance; judge-policy error correlation
Sharma et al. 2024	LAIF pipelines vs. strong-teacher SFT	Automated + human	Some LAIF gains may be teacher-strength artifacts	(This is the isolation argument itself)
Rafailov et al. 2023	DPO vs. PPO on preference data	Multiple	DPO can replace PPO with similar quality	Feedback source — DPO works on human or AI preferences

A reader who wants the one-sentence empirical verdict should take this: RLAIF is a real and useful substitution at the preference-labeling step of the pipeline; it has been shown to produce policies that human raters rate comparably to RLHF on specific bounded tasks; whether its apparent gains generalize beyond those tasks, and how much of them survive a clean isolation of the AI-feedback step from teacher strength, remains contested.

Relationship to adjacent methods

The article-level confusion most worth dispelling is the assumption that RLHF, RLAIF, Constitutional AI, DPO, and deliberative alignment form a linear progression in which each method replaces the last. They do not. They occupy different axes of the design space and frequently compose.

Approach	Axis it specifies	Axis it leaves open	Composes with
RLHF	Feedback source = human	Optimizer, model, constitution	Any optimizer; supervised stages; AI-augmented helpers
RLAIF	Feedback source = AI	Optimizer, model, constitution, evaluation	DPO or PPO; constitutional or unconstrained judge; hybrid with human labels
Constitutional AI	RLAIF + written constitution + self-critique stage	Optimizer choice; helpfulness signal source	Typically PPO-style RL; helpfulness usually trained on human preferences
DPO	Optimizer = direct preference objective	Feedback source, model, constitution	Human, AI, or hybrid preferences
Deliberative alignment	Policy reasons over an explicit spec during training/inference	Feedback source, optimizer	RLAIF-style AI judging; chain-of-thought policies

Direct Preference Optimization ([Rafailov et al. 2023]) is a frequent point of confusion because it appeared in the wake of RLAIF discussion and is often discussed alongside it. DPO is not a feedback-source method. It is an optimization technique that fits a policy directly to preference pairs without explicitly training a separate reward model and without an online RL loop. Mathematically, DPO derives a closed-form relationship between an optimal policy and a preference model under the Bradley-Terry assumption, allowing the policy to be optimized via a simple supervised-style objective on the preference data. It is feedback-source-agnostic: DPO trained on human preferences is a form of RLHF (loosely); DPO trained on AI preferences is a form of RLAIF (loosely); DPO trained on hybrid preferences is whatever the data is. The optimizer choice and the feedback source are orthogonal and should not be conflated.

Deliberative alignment ([Guan et al. 2024]) is adjacent and frequently elided into RLAIF. The technique trains a model to reason explicitly over a safety specification before producing its final answer; the training data includes the chain-of-thought reasoning that grounds the response in the specification. Deliberative alignment overlaps with CAI in that both use written policies, and it overlaps with RLAIF in that synthetic data and AI critique can supply training signal. But its core methodological contribution is specification-conditioned reasoning at inference time, not the substitution of an AI labeler for a human labeler. Treating it as "RLAIF v2" misses what it adds. A pipeline can be deliberative-alignment-trained on AI-feedback-generated data and then deployed with run-time policy reasoning; deliberative alignment specifies the reasoning behavior, RLAIF specifies the labeling source.

The cleanest taxonomy for a reader trying to navigate this space is to ask, for any system claimed to be "AI-trained" or "AI-aligned," four separate questions: where did the preference labels come from, what did the optimizer do with them, did the judge consult an explicit specification, and does the deployed model reason over a specification at inference time. These four answers map cleanly onto the table above. Conflating any two of them produces vague claims that the field cannot evaluate.

The relocation argument: where human judgment goes when "human feedback" goes away

A common framing — including framings sometimes used to motivate RLAIF — is that AI feedback "removes the human-annotation bottleneck." This is a partial truth that hides a structural fact: in any RLAIF pipeline deployed at the alignment frontier, human judgment is not eliminated. It is relocated.

A complete RLAIF stack still requires human decisions about:

• The constitution or principle set. In CAI, the constitution is human-authored. Even when constitutions are model-drafted, the selection, editing, and final acceptance of the constitution are human acts. This is the most consequential normative input in the system.
• The judge model. Selecting which model produces preference labels — its training history, its capabilities, its biases — is a human choice with strong effects on downstream policy behavior.
• The judge prompt. What instruction is given to the AI labeler when it produces a preference? How are principles surfaced? How is reasoning elicited? These prompts function as soft policy.
• The seed data. Supervised fine-tuning data, including any teacher completions used in CAI's revision stage, is selected and curated by humans.
• The evaluation suite. Red-team prompts, adversarial probes, OOD evaluations, and human-rater protocols are designed by humans.
• The deployment policy. Acceptance thresholds, refusal categories, escalation rules, and release decisions remain human.
• The audit and red-team process. Internal teams and external evaluators continue to probe the deployed model.

Counting only the cost of pairwise preference annotation as "the human cost of alignment" therefore systematically understates the human labor in any RLAIF pipeline. The honest framing is that RLAIF scales preference-label volume and reduces the marginal labor of generating additional comparisons, while concentrating normative judgment in a smaller number of higher-leverage artifacts (constitutions, judge prompts, eval suites, deployment policies).

This is not a critique of RLAIF — concentrating normative judgment in inspectable artifacts has real virtues over diffusing it through tens of thousands of rater decisions. But it does affect how cost and oversight claims should be read. "Fraction of the human cost" is a defensible claim about pairwise annotation labor; it is not a defensible claim about total human labor invested in alignment.

Critiques and failure modes

The empirical literature on RLAIF is small. The empirical literature on the adjacent failure modes — LLM-as-judge bias, sycophancy, reward hacking, preference-model overoptimization — is large, and most of its findings transfer.

Correlated error between judge and policy

The structural risk most specific to RLAIF is common-mode failure between the AI judge and the trained policy. If the judge and the policy share training data, share architectural priors, share fine-tuning lineage, or share constitutional context, errors that appear in the policy may be invisible to the judge. The benchmark agreement rate that results may look like high-quality supervision while in fact reflecting shared blind spots.

This risk is not purely theoretical. Wang et al. 2023 document position bias in LLM evaluators: GPT-4 used as a judge prefers the first-presented response at substantially above 50% even when the order is randomized. Saito et al. 2023 document verbosity bias: LLM judges, GPT-4 included, systematically prefer longer responses, holding quality constant. Koo et al. 2024 catalog a broader set of cognitive biases in LLM evaluators and report imperfect alignment with human preferences. None of these studies are RLAIF-specific, but each describes a failure mode that an RLAIF judge can plausibly transmit to a trained policy via the preference signal.

The detection question — how to know whether common-mode failure is happening in a specific RLAIF pipeline — is harder than it sounds. AI-judged held-out evaluations may inherit the same bias as the training judge. Human-judged held-out evaluations help, but human raters are also subject to verbosity preference and to sycophancy preference, which suggests that some shared-bias surfaces will not be cleanly detected by switching evaluators. Adversarial and out-of-distribution probes — prompts deliberately constructed to expose specific biases — are currently the most credible detection tool, and most published RLAIF comparisons do not run them.

Sycophancy and preference-model exploitation

Sharma et al. 2023/2025 show that preference models — both human and AI — can systematically prefer responses that validate user beliefs over responses that correct them. Trained policies inherit this preference. The phenomenon is not unique to RLAIF; it appears in RLHF as well. But RLAIF can intensify it: an AI judge trained on human preference data may have learned and now amplifies the original human preference for agreement.

Ibrahim et al. 2026 document a related but distinct failure mode: warmth-targeted training can reduce factual accuracy and increase validation of incorrect beliefs. This work is recent enough that its full implications for AI-feedback pipelines are still being worked out, but the mechanism it describes — that one optimization target (warmth, agreeableness, helpfulness-as-validation) can quietly trade off against another (accuracy, calibrated disagreement) — is exactly the kind of multi-objective tension that RLAIF pipelines must manage carefully when the AI judge has implicit preferences along multiple dimensions.

Normative lock-in via the constitution

In CAI-style RLAIF, the written constitution is the single most influential normative artifact. Its authorship, its selection from possible source documents, its principle hierarchy, and its revision process collectively determine which behaviors the trained policy will be optimized toward. This concentration is a feature for inspectability, but it is also a vulnerability: a constitution that is well-written for current deployment may become outdated, miscalibrated against new failure modes, or systematically blind to harms that did not appear in the source material used to draft it.

This is a governance problem more than a technical one. The risk is not that constitutions are arbitrary — most published constitutions reflect serious normative work — but that the labor of keeping them well-calibrated to deployment realities is non-trivial, ongoing, and undervalued in cost analyses that count only the pairwise labels avoided.

Distribution narrowness

Most published RLAIF results are on text-only tasks, in English, evaluated on prompts that are reasonably close to instruction-following or dialogue distributions. The generalization of those results to multimodal models, to long-horizon agentic tasks, to non-English settings, to tasks requiring factual grounding in fast-moving domains, and to tasks requiring tool use and external verification is largely an open question. Published RLAIF papers do not establish that the method extends well to these settings; they establish that it works in the narrower settings actually tested.

Cost accounting

The claim that RLAIF achieves alignment "at a fraction of the human cost" deserves explicit unpacking. The numerator usually being measured is the cost of producing pairwise preference labels: dollar-per-label for human annotators versus dollar-per-label for AI inference. By that measure, AI feedback is straightforwardly cheaper, often by orders of magnitude.

The numerator that is not usually being measured includes: the cost of designing and iterating on the constitution; the cost of selecting and prompting the judge model; the cost of running the judge model at scale (often substantial — a large judge model running over millions of preference pairs is non-trivial); the cost of evaluation, including adversarial and red-team work; the cost of audit; the cost of failed training runs that human raters might have caught earlier; the cost of correcting deployed behavior that resulted from undetected judge bias. None of these costs are zero, and some — particularly judge-model inference at frontier scale — can rival the labor cost they replace.

The defensible cost claim is therefore narrower than "fraction of human cost": AI feedback reduces direct pairwise-annotation labor and allows faster iteration on behavioral constraints, especially when human raters are expensive, inconsistent, or exposed to harmful content during collection. Whether total alignment cost goes down is a more complex calculation that depends on judge scale, audit intensity, and how often correlated errors necessitate retraining.

Deployment implications: Claude and beyond

Public Anthropic materials make Constitutional AI and AI feedback central to the Claude training lineage. The original Constitutional AI paper (Bai et al. 2022) was published by Anthropic and explicitly framed the method as part of their alignment research program. Anthropic's public communications (Anthropic 2022; Anthropic 2023 on Claude's Constitution) describe the constitution and the RLAIF-for-harmlessness loop as production-relevant components of how Claude is trained.

The honest scope of this claim matters. Public sources support:

• That Constitutional AI methodology is part of the Claude training program.
• That AI-generated preference feedback is used at the harmlessness stage.
• That the constitution itself has been published in versions and reflects principles drawn from named sources.

Public sources do not establish:

• The full current production training recipe for any specific Claude release.
• That RLAIF is the sole or dominant alignment mechanism used for current Claude models.
• The exact mixture ratios of human and AI preference data across training stages.
• Which optimizer (PPO, DPO, hybrid) is currently in production.
• What additional stages — deliberative alignment, character training, reinforcement learning from verifiable rewards — compose with RLAIF in current pipelines.

The defensible deployment statement is: Constitutional AI and RLAIF are part of the documented method stack used to train Claude, particularly for harmlessness training; the complete contemporary training recipe is not public; "Claude is trained with RLAIF" is true in the sense that RLAIF is among its methods, not in the sense that RLAIF is its full training procedure.

Other frontier labs have published less detail about the role of AI feedback in their own training. References to AI-feedback techniques appear in research from OpenAI, DeepMind, Meta, and others, but the same scoping caution applies: published mention of a method does not establish its current production role.

What would resolve the open questions

The current evidence permits a positive scoped claim about RLAIF and leaves a broader claim — that AI feedback is a general substitute for human feedback at the frontier — undecided. A reasonably small set of experiments would tighten this considerably.

The cheapest decisive experiment is a factorial, cost-accounted comparison. Holding base model and prompt distribution fixed, the relevant axes to cross are:

• Feedback source: human, AI, hybrid (and within hybrid, multiple mixing ratios).
• Teacher strength for SFT: weak, medium, strong, matched across all conditions (this is the Sharma et al. control).
• Optimizer: explicit RM + PPO, DPO, direct RLAIF score.
• Judge family: judge drawn from same family as policy, judge drawn from different family, judge with explicit constitution, judge without.

The evaluation must include all of the following, not any one of them alone:

• Blind human evaluation, with rater selection independent of any group involved in training-data collection.
• Adversarial probes for the bias classes documented in the LLM-judge literature (position, verbosity, sycophancy, self-preference).
• Out-of-distribution evaluation on tasks not represented in the training prompt distribution.
• Direct measurement of judge-policy error correlation on held-out cases, to detect common-mode failure.
• Full cost accounting: annotation labor, judge inference, training compute, audit and red-team time, retraining for failed runs.

This experiment does not exist in the published literature in fully crossed form. Pieces of it exist across Lee et al. 2023 (the human-AI feedback comparison), Sharma et al. 2024 (the teacher-strength control), and the LLM-as-judge bias literature (the adversarial probes). A laboratory with the resources to run it as a single factorial study could substantially narrow the current uncertainty.

Until such a study exists, the field's evidence base on RLAIF will continue to be a patchwork in which each individual study supports a defensible narrow conclusion, but the aggregate is read more confidently than the parts warrant.

Open question: scalable feedback or scalable blind spots

The deepest unresolved question about RLAIF is not whether it works today, but what happens to it as models become more capable.

The optimistic view is that as judge models become stronger, AI feedback becomes more reliable than human feedback for an expanding range of tasks. Strong judges have effectively unlimited time per comparison, can be asked to reason explicitly, can consult tools, can be repeatedly re-prompted with different framings, can be calibrated against ground truth where it exists, and do not suffer from rater fatigue or exposure to harmful content. In this view, RLAIF is not merely a stopgap for current models but a path to scalable oversight — feedback that can supervise models whose outputs are too numerous, too long, or too technical for human review to scale.

The pessimistic view is that as judge models and policy models both scale, they tend to share training distributions, architectural priors, and benchmark exposure. The common-mode failure surface grows rather than shrinks. A judge that is more capable than the policy may still share its blind spots, and "blind spot" is by definition the region in which agreement provides no information. In this view, RLAIF's gains plateau or even regress at the frontier, and the alignment problem the field thought it was scaling its way out of reappears in a different form: scalable feedback collection without scalable value discovery.

The intermediate and probably most defensible view is that AI feedback scales well in the regime where the target behavior remains within human audit capacity, because in that regime humans can detect judge-policy collusion when it matters. AI feedback becomes increasingly fragile as target behavior approaches and exceeds human audit capacity, because the very condition that makes AI feedback most attractive (humans can't keep up) is also the condition under which humans can no longer verify that the AI judge is correctly steering the policy. In this view, RLAIF is necessary but not sufficient for frontier alignment, and the field's future research on debate, recursive reward modeling, scalable oversight more broadly, interpretability-augmented evaluation, and deliberative alignment reflects an implicit acknowledgment that AI feedback alone is unlikely to carry oversight through superhuman regimes.

None of these views is established by current empirical evidence. They are positions in a methodological debate about what RLAIF's ceiling will turn out to be. The honest framing for a reader new to the topic is that this is the live question — not whether RLAIF works on summarization, but whether the techniques that work on summarization continue to work when the policy being trained begins to do things its human-and-AI feedback loop cannot independently verify.

Companion entries

Core theory:

• Reinforcement Learning from Human Feedback
• Constitutional AI
• Direct Preference Optimization
• Deliberative Alignment
• Reward Modeling
• Scalable Oversight

Practice:

• LLM-as-Judge
• Preference Optimization Pipelines
• Red Teaming and Adversarial Evaluation
• Synthetic Training Data
• Character Training

Counterarguments and failure modes:

• Sycophancy in Language Models
• Reward Hacking
• Preference Model Overoptimization
• Common-Mode Failure in AI Evaluation
• Verbosity Bias in LLM Judges
• Normative Lock-in via Specifications

Adjacent topics:

• The Alignment Problem at Scale
• Anthropic's Alignment Research Program
• Cost Accounting in Frontier Training

---