RLAIF: What Happens When the Labeler Is You

RLHF works, but it has a scaling problem with a human face: every preference label costs annotator time, and frontier capabilities will eventually exceed what annotators can reliably judge. RLAIF — reinforcement learning from AI feedback, first introduced by Bai et al. in the Constitutional AI work — proposes replacing the human labeler with an off-the-shelf LLM. This paper, from Google DeepMind, is the one that made it a controlled science: the first direct head-to-head comparison of RLAIF versus RLHF, plus a striking self-improvement result — RLAIF improves the policy even when the AI labeler is the exact same checkpoint as the policy being trained. The model, grading its own outputs, still learns something.

How you get preferences from a model

The method is disarmingly mechanical. Take an off-the-shelf LLM — pretrained or instruction-tuned, not fine-tuned for the task — and prompt it with a preamble describing the rating task, optionally few-shot exemplars, then a context plus two candidate responses, ending with something like "Preferred Response=". Extract the log-probabilities of the tokens "1" and "2", softmax them, and you have a soft preference label — a distribution, not a one-hot vote, which the authors note carries more information.

Two refinements matter. First, position bias: the order candidates appear in biases the verdict, especially in smaller labelers — so every pair is scored twice with the order flipped, and the results averaged. Second, chain-of-thought: a two-step procedure where the model first writes its reasoning about the candidates, then that reasoning is appended to the prompt for the final scoring. CoT consistently improved the AI labeler's alignment with human preferences; detailed preambles and few-shot examples helped only on some tasks.

The paper also ran the labeler-size question directly: larger LLM labelers align better with human preferences, with the expected diminishing shape — and smaller labelers suffer worse position bias, which is why the double-scoring mitigation matters most exactly where you'd like to use cheap labelers. The practical read is that RLAIF's economics favor large labelers scoring many pairs over small labelers scoring carefully, a tradeoff that foreshadows the "frontier model as judge" pattern the industry later adopted wholesale.

Canonical RLAIF then trains a reward model on these AI labels (cross-entropy on the soft targets — effectively a form of distillation) and runs RL against it. But the paper also introduces direct-RLAIF (d-RLAIF), which skips the reward model entirely: during RL, the LLM directly scores each generation on a 1–10 scale, the token likelihoods become a weighted average normalized to [−1, 1], and that is the reward. This fixes a real pathology — reward-model staleness: a reward model trained on the initial policy's outputs goes increasingly out-of-distribution as the policy improves, while d-RLAIF's judge never goes stale because it's never trained.

The head-to-head

Across summarization, helpful dialogue, and harmless dialogue: human evaluators preferred RLAIF over the SFT baseline 71% of the time for summarization (RLHF: 73%) and 63% for helpful dialogue (RLHF: 64%) — differences that aren't statistically significant. Head-to-head, RLAIF and RLHF were equally preferred, ~50/50. On harmlessness, RLAIF actually won outright: an 88% harmless rate versus 76% for RLHF and 64% for SFT. AI feedback, at least in these domains, is a viable substitute for human feedback.

Then the self-improvement finding: RLAIF significantly beat the SFT baseline even when the labeler was the same size as the policy — and even when it was the same checkpoint. For helpful dialogue under d-RLAIF, the initial policy and the reward-giving LLM were literally the same model. Strict self-improvement: the model improving itself with nobody else's judgment in the loop.

RLAIF's uncomfortable implication: if a model grading its own outputs — the same checkpoint, no outside knowledge — can still produce a training signal that beats supervised fine-tuning, then "human judgment" was doing less work in RLHF than we thought, and the bottleneck was always the structure of the feedback loop, not its source.

Limitations worth naming

The labeler may be AI, but in this paper it's still frozen — an off-the-shelf model that never learns from the process, which is exactly the limitation Self-Rewarding LMs would later attack by unfreezing it. AI preferences inherit the labeler's biases, and the paper's own ablations show smaller labelers are noisier and more position-biased. The harmlessness win deserves a skeptical read: an AI labeler may simply be more conservative than human raters, and "more harmless" isn't the same as "more aligned." And all of this is demonstrated on tasks where LLM judgment is already known to correlate with human judgment — summarization, chit-chat, safety refusals. Whether AI feedback transfers to domains where models are weak judges (novel reasoning, superhuman tasks) is precisely the open question the paper's framing raises but can't answer.

Where it sits in the RSI arc

This paper is the hinge between RLHF and the self-improving post-training era. Bai et al. invented RLAIF; Lee et al. proved it matches RLHF and — more importantly — demonstrated that the human can be removed from the same-checkpoint loop entirely, which is the minimal definition of recursive self-improvement in post-training. From here the arc is clear: Self-Rewarding LMs unfreezes the labeler so judge and policy co-evolve; the 2026 on-policy distillation work (DCE) pushes that co-evolution down to the token level. Each step weakens the role of the human in the loop while trying to keep the signal honest — and the field still hasn't settled what, if anything, ultimately grounds it.

Paper: RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Harrison Lee et al., 2023. (RLAIF introduced by Bai et al., 2022.)