Self-Rewarding Language Models: The Judge That Learns to Judge Better

RLHF has a structural flaw that everyone politely ignores: the reward model is frozen. You train a judge on human preferences, freeze it, and then optimize a policy against it for thousands of steps — while the policy keeps changing and the judge never learns a thing. The judge is a snapshot of human taste at one moment, and it becomes the ceiling the policy can never see past. The Self-Rewarding Language Models paper, from Meta, asks what happens if you refuse that separation: one model that both follows instructions and judges responses, trained so that each iteration's model supplies better training signal to the next. The judge learns to judge better, and the student learns from a teacher that is itself — one iteration ahead.

One model, two skills, iterated

The setup starts from Llama 2 70B, fine-tuned on two kinds of seed data. First, IFT data: 3,200 high-quality human instruction-following examples from Open Assistant. Second, EFT (evaluation fine-tuning) data: 1,630 examples teaching the model to act as a judge — given an instruction and a response, produce chain-of-thought justification and a score out of 5, using five additive criteria: relevance, coverage, usefulness, clarity, and expertise. Judging is framed as just another instruction-following task, which means the same weights that generate answers can score them.

Then the loop, called AI Feedback Training (AIFT), runs per iteration:

• Self-instruction creation. Generate fresh prompts (few-shot sampled from the seed data), sample N diverse candidate responses from the current model, and have that same model score each one via LLM-as-a-Judge prompting.
• Preference construction. Take the highest- and lowest-scoring responses as win/lose pairs (discarding ties), building a preference dataset with zero human labels.
• DPO training. Train the next model M_{t+1} with Direct Preference Optimization on those pairs.

The sequence runs M_0 (base) → M_1 (SFT on seeds) → M_2, M_3 (iterative DPO on self-generated preferences). The key measurement is two-dimensional: track both instruction-following quality and reward-modeling quality across iterations. Both go up. The model at iteration t literally provides a higher-quality preference dataset to itself than it did at iteration t−1 — the training signal compounds.

One design choice deserves attention: the judge wasn't assumed competent — it was trained. The EFT seed data (1,630 train, 541 eval examples) was built by having the SFT baseline generate chain-of-thought justifications and 0–5 scores for Open Assistant responses, keeping only generations whose rankings agreed with human rankings, then rebalancing away from the modal score of 4. And evaluation ran on two independent axes: head-to-head instruction-following judged by GPT-4 over 256 prompts (with the human authors as a check), and reward-modeling accuracy measured separately. Both axes improved across M_1 → M_3 — the model didn't just get better at pleasing its own judge; the judge itself got better at matching held-out human preferences. That's the compounding the paper claims: not a model gaming a fixed scorer, but a scorer learning alongside.

The result that got attention

After three iterations, the self-rewarding model outperformed many existing systems on the AlpacaEval 2.0 leaderboard — including Claude 2, Gemini Pro, and GPT-4 0613. A Llama 2 70B seed, improved with no additional human preference data beyond the seed set, beating frontier proprietary models of its era on instruction following. The paper is careful to note the effect likely saturates, but the conceptual point stands: reward models superior to what the original human data alone could produce are reachable, because the judge is no longer frozen. The frozen judge was the binding constraint on the whole alignment pipeline; unfreezing it is what lets the loop keep climbing past the quality of its own seed data.

The frozen reward model is a snapshot of human taste; a self-rewarding model is a moving picture. When the judge improves alongside the student, the ceiling on self-improvement lifts every iteration — until the judge's own taste becomes the new ceiling.

Limitations worth naming

The optimism needs tempering. The judge's taste is bootstrapped from human seed data, so the whole edifice inherits whatever biases and blind spots those 3,200 Open Assistant examples contain — the model can only ever refine its own notion of quality, which may drift from human preferences rather than toward them. There's a self-reinforcement hazard: a model that judges by its own stylistic tics will reward those tics in the next generation, potentially collapsing toward verbose, sycophantic, or otherwise degenerate-but-high-scoring outputs. And the headline benchmark is itself LLM-judged (AlpacaEval uses GPT-4 as evaluator), which introduces an uncomfortable circularity when the claim is about judging ability. Finally, the paper's own caveat stands: the compounding likely saturates — each iteration's gains were real but the curve has to flatten, because the seed distribution is finite.

Where it sits in the RSI arc

Self-Rewarding LMs is STaR's loop with the filter turned inward: where STaR used ground-truth answers as the correctness signal, here the model is the signal. It's also the direct answer to RLAIF's limitation — Lee et al. showed AI feedback works even when the labeler is the same checkpoint, but their labeler stayed frozen; this paper unfreezes it. And it names the problem that the 2026 on-policy distillation work (DCE/SRCL) would later attack at the token level: a supervisor that doesn't co-evolve with the student becomes the bottleneck. In the RSI arc, this is the paper that made the evaluator a first-class citizen of the loop — improvement isn't just about better policies, it's about better judges, and the two have to move together.

Paper: Self-Rewarding Language Models — Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston, 2024.