Before ChatGPT, There Were Atari Preferences

Before anyone talked about RLHF, a small team asked a stranger question: what if you never write down a reward function at all? Their answer — learn it from human preferences between pairs of trajectory segments — became the conceptual seed of the entire post-training era. ChatGPT's training pipeline is, at its core, a scaled-up, language-flavored version of this 2017 idea.

The reward specification problem

Reinforcement learning needs a reward signal, and for most interesting tasks that signal is brutally hard to specify. Write a reward for "play this Atari game well" and you'll get an agent that exploits your scoring quirks. Write one for "be helpful" and you'll discover just how many behaviors the word "helpful" fails to pin down. The paper's framing is blunt: for sophisticated RL systems to interact usefully with the real world, we need to communicate complex goals to them, and code is a terrible medium for that communication.

Their alternative: show a human two short clips of agent behavior — trajectory segments of a second or two — and ask which one better achieves the goal. Fit a reward model to those pairwise judgments (the classic Bradley–Terry model, the same math behind Elo ratings), then run ordinary RL against the learned reward. The human never writes a reward, never demonstrates the task, never even needs to understand the environment's dynamics. They just express preferences, which turns out to be the easiest thing to ask of a person.

What made it practical

The obvious objection is cost: human feedback on every step would never scale. The paper's central empirical claim is that it scales better than you'd think. Because the reward model generalizes across states, the agent needed human feedback on less than one percent of its interactions with the environment — the learned reward filled in the rest. On Atari games and simulated robot locomotion, the approach solved complex tasks with no access to the true reward function at all.

Even more striking: the authors trained genuinely novel behaviors — things like a simulated robot doing backflips — with about an hour of human time. Not an hour of expert time. Non-expert labelers, one hour, new behaviors. Consider what that implies about the division of labor: the human provides judgment ("this looks better than that"), the reward model provides generalization, and the RL algorithm provides optimization. Each component does what it's best at, and the interfaces between them are thin. That modularity is why the pipeline survived the jump from simulated robots to language models five years later with its structure intact.

The preference model itself deserves a closer look. Given two segments, the model assigns each a scalar reward, and the probability the human prefers one is the logistic function of the reward difference — the Bradley–Terry model, borrowed from chess ratings. Training is just cross-entropy against the human's choices. And the queries weren't random: the system actively selected segment pairs where the reward model was most uncertain, an early form of active learning that squeezed maximum information out of each judgment. That combination — a simple probabilistic model plus uncertainty-driven querying — is why the human-time cost stayed so low.

The pipeline had three moving parts that every RLHF system since has inherited:

• Preference elicitation: trajectory segments, pairwise comparisons, non-expert labelers.
• Reward modeling: learn a scalar reward function from the comparisons.
• Policy optimization: run RL against the learned reward instead of the true one.

The cracks visible from the start

Read closely and the paper is honest about the fragility it introduces. The policy is only as good as the learned reward model, and the reward model is only as good as the preferences — which means reward hacking isn't a bug to be patched later, it's structural. An agent optimizing a learned proxy will find the proxy's blind spots; the paper's setup just hopes the proxy is good enough. Human labelers are also noisy and inconsistent, and the paper's aggregation doesn't fully solve that — it mostly shows the pipeline survives it.

There's a deeper limitation the authors couldn't have seen: everything here lives in simulated worlds with crisp, short-horizon tasks. Whether preferences scale to the fuzzy, long-horizon, open-ended judgments that language models require — "is this a good answer?" — was completely untested. The 2022 InstructGPT paper would be the experiment that answered it.

It's also worth noting what the paper's reward model couldn't represent: uncertainty about human values. A single scalar per state collapses disagreement between labelers into an average, and RL then optimizes that average relentlessly. Later work on pluralistic alignment and distributional reward models starts from this observation — that "what humans want" was never a single function, and pretending it is creates its own failure modes. The 2017 paper got away with it because backflips don't have moral philosophy; language models weren't so lucky.

The paper's real bet: that human judgment is easier to elicit than human specification — and that a learned model of what we want is a good enough stand-in for wanting it ourselves.

Where it sits in the arc

This is the paper that made RLHF thinkable. Christiano and co-authors went on to found the alignment teams that built InstructGPT; the preference-comparison interface they designed is essentially unchanged in today's RLHF data pipelines. Re-reading it now, the most prescient line is the problem framing itself: not "how do we train better agents" but "how do we communicate complex objectives to systems that optimize literally." Every reward-hacking incident since — and there have been many — is a footnote to that sentence. The machinery changed; the problem didn't.

Paper: Deep Reinforcement Learning from Human Preferences — Christiano et al., 2017.