Guest lecture · Part 7

Eric Mitchell · RLHF

Guest lecture from Stanford's Eric Mitchell on the technique that turned GPT-3 into ChatGPT. He builds up the classic four-stage RLHF pipeline — pretraining, supervised fine-tuning, a reward model fit to human preferences, then PPO with a KL leash — and then derives Direct Preference Optimization, which collapses the last two stages into a single classification loss on the policy itself. He closes with an application: making language models more factual using DPO on automatically generated preference labels, no humans required.

▶ Lecture video

Why reinforcement learning for language models

Eric Mitchell frames the motivation with a before-and-after: “GPT-3 was cool — all of your researcher friends were like, whoa, have you seen this new model. Then ChatGPT came around and your grandma was like, yo, have you seen this new model.” What sits between those two levels of public permeation is “more or less one weird trick” — reinforcement learning from human feedback.

The setup: a language model is an autoregressive model over tokens (roughly, words) from a vocabulary of around 50,000, mapping a token sequence to a distribution over the next token. To make it an RL problem, states are token sequences and actions can be either single tokens or — as in most of the work so far — entire response sequences. That second choice makes RLHF a contextual bandit: the prompt is the state, the whole response is one action, and in multi-turn dialogue the state simply grows to include everything said so far. Mitchell points to the contrastive preference learning paper for a careful discussion of the bandit versus per-token MDP formulations, but sticks with the bandit view because it underlies the algorithms he covers.

The pipeline: pretraining, SFT, and a reward learned from preferences

RLHF as practiced has four steps, and the first two are familiar. Step zero is unsupervised pretraining on trillions of words — “if you stop after step zero, you get GPT-3.” Step one is supervised fine-tuning (SFT) on human-written demonstrations of good responses, which puts the behaviors you care about in distribution for the model. Why not stop there? Two reasons: demonstrations are laborious to scale (“write me a sassy poem in the style of Snoop Dogg about quantum mechanics”), and imitating humans cannot give you a policy that exceeds human performance.

So we want RL — but what reward? Knowing what deserves high reward requires knowing something about humans, so the reward must be elicited from them, in an inverse-RL-like setup. The obvious move — ask people for a number between one and ten — fails in an instructive way: Mitchell polls the room on a sample response and gets a five, an “eight out of five,” and a three. Different labelers can agree on rankings while differing by a monotonic transformation of the scores, and the disagreement becomes annoying to model. Asking which of two responses is better gets far higher agreement, and far more hands raised. “It’s much easier to put two responses in front of a person and say which do you like better than it is to say write me a good demonstration.”

Replace the latent score with a parameterized reward model $r_\phi(x, y)$ and do maximum likelihood on the preference dataset:

$\mathcal{L}(\phi) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\right]$

Out pops a reward model. A sobering practical note: measured as a classifier, its accuracy at ranking pairs the way the human did is often under 70% — real preferences are extremely noisy.

RL with a KL leash

Step three is “just RL”: maximize the learned reward. But optimizing the raw reward model goes wrong through a specific kind of distribution shift. The reward model was trained on pairs of SFT-model samples, so it is only trustworthy near what the SFT model assigns high likelihood. Optimize unconstrained “to oblivion” and you land in a region of response space where the reward model emits garbage — measured reward climbs and climbs while humans call the outputs trash. The fix is a KL penalty keeping the policy near the SFT reference:

$\max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\left[r_\phi(x, y)\right] - \beta\, D_{KL}\big(\pi_\theta(\cdot \mid x)\,\|\, \pi_{\text{ref}}(\cdot \mid x)\big)$

This objective is optimized with PPO — “you don’t have to use PPO, but that’s what John Schulman decided, and therefore that’s the world that we live in.” The full machinery is genuinely complicated: an old policy, the frozen SFT reference for the KL term, a reward model, a value function fit online, the policy itself, and a replay buffer.

Nonetheless the recipe works. In the InstructGPT results, humans prefer the 1.3B RLHF model to the 175B SFT model — the difference between “I can run it on my 2070 Ti” and “I pay an engineering team to fine-tune this model” — and the gains persist as models grow, so it is not a small-model crutch. Two variations worth knowing: human feedback can train things other than reward models, such as a diversity metric (quality diversity through human feedback, which yields visibly more diverse images for the same prompt), and the annotator need not be human — RL from AI feedback, as in Claude’s training, uses a helpfulness-trained model to label harmlessness comparisons instead of asking people.

DPO: your language model is secretly a reward model

What’s not ideal about RLHF-with-PPO: implementation complexity, the resource cost of holding four-plus models, training instability — and a subtle degree of freedom. The Bradley–Terry loss only sees reward differences, so for any fixed prompt you can shift all rewards by an arbitrary constant, different per prompt, without changing the loss. Conceptually you could get a reward function whose values cannot be compared across prompts at all, which makes fitting a value function miserable.

Direct preference optimization removes the RL loop entirely. The key fact is a one-to-one correspondence between reward functions and optimal policies.

Closed-form optimal policy, and the reparameterization

For the KL-constrained objective above with any reward $r$ , a couple of lines of algebra give the optimal policy in closed form:

$\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\left(\tfrac{1}{\beta}\, r(x, y)\right)$

— high probability exactly where the reference model already assigns reasonable probability and the reward is high. $Z(x)$ sums over all possible response sequences, so this is intractable to sample from directly. But rearranging expresses the reward in terms of its optimal policy:

$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$

A reward function can be parameterized as a log-probability ratio between a policy and the reference model.

Now substitute this parameterization — with a trainable policy $\pi_\theta$ in place of $\pi^*$ — into the ordinary Bradley–Terry reward-modeling loss. Since both responses in a pair share the same prompt, the intractable $\beta \log Z(x)$ terms cancel, leaving a fully tractable objective directly on the policy:

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$

That is DPO: the same reward-learning loss, but training a policy that is by construction optimal for the implicit reward being fit — no rollouts, no value function, no replay buffer, just a classification objective. The per-prompt degree of freedom disappears too: the implicit reward is built from a normalized distribution, and you cannot raise the log-probability of every response at once. And on the question of whether PPO gains something by seeing fresh unlabeled prompts during RL, Mitchell’s answer is no — you are bottlenecked by reward-model accuracy either way, and empirically DPO generalizes at least as well.

Reading the gradient, and checking the frontier

The most intuitive view of DPO is its gradient: an expectation over the data of

$-\beta\, \sigma\big(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w)\big) \left[\nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x)\right]$

where $\hat{r}_\theta$ is the implicit reward $\beta \log(\pi_\theta / \pi_{\text{ref}})$ . Two forces — maximum likelihood on the chosen response, unlikelihood on the rejected one — scaled by a per-example weight that is large when the implicit reward ranks the pair incorrectly and shrinks to zero once it gets the pair right. That adaptive weight is where the KL constraint implicitly lives: examples the model already ranks correctly stop contributing, so it never over-trains its way far from the reference.

The empirical check uses a deliberately simple task — complete movie-review prefixes with maximal positive sentiment, with a ground-truth sentiment classifier as reward, so the reward–KL trade-off can actually be plotted. Across many runs and checkpoints, DPO traces the strongest reward-versus-KL frontier; the unlikelihood ablation (DPO without the per-example weight) collapses; preferred-response fine-tuning barely moves; and PPO falls short of the frontier even when handed the ground-truth reward — partly because that true reward is sparse and badly shaped, whereas a learned reward can be smoother and easier to optimize. A practical bonus: DPO’s $\beta$ maps predictably onto the final KL ( $\log \beta$ against $\log KL$ is nearly linear), whereas PPO practitioners resort to dynamic controllers that tune $\beta$ on the fly.

Application: fine-tuning for factuality

Language models “just cannot be trusted”: Bard’s James Webb telescope flub, a Bing demo whose analysis of a quarterly report fabricated earnings-per-share figures and its entire future outlook, and a reproducible oddity where ChatGPT correctly says Chelsea Finn got her PhD at Berkeley — unless you ask for just the university name, in which case it says Stanford. When a model says something true or false is weird and mysterious.

Why doesn’t standard RLHF fix this? Because RLHF encourages behaviors that make humans happy, “and doing fact-checking does not make humans happy, let me tell you.” An Anthropic study ranked which attributes predict human preference: truthfulness was not near the top, while agreement with the rater’s beliefs — “surprise, surprise” — was the single most predictive feature. Humans are simply bad annotators for factuality, so this is a natural home for AI feedback: score truthfulness automatically against a reference like Wikipedia, assemble preference pairs where one response is more factual than the other, and train with DPO. On biography generation and medical question answering, this factuality tuning significantly cuts incorrect facts per response while increasing correct ones — the only method tested that strictly improves on SFT, which RLHF itself does not.

The place of RL here is telling: pretraining, over trillions of tokens, is roughly where the model learns what is true; the lightweight fine-tuning phases cannot teach new facts, but they can teach the model to “just say the true stuff.”

Back of the chalk tray — what to remember

RLHF treats a language model as a contextual-bandit policy: prompt as state, whole response as one action — this is the “one weird trick” between GPT-3 and ChatGPT.
Comparisons scale where demonstrations and scalar ratings don’t: the Bradley–Terry model converts binary choices into a maximum-likelihood objective for a latent reward.
The learned reward is only valid near the SFT distribution, so RLHF maximizes reward under a KL penalty to the reference model — classically with PPO, at the cost of juggling policy, reference, reward model, value function, and replay buffer.
DPO exploits the closed-form optimal policy $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$ to reparameterize the reward as a log-probability ratio; substituting into the Bradley–Terry loss cancels the partition function and yields a simple classification loss directly on the policy — same objective, no RL loop.
DPO’s gradient pushes up chosen and down rejected responses, weighted by how wrongly the implicit reward ranks the pair — an implicit KL constraint; but it learns only differences, so tokens shared between chosen and rejected (the TL;DR failure) teach it nothing.
Human preferences reward agreement more than truth, so factuality tuning uses AI-generated preference labels with DPO — cutting hallucinations with no human annotation at all.

Check yourself

Three exercises in the lecture’s spirit — each is a derivation you can do on paper.

Problem G.1 reward degree of freedom

Consider the Bradley–Terry reward-modeling loss $-\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$ . (a) Show that replacing $r_\phi(x, y)$ with $r_\phi(x, y) + c(x)$ , for an arbitrary per-prompt function $c$ , leaves the loss unchanged, and explain why this is worse than a single global shift. (b) Show that DPO’s implicit reward $\hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$ does not have this freedom.

Show solution

(a) Both responses in a pair share the same prompt, so the difference becomes $r_\phi(x, y_w) + c(x) - r_\phi(x, y_l) - c(x) = r_\phi(x, y_w) - r_\phi(x, y_l)$ : the shift cancels inside $\sigma$ for every training pair, so the loss — hence the fitted reward model — cannot pin down $c$ . Because $c$ can differ per prompt, rewards are only comparable within a prompt, not across prompts; a value function must predict expected reward across many states, so an arbitrary per-state offset makes value fitting in the PPO stage ill-posed. (b) Shifting $\hat{r}_\theta(x, \cdot)$ up by a constant $c$ for all responses to prompt $x$ would require multiplying $\pi_\theta(y \mid x)$ by $e^{c/\beta}$ for every $y$ . But $\pi_\theta(\cdot \mid x)$ is a normalized distribution — its probabilities must sum to one — so you cannot raise the probability of everything at once. Normalization spends exactly the degree of freedom that the unconstrained reward model had left over.

Problem G.2 deriving the DPO loss

Starting from the closed-form optimal policy $\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp(\frac{1}{\beta} r(x, y))$ , derive the DPO objective. Identify exactly where the intractable partition function $Z(x)$ disappears, and state the assumption that makes the cancellation work.

Show solution

Take logs and solve for the reward: $r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$ . Parameterize the reward this way with a trainable $\pi_\theta$ in place of $\pi^*$ , and substitute into the Bradley–Terry loss $-\log \sigma(r(x, y_w) - r(x, y_l))$ . The difference is

$\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} + \beta \log Z(x) - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} - \beta \log Z(x)$

The two $\beta \log Z(x)$ terms cancel because both responses in a preference pair answer the same prompt $x$ — that is the load-bearing assumption. What remains is $\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$ , which involves only log-probabilities of observed sequences under $\pi_\theta$ and the frozen reference — every term computable with forward passes, no sampling. Minimizing it fits the same Bradley–Terry model as before, but the “reward model” is now constrained to be one whose optimal policy is $\pi_\theta$ itself, so reward learning and policy optimization happen in one step.

Problem G.3 what DPO cannot learn

Every chosen and every rejected response in a summarization dataset begins with the same three tokens. The autoregressive log-likelihood of a response decomposes as $\log \pi_\theta(y \mid x) = \sum_t \log \pi_\theta(y^{(t)} \mid x, y^{(1..t-1)})$ . (a) Show from the DPO gradient that the parameters receive no learning signal about generating those three tokens as the opening of a response. (b) Explain why the standard pipeline is robust to this in practice, and why the failure appeared in Mitchell’s experiments.

Show solution

(a) The gradient’s direction term is $\nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x)$ , and each log-likelihood splits into per-time-step conditionals. For $t = 1, 2, 3$ the chosen and rejected responses have identical tokens and identical preceding contexts (the prompt plus the same earlier tokens), so the two gradient contributions at those steps are the same term with opposite signs and cancel exactly. The per-example sigmoid weight multiplies the whole bracket and cannot resurrect a term that is identically zero. DPO trains only on where the pair differs. (b) In the standard pipeline the policy is initialized from an SFT model that was trained by plain maximum likelihood on demonstrations — SFT learns the shared structure (including the opening tokens), and DPO only needs to learn the delta between better and worse responses. The failure surfaced when the reference/initialization had not already internalized the shared prefix, so nothing in the DPO stage could teach it. The general lesson: any structure common to both elements of every preference pair is invisible to DPO, which is why the SFT stage is not optional.