Guest lecture · Part 7

Aviral Kumar

Guest lecture by Aviral Kumar on offline RL as a pre-training and fine-tuning paradigm rather than a train-and-deploy pipeline. Three obstacles, three fixes. An implicit-regularization analysis explains why Q-learning fails to scale with model size and yields a regularizer that restores scaling; value functions trained on action-free human video give robots useful pre-trained representations; and calibrating pessimistic Q-values against the behavior policy's returns eliminates the unlearning dip when fine-tuning online.

▶ Lecture video

Offline RL as pre-training, not train-and-deploy

Aviral Kumar needs little introduction in this course — Levine notes that much of the offline RL lectures covers work he co-authored, and the RL theory lecture “was just copied from a lecture that he made.” His subject here: offline reinforcement learning for pre-training and utilizing large models — large models “not in the context of NLP or language but for robotics, for many decision-making problems.”

The standard offline RL paradigm you have studied maps directly onto the classic ML pipeline: take training data, fit a model, deploy it on the real task. But modern machine learning increasingly follows a different picture. You pre-train a general model on lots and lots of data — much of it not very related to your downstream task — and then fine-tune it on the task you actually care about. Large language models are the canonical example. Kumar’s question is how offline RL can realize the same paradigm for decision-making: ingest not just your task’s data but “all possible relevant data for your problem” — robot data, gameplay data, hospital decision-making data — and produce not merely reward-maximizing policies but good initializations, features, and representations to fine-tune.

Realizing that picture requires three components, which structure the talk: scaling (making offline RL amenable to large models), data (learning from arbitrary sources, concretely human video for robots), and algorithms (fine-tuning methods that improve a pre-trained initialization with limited task-specific data). Throughout, the dataset is a replay buffer of transition tuples $(x_i, a_i, r_i, x_{i+1})$ : observation, behavior-policy action, instantaneous reward, next observation.

A scaling stress test: one policy, many Atari games

Does offline RL scale if you just throw more data at a bigger model? To find out, Kumar’s group made the standard Atari domain harder in a “baby step” way: instead of one policy per game, train a single policy to play multiple games together. Concatenating the per-game datasets gives about two billion highly suboptimal transitions — a genuinely challenging problem, with ResNet encoders mapping pixels to discrete actions.

The result was counterintuitive. Imitation-learning-style methods (supervised learning on the data, including filtered imitation) got a huge performance boost from switching to a larger model. Off-the-shelf offline RL — conservative Q-learning — showed only a small gain from added capacity, and with enough scaling “your performance might even start to degrade when you use a larger model.” Supervised learning scales; Q-learning, naively, does not.

Implicit regularization: the delta between SGD and TD

Rather than seek an absolute explanation, Kumar asks a comparative question: if we can characterize the difference between supervised learning and offline RL at scale, maybe we can compensate for it in the algorithm.

The crucial structural difference in Q-learning: the regression targets are not constants. A supervised regression fits fixed labels; TD learning regresses $Q_\theta$ toward target values computed from a previous copy of the same Q-function, “creating a cyclic loop” in the learning dynamics. Writing $\phi_\theta(x, a)$ for the last-but-one layer features of the Q-network, the implicit regularizer can be derived for both cases:

The implicit regularizer of TD learning fights itself

For supervised regression, under suitable modeling assumptions, $R(\theta)$ penalizes the norm of the learned features — “kind of like weight decay, but not on weights but on features instead,” which is benign. For TD learning the regularizer picks up a second term, schematically

$R_{\text{TD}}(\theta) \;=\; \underbrace{\textstyle\sum_i \big\lVert \phi_\theta(x_i, a_i) \big\rVert^2}_{\text{feature-norm penalty}} \;-\; \gamma \underbrace{\textstyle\sum_i \phi_\theta(x_i, a_i)^\top \phi_\theta(x_{i+1}, a_{i+1})}_{\text{dot product of consecutive features}}$

The second term comes “precisely from the target value,” which evaluates the Q-function at $x_{i+1}$ while the prediction is evaluated at $x_i$ . Because it enters with a minus sign, minimizing $R_{\text{TD}}$ maximizes the dot product of consecutive features — and an easy way to maximize a dot product of unbounded vectors is to grow their lengths. The two terms conflict: one shrinks feature magnitudes, the other inflates them. (Precisely, the regularization effect holds within a band around this solution whose width depends on the learning rate and gradient noise.)

The fix is unapologetically empirical: if the second term is the delta between supervised learning and RL, undo it. Add the dot-product term back as an explicit loss so that the net regularization matches supervised learning. With this one change, the same conservative Q-learning setup goes from degrading with capacity to benefiting from it — the scaling curve “now looks very similar to how the performance of supervised learning scales.” On the multi-game benchmark, CQL plus this regularizer beat imitation, decision-transformer-style filtered imitation, and average behavior-policy replay — the first Q-learning-based offline RL approach to surpass the behavior policy there with a large enough model. In Q&A, Kumar notes the term can also be implemented as a normalization layer inside the model, removing the regularization weight hyperparameter, and that they never found the correction harmful.

Pre-training on human video: value functions without actions

With robot data alone, an offline RL pre-training recipe already exists: pool multi-task demonstration data, annotate the last state of each rollout with reward $+1$ , condition on a one-hot task ID, run offline RL, then fine-tune — mixing pre-training batches with target-task batches — on the limited data for the task you care about.

But robot data is tiny next to human video on YouTube or in curated datasets like Ego4D. Video shows how humans interact with the world — and has no actions, plus a large embodiment gap (five fingers versus a parallel-jaw gripper). Kumar’s answer: you can still run value-based offline RL on video, not to learn policies but to learn representations. A value function $V(x)$ is a function of the observation only — no actions required — and training it with Bellman backups bakes the sequential, dynamical structure of the world into the visual encoder, exactly the structure the downstream Q-function and policy need.

Which value function, though? Kumar walks a spectrum. A hand-designed reward (“hand close to object”) is too specific to one capability. Goal-reaching values for a single policy $\pi$ are still too specific — features that represent one policy’s values. Values for all possible policies and all goals are overly broad: real robots “are not going to run any arbitrary sequence of random actions.” The formulation that “strikes a good balance between breadth and specificity” models values only for optimal goal-reaching policies, over all goals — any frame of any video can serve as a goal. This is inspired by the intent-conditioned value function line of work.

The full pipeline (the V-PTR system): train this value function on video; take the resulting visual encoder; initialize an off-the-shelf offline RL method (conservative Q-learning) on multi-task robot data; fine-tune on the target task. The learned features behave like values should: plotted against trajectory time step, the predicted values increase near-monotonically — even on out-of-distribution trajectories, where baselines trained without video or with other video objectives go non-monotonic — and they achieve the smallest mean-squared error against ground-truth values. On real robots, the fine-tuned policies were noticeably smoother and more robust to object variation and distractor objects than self-supervised video representations (masked autoencoding, contrastive learning) or no video at all.

Fine-tuning without unlearning: calibrate the Q-values

The last piece is the algorithmic one: you have an offline-pre-trained policy and a limited budget of online interaction to specialize it. The naive approach — keep running the same offline algorithm, appending online data to the buffer — fails in two distinct ways. Conservative Q-learning suffers a sharp dip in performance right at the start of fine-tuning before recovering; implicit Q-learning improves steadily but with a much lower slope. Neither matches the curve you want: fast improvement, no dip.

Worked example — Diagnosing the dip

Among many metrics, one tracks the dip exactly: the average Q-value on the dataset undergoes a sharp scale correction — “your values are minus 40 and now your values become minus 10” — precisely when the return crashes. The mechanism: the TD error is a relative loss (fit your own targets), so many Q-functions of similar shape attain the same TD error; but CQL’s pessimism term is an absolute loss that pushes Q-values down on out-of-distribution actions, so among all those solutions it finds the one with the smallest magnitude. When online exploration then executes an action and observes its actual ground-truth reward — far above the deflated scale — updating on it introduces an erroneous peak. Policy optimization dutifully finds the peak of the learned Q-function, which is no longer the peak of the true one: the policy switches to a bad action, and a previously good policy unlearns.

The fix — the Cal-QL paper — is a one-line constraint: never let the learned Q-values fall below a reference Q-function, so the pessimistic solution is lower-bounded and a single observed real reward can no longer tower over it. A convenient reference is the Q-function of the behavior policy itself, computable from Monte Carlo return-to-go estimates on the dataset with no Q-learning at all. With calibration, fine-tuning curves improve from the start with no dip, beat IQL’s slope, tolerate more gradient steps per sample, and achieve the smallest cumulative regret — the standard sample-efficiency measure — among the methods compared. On a real robot opening a microwave door, the offline initialization only reaches the handle; 20K steps of fully online fine-tuning yields reliable opening — feasible only because nothing is unlearned along the way.

Back of the chalk tray — what to remember

Offline RL can mirror the modern ML paradigm — pre-train a generalist on all relevant data, then fine-tune — instead of train-and-deploy; the obstacles are scaling, arbitrary data sources, and fine-tuning algorithms.
Naive offline Q-learning fails to benefit from model capacity (and can degrade), while imitation-style supervised learning scales — demonstrated on a two-billion-transition multi-game Atari benchmark.
SGD’s implicit regularizer explains the gap: TD targets computed from the network’s own previous copy add a term that inflates feature norms; explicitly undoing that term restores supervised-like scaling and lets CQL surpass the behavior policy.
Action-free human video can pre-train robot policies through value functions, which need no actions; values for optimal goal-reaching policies over all goals balance breadth against specificity, and the resulting encoders transfer to real-robot control.
Fine-tuning a pessimistic offline policy online causes unlearning because pessimism selects the smallest-magnitude Q-function fitting the TD error; observed rewards then create false peaks that the policy chases.
Cal-QL’s fix is to lower-bound learned Q-values by the behavior policy’s Monte Carlo return-to-go — no dip, faster improvement, smallest cumulative regret.

Check yourself

Three exercises in the talk’s spirit — work them on paper.

Problem G.1 implicit regularizer conflict

The implicit regularizer of TD learning contains $-\gamma \sum_i \phi_\theta(x_i, a_i)^\top \phi_\theta(x_{i+1}, a_{i+1})$ alongside a feature-norm penalty. (a) Explain why SGD implicitly minimizing this term tends to blow up feature magnitudes, and why the analogous term is absent in supervised regression. (b) State the empirical fix and what it accomplishes, and explain in what sense the fix is justified comparatively rather than provably.

Show solution

(a) Minimizing a term with a leading minus sign means maximizing the dot product $\phi_\theta(x_i, a_i)^\top \phi_\theta(x_{i+1}, a_{i+1})$ ; since the feature vectors are unbounded, the easiest way to grow a dot product is to grow the vectors’ lengths, so this term pushes feature norms up — in direct conflict with the norm penalty, which pushes them down. The term originates in the regression target: TD learning fits $Q_\theta(x_i, a_i)$ to a target computed from a previous copy of the same network evaluated at $x_{i+1}$ , coupling consecutive features. Supervised regression fits constant labels, so no cross-step feature coupling — and no such term — appears; its regularizer is a pure feature-norm penalty, akin to weight decay on features, which is benign at scale. (b) Add the dot-product term back as an explicit loss (or a normalization layer, which removes the weighting hyperparameter), so the net regularization matches supervised learning’s. This restored capacity scaling for CQL on multi-game Atari. The justification is comparative: supervised learning demonstrably scales and RL demonstrably didn’t, so the analysis identifies the delta between the two regularizers and the fix undoes it — Kumar is explicit that this step is empirical, not something he can provably show.

Problem G.2 value pre-training design

You want to pre-train a robot’s visual encoder on action-free human video. (a) Why can a value function be trained on such data when a Q-function cannot, and why should value training produce useful features at all? (b) Rank these three choices from too specific to too broad, and state the formulation that strikes the balance: (i) goal-reaching values for all possible policies, (ii) values for one hand-designed reward and one policy, (iii) goal-reaching values for a single fixed policy.

Show solution

(a) A Q-function $Q(x, a)$ takes the action as input, and video contains no action labels; a value function $V(x)$ is a function of the observation alone, so it can be trained on raw frames. It should yield useful features because Bellman backups force the encoder to capture the sequential, dynamical structure of the world — how scenes evolve toward goals — which is precisely the structure the downstream Q-function and policy (similar kinds of objects) must represent. (b) From specific to broad: (ii) is most specific — one reward, one capability, useless for the many other downstream tasks; (iii) is next — arbitrary frames as goals give many reward functions, but features still only represent one policy’s values; (i) is overly broad — it models values for arbitrary random action sequences a real robot will never execute. The balanced choice models goal-reaching values for optimal goal-reaching policies over all goals (the intent-conditioned value function idea): every reward function in the goal-reaching family, but only the useful, efficient policies.

Problem G.3 unlearning mechanism

A CQL-pre-trained policy achieves 50 percent success. Online fine-tuning begins, and performance crashes before recovering. (a) Using the fact that the TD error is a relative loss while CQL’s pessimism penalty is absolute, explain the crash step by step. (b) Explain how constraining $Q_\theta \ge Q^{\beta}$ (the behavior policy’s Q-function) prevents it, and why $Q^{\beta}$ is cheap to obtain. (c) Why does this matter more, not less, when the online budget is small — say 20K steps on a real robot?

Show solution

(a) Many Q-functions of the same shape attain the same TD error, since each fits targets from its own previous copy; the pessimism penalty is an absolute push down on out-of-distribution actions, so training selects the solution with the smallest Q-value magnitude — values far below the ground truth (e.g. minus 40 where minus 10 is right). When online exploration executes an action, the observed actual reward is far above this deflated scale, so the update creates an erroneous peak in the learned Q-function at that action. Policy optimization finds the peak of the learned Q — previously aligned with the true optimum, now an artifact — so the policy switches to a genuinely bad action and performance crashes until the overall Q-scale corrects. (b) Lower-bounding by a reference Q-function keeps the learned values calibrated in scale, so one observed true reward produces only a small local bump that cannot exceed the existing maximum; the policy never abandons the good action. The behavior policy’s Q-function is computable directly as Monte Carlo return-to-go along dataset trajectories — no Q-learning needed. (c) With 20K steps there is no budget to dip and recover: unlearning early would consume the whole interaction budget climbing back to the starting point. The microwave-door robot works precisely because the policy improves monotonically from its offline initialization.