Transfer Learning & Meta-Learning

Why Montezuma’s Revenge is a transfer problem

Some Atari games fall to the Q-learning algorithm you implemented in homework 3; Montezuma’s Revenge “really struggles to get past even the first level.” The difference is prior knowledge. The reward structure provides little guidance — a little reward for the key, a little for the door, and “getting killed by a skull is bad, but you don’t actually get any negative reward for it” — yet we might not even watch the score: we know skulls denote bad stuff, keys open doors, ladders can be climbed. When we learn this game, “we’re essentially doing a kind of transfer learning” — transferring what we know about the world into a new MDP. Perhaps the right response to hard-exploration domains is not a better de novo exploration algorithm but one that transfers knowledge from tasks already solved.

What could that knowledge be? A Q-function tells you which states and actions are good — but an Indiana Jones Q-function may not transfer to a game where you move by pushing buttons rather than legs. A policy tells you which actions are potentially useful; some actions are never useful. A model might transfer when the laws of physics agree across domains. And features or hidden states — a good skull, ladder, and key detector — can carry surprisingly far: “don’t underestimate it,” a little visual pre-training can go a long way.

There is no single problem statement here — much of the area is ad hoc, tied to the particular source/target pair. The lecture covers three framings: forward transfer (train on one source task, run or fine-tune on a target), multitask transfer (train on many source tasks so the target sits, intuitively, within their convex hull), and meta-learning, where you train on the source domains in a way that is aware you will be adapting to a new domain later — learning to learn.

Forward transfer: fine-tuning and the domain-shift problem

The pre-train-then-fine-tune workflow from vision and NLP imports directly: learn representations with RL or supervised learning, fine-tune a few layers with RL on the target task. The interesting part is what goes wrong. First, domain shift: representations learned in simulation may not work on real images. Second, the dynamics themselves may differ. Third, fine-tuning in RL has a peculiar failure mode we’ll get to shortly.

The standard way to impose invariance goes under names like domain confusion or domain-adversarial neural networks: pick a middle layer (often right after the convolutions), train a binary classifier $D_\phi(z)$ that predicts which domain the activations came from, then reverse its gradient into the network — teach it to produce a $z$ the classifier cannot tell apart. This needs example images from the target domain, but no target-domain RL. One warning: if your target-domain data comes from very bad drivers while your source-domain RL drives well, the representation becomes invariant not just to sim-versus-real but to good-versus-bad — “that will really mess up your Q-learning algorithm.”

When the dynamics differ, invariance is the wrong tool — you don’t want to ignore functionally relevant differences. Instead, change the behavior: punish the agent for doing things in the source domain that would be impossible in the target domain, via the reward modification

$\Delta r(s_t, a_t, s_{t+1}) = \log p_{\text{target}}(s_{t+1} \mid s_t, a_t) - \log p_{\text{source}}(s_{t+1} \mid s_t, a_t)$

estimated without a dynamics model by training two discriminators — one on $(s, a, s')$ tuples, one on $(s, a)$ — and taking their difference. Levine’s metaphor is the sailor in the film who reaches the edge of the green screen: the term prevents you from violating the illusion that you are in the target domain. The caveat: this learns, intuitively, the intersection of the two domains — it stops you exploiting source-only shortcuts, but cannot give you skills the source domain doesn’t permit.

Finally, the RL-specific fine-tuning pathology. Pre-training in vision and NLP draws on enormously broad distributions; RL source tasks are usually narrow, so features are less general and policies over-specialized. Worse, the optimal policy in a fully observed MDP can be deterministic — the longer you train, the more deterministic the policy becomes, and a deterministic policy deployed on the target task doesn’t explore. Loss of exploration at convergence makes naive fine-tuning extremely slow. The remedies are pre-training procedures that yield more diverse solutions: maximum-entropy RL (control-as-inference lecture) and unsupervised skill discovery (exploration II lecture). There isn’t one general principle yet — “maybe borrow some of the ideas” from those lectures is, candidly, the state of the art.

Manipulate the source domain: randomization

If you control the source domain, the single most reliable lever is variety: the more varied the training domain, the more likely zero-shot generalization to a slightly different domain. In practice this means randomization — intentionally adding more randomness to the simulator, perhaps more than the real world has.

This idea — applied to physical parameters, to visual appearance for drone flight, with recurrent policies that adapt online — has been extremely influential, especially in robotics; the ETH Zurich locomotion work shows a policy trained entirely in a heavily randomized simulator achieving fairly extreme physical robustness.

Multitask transfer and contextual policies

Training on many source tasks accelerates all of them — if one task figures out how to pick up an object, the shared representation hands that to the others — and gives a new task more to latch onto. And it requires no new algorithm, because multitask RL is single-task RL in a joint MDP: just make the first step of the episode sample which MDP you’re in, as part of $p(s_1)$. When the state alone doesn’t reveal the task (the same kitchen admits laundry or dishes), append a task context $\omega$ — a one-hot indicator, a goal image, a language description — to the state:

$\theta \leftarrow \arg\max_\theta \, \mathbb{E}_{\omega \sim p(\omega)}\!\left[\mathbb{E}_{\tau \sim \pi_\theta(\tau \mid \omega)}\left[r_\omega(\tau)\right]\right]$

with the contextual policy $\pi_\theta(a \mid s, \omega)$, and $\omega$ fixed for the episode after being drawn at the first step. A particularly common choice is the goal-conditioned policy, where $\omega$ is another state $g$ and the reward is an indicator (or a small ball) around reaching it — no per-task reward engineering, and zero-shot transfer to any new task that is a goal. The costs: goal-conditioned RL is a genuinely hard RL problem in practice, and not every task is goal-reaching (reach the green region while avoiding the red one is not a function of a single goal state).

Meta-learning: learning to learn

If you’ve learned 100 tasks, can you learn the 101st more efficiently? A meta-learned RL procedure might explore more intelligently, skip actions it knows are never useful, and acquire the right features more quickly. The clean way to see what “learning to learn” means is the supervised recipe: instead of a training set and a test set, you have a set of training sets and a set of test sets (meta-training is the source domain; meta-testing the target). Regular supervised learning is a function $f(x) \to y$; supervised meta-learning is a function $f(\mathcal{D}^{\text{train}}, x) \to y$ that reads in an entire small training set plus a test input.

Formally, generic learning computes $\theta^\star = \arg\min_\theta \mathcal{L}(\theta, \mathcal{D}^{\text{train}})$ — call that procedure $f_{\text{learn}}$. Generic meta-learning trains a learned learner:

$\theta^\star = \arg\min_\theta \sum_{i} \mathcal{L}\!\left(\phi_i, \mathcal{D}_i^{\text{test}}\right), \qquad \phi_i = f_\theta\!\left(\mathcal{D}_i^{\text{train}}\right)$

$f_\theta$ ingests a training set and emits task parameters $\phi_i$ — everything you need to know about that training set to do well on its test set. The simplest instantiation: an RNN reads the $(x, y)$ pairs, its hidden activation $h_i$ plus the weights $\theta_p$ of a small classifier head constitute $\phi_i = (h_i, \theta_p)$, and “learning” a new task is just a forward pass.

Meta-RL as a history-conditioned policy

Transplant the definitions: regular RL computes $\theta^\star$ maximizing expected reward — a function $f_{\text{RL}}(\mathcal{M})$ of an MDP. Meta-RL trains

$\theta^\star = \arg\max_\theta \sum_i \mathbb{E}_{\pi_{\phi_i}(\tau)}\left[R(\tau)\right], \qquad \phi_i = f_\theta(\mathcal{M}_i)$

with meta-training MDPs $\mathcal{M}_i \sim p(\mathcal{M})$ and a test MDP drawn from the same distribution — like supervised learning, meta-learning only works in-distribution. The tasks might be a robot’s chores, or the half-cheetah running at different speeds. $f_\theta$ must do two things: improve the policy using experience from $\mathcal{M}_i$, and choose how to interact — exploration is now part of the learned procedure, which supervised meta-learning never had to face.

The black-box instantiation: an RNN reads the transitions $(s, a, s', r)$ — crucially, across episode boundaries, without resetting the hidden state — and its hidden vector feeds a policy head $\pi(a \mid s, h)$. This is exactly a contextual policy, except the context is inferred from experience rather than given: $\phi_i$ is the hidden state plus the head’s weights.

People have built this with actor-critic and policy-gradient learners, and with attention, temporal convolutions, and — more recently — Transformers in place of the RNN; the principle is unchanged.

Gradient-based meta-RL and the POMDP view

Is pre-training and fine-tuning itself a type of meta-learning — and can we meta-train so that fine-tuning works well? That is model-agnostic meta-learning (MAML): let $f_\theta$ be an RL algorithm, whose “parameters” are just the initial weights it starts from:

$f_\theta(\mathcal{M}_i) = \theta + \alpha \nabla_\theta J_i(\theta)$

Viewed through the supervised lens, $f_{\text{MAML}}(\mathcal{D}^{\text{train}}, x) = f_{\theta'}(x)$ with $\theta'$ one gradient step away — “just another computation graph,” another architecture for $f$, with gradient descent inside the network. Why prefer it? Its inductive bias: insofar as policy gradients are a good learning algorithm, the adaptation procedure inherits that, and in practice you can take many more gradient steps at meta-test time than you meta-trained for. An RNN has no such recourse — its learning process is a forward pass, and there is no notion of running it longer.

The third perspective: meta-RL is a POMDP. Write the policy as $\pi_\theta(a \mid s, z)$, where $z$ encapsulates what the policy needs to know about the current task; learning the task is inferring $z$. Augment the state to $\tilde{s} = (s, z)$, with $s$ observed and $z$ never observed: solving this POMDP is equivalent to meta-learning. Policies-with-memory solvers recover the RNN meta-learners; explicit state estimation gives something new. Train a variational inference network $q(z \mid s_{1:t}, a_{1:t}, r_{1:t})$ and explore by posterior sampling: sample $z$ from your current belief, act as if that task were correct, gather data, re-infer, repeat. PEARL instantiates this with an off-policy soft actor-critic, maximizing expected reward under sampled $z$ while a KL term $D_{KL}(q(z \mid \cdot) \,\|\, p(z))$ keeps $z$ minimally informative. The encoder can be strikingly simple: featurize each transition and average the features (a Q-learning-style learner doesn’t care about transition order, so the encoder needn’t either), outputting the mean and variance of the $z$ posterior. On a 2D point-mass whose goal lies somewhere on a semicircle, the agent visits random points on the semicircle until it hits reward, then returns there — good, but not optimal: an optimal explorer would sweep the semicircle within a single episode. Posterior sampling is provably decent but not the best possible exploration, in meta-learning as elsewhere.

The perspectives converge. The RNN is conceptually simple but vulnerable to meta-overfitting — a slightly out-of-distribution test task yields a bad hidden state with no recourse. Gradient-based meta-learning extrapolates (keep taking gradient steps) but needs many meta-training samples and resists extension to value-based methods — temporal-difference learning “is not really gradient descent.” The inference view buys principled exploration and an elegant POMDP reduction, but shares the RNN’s meta-overfitting and optimization headaches. And they are nearly the same algorithm: inference is the RNN with stochastic hidden variables ($\phi$ is $z$); MAML is a particular architecture for the same $f$; add noise to MAML’s gradients and it starts to look like the inference procedure. A closing note at the intersection with cognitive science: humans and animals seem to mix model-free RL, episodic recall, and model-based reasoning, and one hypothesis is that these are emergent properties of an overarching learned learner — papers have shown meta-RL giving rise to episodic learning, model-free meta-training producing model-based adaptation, and even behavior resembling causal reasoning.

Check yourself

Conceptual and derivation exercises in the lecture’s spirit — work them before reading the solutions.

Problem 22.1 invariance assumption limits

A driving simulator renders no rain; the real world has rain, and rain genuinely changes optimal driving (slower speeds, longer braking distances). You train with a domain-adversarial loss so that a mid-network layer $z$ satisfies the invariance conditions. (a) Which part of the invariance assumption fails, and what is the consequence for the transferred policy? (b) Separately, your real-world images all come from dashcams of unusually bad drivers, while your simulated policy drives well. What additional failure does the adversarial loss introduce?

Show solution

(a) The assumption requires some $z = f(x)$ with $p(z)$ equal across domains and $p(y \mid z) = p(y \mid x)$. Rain-relevant information cannot appear in $z$ — if it did, $p(z)$ would differ between domains, since rain occurs only in the target. So the adversarial loss forces $f$ to discard rain, and $p(y \mid z)$ loses information relative to $p(y \mid x)$: the invariant representation is lossy for the label. The transferred policy drives identically in rain and shine — it mostly works, but is strictly worse than a policy that could perceive rain, and nothing in the source domain can fix this because the simulator cannot render what was discarded. (b) The classifier can now distinguish domains not only by rendering artifacts but by behavior quality visible in the observations (positions relative to lanes, closing speeds). Making $z$ domain-indistinguishable therefore also makes it invariant to good-versus-bad driving — exactly the features a Q-function needs to rank actions. This is the lecture’s warning: the nature of the target-domain data really does affect whether the trick helps, and behaviorally skewed target data can corrupt the representation for value learning.

Problem 22.2 dynamics-mismatch reward

A gridworld target domain has a wall between start and goal; the source simulator lacks the wall. Consider the transition (state $s_w$ adjacent to the wall, action “move through the wall,” next state $s'$ on the far side). Using $\Delta r = \log p_{\text{target}}(s' \mid s, a) - \log p_{\text{source}}(s' \mid s, a)$: (a) evaluate $\Delta r$ for this transition and for an ordinary open-corridor transition possible in both domains; (b) explain what behavior the modified reward induces in the source domain and why the lecture describes the result as learning the intersection of the domains; (c) name a situation this fix cannot handle.

Show solution

(a) Through-the-wall: $p_{\text{source}}(s' \mid s_w, a)$ is high (say near 1) but $p_{\text{target}}(s' \mid s_w, a) = 0$, so $\Delta r = \log 0 - \log 1 = -\infty$ — in practice a very large negative reward from the discriminator-based estimate. Open-corridor: both probabilities are (near) equal, so $\Delta r \approx \log 1 = 0$; transitions equally likely in both domains are neither punished nor rewarded. (b) The agent is heavily penalized for any transition that is possible in the simulator but impossible in reality, so it routes around where the wall would be despite the wall not existing in its world — it avoids violating the illusion that it is in the target domain. It can only ever exploit dynamics available in both domains, hence the intersection. (c) The converse gap: if the target domain permits or requires transitions the source domain cannot produce (a skill the simulator physically cannot exercise), no reward shaping of source behavior can teach it — the intersection excludes it, and only target-domain fine-tuning can help.

Problem 22.3 meta-episode exploration

A meta-RL task distribution: two-armed bandits with two-episode meta-episodes, one pull per episode. In each sampled bandit, one arm (equally likely to be arm 1 or arm 2) pays $+1$ deterministically; the other pays $0$. Compare (a) the best memoryless policy $\pi(a)$, (b) the best history-conditioned policy trained to maximize total meta-episode reward, computing each one’s expected return over the meta-episode, and (c) explain which component of the RNN meta-learner is doing the work that (a) lacks.

Show solution

(a) A memoryless policy pulls each episode independently. Any fixed or stochastic choice hits the paying arm with probability $\tfrac{1}{2}$ per pull (the paying arm is symmetric across tasks), so the expected meta-episode return is $\tfrac{1}{2} + \tfrac{1}{2} = 1$. (b) A history-conditioned policy: pull arm 1 in episode one (expected reward $\tfrac{1}{2}$). If it paid, pull it again (reward 1); if it paid nothing, the history reveals the other arm pays, so switch (reward 1). Episode two earns 1 deterministically, for a total of $\tfrac{1}{2} + 1 = \tfrac{3}{2}$. (c) The hidden state carried across episode boundaries. Because the RNN is not reset between episodes and the objective is the total reward of the concatenated meta-episode, the first pull functions as exploration whose information the second pull exploits — maximizing meta-episode reward with a history-dependent representation recovers the try-then-switch strategy automatically, which is precisely the lecture’s claim that meta-RL learns to explore without any explicit exploration bonus.

Problem 22.4 comparing the three perspectives

For each scenario, say which meta-RL family (black-box RNN/Transformer, gradient-based MAML, or variational inference à la PEARL) is best suited, and justify from the properties discussed in lecture: (a) meta-test tasks are mildly outside the meta-training distribution, and you can afford extra environment interaction at test time; (b) you want an off-policy, sample-efficient meta-learner with a principled exploration story; (c) you want the conceptually simplest method and your tasks are safely in-distribution. Then: why is MAML hard to combine with value-based (TD) methods?

Show solution

(a) MAML. Its adaptation procedure is genuine policy-gradient fine-tuning, so at meta-test time you can keep taking gradient steps — more than you meta-trained with — and extrapolate toward out-of-distribution tasks. An RNN’s “learning” is a single forward pass: if the hidden state is bad, “there’s no recourse past that.” (b) PEARL-style inference. It meta-trains with an off-policy soft actor-critic (sample-efficient), and its stochastic latent $z$ gives exploration by posterior sampling — sample a task hypothesis from the belief, act as if it were true, refine. (Good, though not optimal: the point-mass example shows an optimal explorer would sweep the goal semicircle within one episode rather than trying random hypotheses.) (c) The black-box RNN/Transformer: condition a policy on the full cross-episode history and run regular RL — conceptually the simplest, at the price of meta-overfitting and sometimes tricky optimization. Finally, MAML’s outer loop must differentiate through the inner learning update, which works when the inner update is a gradient step on a well-defined objective; temporal-difference learning “is not really gradient descent” — its bootstrapped target is not the gradient of any fixed loss — so there is no clean second derivative to backpropagate through.