Introduction

A problem supervised learning can’t touch

The lecture opens with a robot that sees a bin of objects through its camera and must output coordinates for a successful grasp. This looks like object localization, but grasping is riddled with special cases — rigid objects are easy, an awkwardly shaped object must be gripped near its center of mass, a soft deformable one calls for pinching — and so many exceptions make it exactly the kind of problem you’d hand to machine learning. Except supervised learning needs images paired with correct grasp locations, and even people can’t produce those labels: grasp points are “really a property of the physical interaction between the robot and its environment,” and, to put it simply, “we don’t have a lot of experience picking things up with robot fingers.”

The alternative is to let the robots collect the dataset themselves — many attempted grasps, each labeled not with the right answer but with its outcome, a reward for success. A reinforcement learning algorithm doesn’t copy what’s in the data; it uses the reward labels to figure out what to do to maximize successes, and can end up with a policy better than the average behavior that collected the data. That, in essence, is the main idea behind reinforcement learning.

Two lineages, and what RL does differently

The recent, headline-grabbing advances in AI — diffusion models painting Salvador Dalí as a half-robot, language models explaining jokes, generative models designing proteins — are, under the hood, massively scaled-up density estimation: learning $p_\theta(x)$ or $p_\theta(y \mid x)$ from enormous datasets. What you learn is the distribution in the data — the kinds of images people put on the web, the kind of text people type. That’s exactly right if the goal is to generate content similar to what humans would have generated; it is not the only thing we want from autonomous systems.

Modern RL traces its lineage to two disciplines. One is psychology and the study of animal behavior — B.F. Skinner and reinforcement, which became the computer-science picture of an agent adapting to its environment in response to rewards. The other is controls, optimization, and evolutionary computation — Karl Sims’s 1994 virtual creatures, whose forms and behaviors were optimized rather than designed, through Yuval Tassa’s trajectory-optimized humanoids that figure out how to walk and run. Deep reinforcement learning is the merger: classical RL’s algorithmic foundations scaled up with modern large-scale optimization.

The reinforcement learning problem, precisely

Supervised learning is given $\mathcal{D} = \{(x_i, y_i)\}$ and learns $f_\theta(x_i) \approx y_i$. Two assumptions ride along so naturally we rarely state them: the data is independent and identically distributed, and every $x$ comes with a known ground-truth label. Reinforcement learning drops both. The data forms a temporal sequence in which previous outputs influence future inputs, and the ground truth is never given — you only learn how good an outcome was. Even those reward labels are hard to interpret: in a sequence ending in failure, you don’t know which choice precipitated it. It wasn’t looking up your grade on CalCentral that caused you to fail the course; it was the exam weeks earlier, and at the time you didn’t realize it. This is the credit assignment problem — the reward may arrive long after the decision that earned it.

Formally, an agent interacts with the world in a loop: at each time step it receives a state $s_t$ (the analog of $x$), outputs an action $a_t$ (the analog of $y$), and the world responds with $s_{t+1}$ and a scalar reward $r_t$. The data — collected by the agent itself, which must decide how to collect it — consists of sequences $(s_1, a_1, r_1, \ldots, s_T, a_T, r_T)$, and the goal is a policy $\pi_\theta(a_t \mid s_t)$ maximizing the cumulative reward $\sum_t r_t$ — not the reward now, which is what makes the reasoning strategic: you might do something unrewarding now to attain higher rewards later. The formulation is very general. Training a dog: actions are muscle contractions, observations sight and smell, reward a treat. A running robot: actions are motor torques, observations sensor readings, reward the running speed. Inventory management: actions are purchases, observations stock levels, reward profit.

What deep RL can do today

RL shines on physically complex tasks where describing how is hard but specifying the reward is easy: a robotic hand hammering in a nail, a quadruped learning to jump obstacles and balance on its hind legs, ETH Zurich’s agile robots clambering over terrain. It finds unexpected solutions — the Atari Breakout strategy (which you’ll implement in your homework) of tunneling through the bricks so the ball bounces around above them. And it scales to the real world: at Everyday Robots, trash-sorting robots practiced in classrooms and real office buildings, learning vision-based skills on the job.

RL also powers generative models. Language models trained on internet data can solve sophisticated problems but are hard to persuade — they complete text, so you must prompt them to index into the right context; RL on human scores trains them to respond the way readers actually want. For image generation, stable diffusion’s mangled “dolphin riding a bike” is scored by a captioning model (LLaVA), the caption’s similarity to the prompt becomes the reward, and over RL iterations the image becomes a genuine dolphin on a bicycle. In chip design, actions place chip parts and the reward is built from cost, congestion, and latency.

Learning and search: the bitter lesson, read correctly

Richard Sutton — who essentially founded RL in computer science — wrote in his 2019 essay The Bitter Lesson that “building in how we think we think does not work in the long run,” and that “the two methods that seem to scale arbitrarily are learning and search.” Many readers take away only “scale”: shovel more data into more GPUs and the algorithm doesn’t matter. But the essay “doesn’t say learning and GPUs, it doesn’t say learning and big data” — it says learning and search. Search, in the technical sense Sutton means, is not A* — it is any optimization process that uses (typically iterative) computation to extract inferences, to reach conclusions beyond what the data states. Learning is what allows you to understand the world; search is what allows you to leverage that understanding for emergent behavior. Data without optimization doesn’t let us solve new problems in new ways; optimization without data is hard to apply outside of simulators. Deep RL is the combination: deep supplies scalable learning from large, complex datasets, and RL supplies the optimization.

Beyond reward maximization

The course is not only about maximizing rewards, because real-world decision making raises problems the basic formalism doesn’t answer. Chief among them: where do rewards come from? In a video game the score is given; for a robot pouring a glass of water, just detecting success is a complex perception problem. A Berkeley exploration paper put it memorably: “as human agents, we are accustomed to operating with rewards that are so sparse that we only experience them once or twice in a lifetime, if at all” — you don’t learn to complete a PhD by trial-and-erroring many PhDs. A cheetah that learned to hunt purely from the reward of eating a caught gazelle — running randomly around the savannah until it stumbles into one — would starve. Real learning draws on richer signals, and the brain’s basal ganglia, a decidedly non-trivial structure, does something sophisticated to manufacture them.

So the course also covers the other forms of supervision RL can exploit. Learning from demonstrations: copy behavior directly (imitation learning, as in NVIDIA’s early end-to-end driving) or infer the intent behind it (inverse reinforcement learning) — this is the hallmark of human imitation, as in the psychology experiment where a child watches an experimenter act inefficiently and fulfills the inferred goal in their own, better way, or the robot that watches a pouring demonstration, infers the objective, and performs the task in new settings. Learning to predict: robots that learn action-conditioned video prediction from self-collected data can be commanded to move objects and even show emergent tool use — improvising a water bottle to sweep trash — and by 2022 diffusion-based video models synthesize crisp, realistic robot and driving futures. And leveraging pre-trained models: RT-2 stacks a language model, a visual-language model, and robot-action fine-tuning, so internet knowledge lets it “put the banana on the answer of the math problem” — something nowhere in the robot data.

One algorithm, maybe

How would you build an intelligent machine? The old answer — enumerate the brain’s modules, program each one, wire them together — might simply be harder than we are smart. The hypothesis that learning is the basis of intelligence offers an easier path: some things we can all do (walking), some things we can only learn (driving — cars weren’t around when our brains evolved), and since we can learn a huge variety of very difficult things, our learning machinery is likely powerful enough for everything we associate with intelligence. A more radical step: perhaps a single flexible algorithm, placed in the right context, implements all of it. The circumstantial evidence — humans learning to see through an electrode array on the tongue, a ferret regaining visual acuity after its optic nerve is rewired into auditory cortex — suggests the neocortex adapts to whatever sensory input it’s given.

What must that single algorithm do? Interpret rich sensory inputs — which demands large, high-capacity models — and choose complex actions — which is what the RL formalism provides. Deep for the learning, RL for the search. There is suggestive neuroscience on both halves: representations in deep networks show statistical similarity to those recorded in brains for vision, audition, and touch, and reward-anticipating percepts acquire firing patterns like the reward itself — exactly what a temporal-difference learner would predict, with the basal ganglia as a plausible reward system. The picture is far from complete: humans learn incredibly quickly where deep RL needs large amounts of data, transfer and reuse remain open, reward specification is unsolved, and the role of prediction is unclear. Those question marks are research space — possibly yours. Levine closes with a quote whose author you should guess before reading the attribution: “Instead of trying to produce a program to simulate the adult mind, why not rather try to produce one which simulates the child’s? If this were then subjected to an appropriate course of education one would obtain the adult brain.” — Alan Turing.

Check yourself

Conceptual exercises in the lecture’s spirit — no algorithms yet, just the framing.

Problem 1.1 casting a problem as RL

You want an automated system to manage the heating and cooling of a large office building to keep occupants comfortable while minimizing energy cost. (a) Cast this as a reinforcement learning problem: propose the states, actions, and rewards. (b) Explain why the i.i.d. assumption of supervised learning fails here. (c) A colleague suggests instead collecting a dataset of thermostat settings chosen by experienced building managers and training a supervised model to imitate them. What can your RL formulation achieve that this supervised approach cannot, even in principle?

Show solution

(a) Observations/states: current temperatures throughout the building, occupancy, time of day, outside weather, current energy prices. Actions: setpoints or commands to heaters, chillers, and vents at each time step. Reward: a combination such as (comfort measure, e.g. negative deviation from target temperature in occupied zones) minus (energy cost this interval) — a scalar received every step, so the objective is the cumulative sum over the day. (b) The data forms a temporal sequence in which previous outputs influence future inputs: the action you take now (e.g., pre-cooling at 7am) changes the thermal state the system will face at 9am, so datapoints are neither independent nor drawn from a fixed distribution — the policy itself shapes the data it sees. There is also delayed reward: an energy-saving decision now may cause discomfort an hour later, a credit assignment problem. (c) The supervised imitator can at best match the average behavior of the building managers in the dataset — it learns the distribution of what humans did, exactly like density estimation. The RL system optimizes the actual objective from outcome labels, so it can discover strategies better than any manager in the data (the same way the grasping robots improve on the average behavior that collected their data, or Kathy Wu’s car regulates traffic in a way human drivers don’t).

Problem 1.2 credit assignment

A hospital deploys a sequential treatment policy: each week it observes a patient’s condition and chooses a treatment, and at the end of twelve weeks it observes whether the patient recovered. Explain (a) why labeling each week’s decision with the final outcome is “difficult to interpret properly” in the sense of the lecture, and (b) why this same difficulty does not arise in supervised image classification.

Show solution

(a) This is the credit assignment problem. The recovery signal arrives only at the end, but the decision that caused the outcome may have occurred at any earlier week — like the student whose failing grade was caused by a poor exam weeks before the grade was posted, not by checking CalCentral. A sequence ending in failure does not tell you which choice in the sequence precipitated the failure; a bad week-2 treatment might doom the patient even though weeks 3 through 12 were handled perfectly, and every one of those good decisions inherits the bad label. The reward labels how good a state or outcome was, not whether the action just taken was good. (b) In supervised classification each datapoint carries its own ground-truth label, produced directly for that input by the same true function for all samples (the i.i.d. and labeled-data assumptions). There is no temporal chain between datapoints, so no question of which earlier prediction caused a later label — the label for one image never depends on the prediction made for another.

Problem 1.3 learning vs. search

Classify each system in Sutton’s terms — does it use learning, search, or both? — and justify each answer in a sentence or two. (a) A language model pre-trained on internet text and sampled directly. (b) AlphaGo. (c) A trajectory optimizer that computes a humanoid’s motion from known equations of motion in a physics simulator. Then explain, using the lecture’s argument, why system (a) will not produce “move 37”-style behavior on its own.

Show solution

(a) Learning only: it extracts patterns from data — density estimation of what people type — but at sampling time performs no optimization toward any objective beyond likelihood under that distribution. (b) Both: learning (networks trained from data and self-play experience give it its understanding of the game) and search (optimization over moves to maximize winning — computation used to extract inferences beyond the data). This combination is exactly why it could produce an emergent, superhuman move. (c) Search only: it optimizes behavior with iterative computation, but nothing is learned from data — the model of the world is hand-written equations of motion. This is the “optimization without data” corner: powerful in simulators, hard to apply in the real world where you can’t write down the dynamics. Finally, (a) alone cannot produce move-37 behavior because pure learning reproduces the distribution in the data: it can index into human knowledge extremely well, but “data without optimization doesn’t allow us to solve new problems in new ways” — going beyond good human behavior requires an objective and a search process that optimizes it, not just a model of what humans typically produce.

Problem 1.4 sparse rewards

The lecture argues that a cheetah learning to hunt purely by trial and error on the reward “meat obtained” would starve. (a) State precisely what makes this reward signal inadequate for learning, in the vocabulary of the lecture. (b) List three alternative sources of supervision from the lecture that a learning system (animal or robot) could exploit instead, and say in one sentence each what the supervision signal is.

Show solution

(a) The reward is far too sparse and delayed: it is received only upon a rare, completed success, so random exploration essentially never encounters it (“rewards so sparse that we only experience them once or twice in a lifetime, if at all”), and when it does arrive, credit assignment across the long sequence of preceding decisions is nearly impossible. A learner relying on that signal alone would need absurdly many trials — the cheetah dies of starvation long before its first random gazelle. (b) Any three of: imitation/demonstrations — observe another agent’s behavior (a parent cheetah, a human driver) and copy it directly, as in NVIDIA’s end-to-end driving. Inverse reinforcement learning — observe behavior and infer the intent or reward function behind it, then optimize that inferred objective your own way, like the child inferring the experimenter’s goal or the robot inferring the pouring objective. Prediction — learn to predict what will happen next (e.g., action-conditioned video prediction) even before knowing the task, then use the predictive model to achieve goals once they’re known. Transfer from prior knowledge/pre-trained models — reuse knowledge from other tasks or large pre-trained models, as RT-2 uses internet-scale vision-language knowledge to execute robot instructions never seen in robot data.