Dorsa Sadigh

Why the standard recipes fail: the assistive feeding problem

Sadigh’s lab works on the intersection of humans, robots, and learning, and she opens with a task that stresses every tool this course has built: assistive feeding. A robot must pick up food — deformable, hard to simulate — and transfer it to a person’s mouth safely and comfortably. The baseline is visual servoing to a fixed offset from the mouth, and “it’s very, like, extremely uncomfortable. Looking at this video makes me sad.”

Why not RL? No good simulator (“I don’t really know how the mouth interacts with the carrots and the fork”), and running trial-and-error on a real face — “I really don’t want to hit the person’s nose and get a negative reward” — is off the table. Why not imitation? Collecting demonstrations needs special teleoperation rigs, and human data is strange: people pause mid-demonstration because they are also thinking about dinner, and the first thing practitioners do is scrub those pauses out. Calling human data “suboptimal” may not even be the right word — it is data generated by an agent running other cognitive processes at the same time. Demonstrations are a good source of data, but they are not the only thing humans leave behind.

Preference queries are hyperplanes: active reward learning

The first alternative source: pairwise comparisons. Show a person two trajectories and ask which they prefer — about how an autonomous car should drive, how an exoskeleton should help you walk, whether a drawer should be opened gently. What is learned is a genuine reward function you can then optimize with RL or control. (She flags recent work suggesting human comparisons actually track advantage functions better than reward functions, but takes rewards as the working representation.)

Suppose the reward is a linear combination of nonlinear features, $r(\tau) = w^\top \phi(\tau)$. Only the relative weighting of features matters, so take $w$ on the unit ball. The geometry is then clean:

The research question becomes active learning: what sequence of informative, diverse questions converges fastest to the true $w$? It is the Netflix-challenge question — which pair of movies do I show you to pin down your taste — except robotics has no library to pull from. Queries must be synthesized from a continuous trajectory space, subject to dynamics constraints, so each question is itself a constrained optimization. The objective her student’s thesis develops: pick the query maximizing the minimum expected volume removed from the hypothesis space over the possible answers — a max-min, since either answer must be informative — and the objective’s structure admits the usual greedy-optimization convergence results. In a driving simulator this works with startling economy: after 0 questions the car is lost, after 30 it holds its heading, after 70 binary questions — no demonstrations at all — the orange car drives and avoids collisions, something she “had a hard time tuning myself.” The neural-reward extension loses the theory but keeps the recipe, and powered a Caltech collaboration learning individual gait preferences for people learning to walk again in exoskeletons.

An LLM inside the RL loop: the opposite of RLHF

The same active-querying machinery applies outside robotics, and the negotiation domain exposes the reward-design problem sharply.

The cheaper thing: ask a large language model. Negotiation is a text game, and the internet is full of text about negotiations — so prompt the LLM with a rollout of the current policy and ask “was Alice versatile? was this fair? was this too stubborn?” The yes/no answer (or its log probabilities) is a reward signal. Feed it back as a reward regularizer or shaping term, retrain the agent, roll out again, re-query, loop.

Does this transfer to robotics? Partly. A Google collaboration uses an LLM as a reward designer: from a high-level instruction — “it is late in the afternoon, make the robot face towards the sunset” — the LLM outputs the weights of a reward function optimized by model-predictive control, yielding quadrupeds that sit like a dog, lift a paw, even moonwalk. But she keeps flagging grounding: these live in simulation with ground-truth state, and in her lab’s experience VLMs as success detectors on real perception are “a little questionable” still — improving, not yet reliable.

Take one: build a robotics foundation model

LLMs exist; what does that mean for robotics? Sadigh sees two takes. The grand one: build the analog of a large language model for robotics by pre-training on large offline datasets — robot interaction data, human videos, language, simulation. The open questions are what representation to pre-train and what fine-tuning and downstream adaptation look like.

Her lab started where the data is — human videos — and asked for visual representations useful for control. The field’s two extremes both fail: masked autoencoders give local spatial features (good for grasping) but “destroy all the semantics” — pouring from a jar of orange juice and a jar of milk are the same task with different pixels; CLIP-style contrastive models capture semantics but destroy the local spatial features fine-grained manipulation needs. The Voltron model uses language as the bridge: a masked-autoencoding backbone conditioned on language (grounded reconstruction), plus language generation to capture what the task is, plus multi-frame conditioning for dynamics — syntax, semantics, and pragmatics together. Fine-tuned on 20 demonstrations for language-conditioned imitation, it beats R3M and masked visual pre-training (she is candid that absolute success rates are low across the board). Most striking: zero-shot intent inference — the representation aligns with when a faucet gets opened in a human video, and does the same on robot videos despite being trained on no robot data at all. And the missing ingredient — large real robot datasets — is being built: the cross-embodiment RT-X effort spanning many labs (including Levine’s), and the R2D2 in-the-wild collection effort, teleoperated robots in Stanford dorms. Open question there: should such a model output actions, or representations — what is the right level of abstraction to pre-train?

Take two: creative uses of the models that already exist

The second take she was “very skeptical of initially”: don’t build a robotics foundation model — tap the existing LLMs and VLMs creatively. Her first reaction to SayCan-style LLM task planning was “was that the problem in robotics?” — but the skepticism has faded, because the models open doors nobody was framing two years ago: Code as Policies (LLMs writing robot code as the scaling path), LLMs and VLMs as reward designers, fine-tuning VLMs to be physically and spatially grounded, common-sense reasoning — a VLM shown two tables knows you may clear the scattered LEGOs but not the assembled set someone spent hours building, a value-alignment problem she considers “kind of solved now” — semantic manipulation (referring to laces and heels), and coaching humans with corrective feedback.

The most surprising item: LLMs as pattern machines, beyond semantics entirely. Discretize an image into tokens, give input-output examples of “red cup onto green plate” transformations, and the LLM completes the pattern — token-invariant, the symbols carry no meaning. Give it the $(x, y)$ coordinates of a sine wave and it continues them; give it a robot end-effector’s XYZ-yaw-pitch-roll sequence and it continues the motion. Most interesting, it can optimize: interleave trajectories with their rewards, sorted by reward, in the context, and it emits higher-reward continuations — stabilizing an inverted pendulum, or clicker-training a reach, where clicks near the cup become the in-context reward signal. Semantically meaningful tokens converge faster, but any tokens work: it is the pattern, not the semantics, being exploited. She is explicit that she is not proposing this as the approach to vision or control — the point is what it reveals about what pre-trained models can do, and what we might fine-tune them toward. The talk closes back at feeding: learned food-pickup policies choosing what bite to take next, a reactive controller with tuned levels of reactivity carrying the fork in and out — “I think it’s better than the first video I showed.”

Check yourself

Three exercises in the lecture’s spirit — geometry of queries, noise models, and the design space of LLM-derived rewards.

Problem G.1 query geometry and volume

Consider reward $r(\tau) = w^\top \phi(\tau)$ with $w$ on the unit ball in $\mathbb{R}^3$, and an idealized noiseless human. (a) Show that the answer to one pairwise query can at best halve the hypothesis volume, and say which queries achieve exactly half. (b) Under ideal halving queries, how many binary answers reduce the feasible volume to a fraction $2^{-k}$ of the ball? Reconcile this with the lecture’s ~70 questions for driving. (c) Why does Sadigh’s method reweight samples via a Boltzmann model instead of discarding the wrong half?

Show solution

(a) The query $w^\top(\phi(\tau_A) - \phi(\tau_B)) = 0$ is a hyperplane through the origin (both sides pass through $w = 0$ scaled copies; on the unit ball it is a great circle cut). The answer keeps one side. In the worst case over answers you keep the larger side, so the guaranteed removal is the smaller side’s volume — maximized when the cut is balanced, i.e. the hyperplane splits the current feasible region into equal volumes, removing exactly half. This is precisely why the objective is max-min: only balanced cuts guarantee information for either answer. (b) Each ideal query halves the feasible volume, so $k$ queries reach $2^{-k}$ — 70 questions would in the ideal noiseless case give a $2^{-70}$ fraction. Reality needs all 70 for a usable driving reward because answers are noisy (no hard pruning, only reweighting), queries are constrained by dynamics (you cannot always synthesize a balanced cut with feasible trajectories), and features are imperfect. (c) A noiseless model would zero out the true $w$ forever after a single mistaken answer. The Boltzmann-rational model treats each answer as evidence, not proof: sample weights on the disfavored side shrink but never vanish, so contradictory later answers can recover the truth.

Problem G.2 Boltzmann preference model

Model the human’s answer as Boltzmann-rational: $P(\tau_A \succ \tau_B) = \frac{\exp(\beta\, w^\top \phi(\tau_A))}{\exp(\beta\, w^\top \phi(\tau_A)) + \exp(\beta\, w^\top \phi(\tau_B))}$. (a) Show this equals $\sigma\!\left(\beta\, w^\top \psi\right)$ where $\psi = \phi(\tau_A) - \phi(\tau_B)$ and $\sigma$ is the logistic function. (b) What do the limits $\beta \to \infty$ and $\beta \to 0$ mean for the update to the hypothesis space? (c) Explain why restricting $w$ to the unit ball is not a real loss of generality for the set of optimal policies — and what freedom the norm of $w$ actually plays in this noise model.

Show solution

(a) Divide numerator and denominator by $\exp(\beta\, w^\top \phi(\tau_A))$: $P = \frac{1}{1 + \exp(-\beta\, w^\top \psi)} = \sigma(\beta\, w^\top \psi)$. The answer depends on $w$ only through which side of the hyperplane $w^\top \psi = 0$ it lies on, and how far. (b) As $\beta \to \infty$, $\sigma(\beta\, w^\top \psi)$ tends to a step function: the human deterministically picks the higher-reward trajectory, and the Bayesian update becomes hard pruning of the wrong half-space. As $\beta \to 0$, $P \to 1/2$ regardless of $w$: answers carry no information and the posterior never moves. (c) Scaling $w$ by any positive constant scales every trajectory’s reward by that constant, which preserves the argmax — the optimal policy is unchanged, so only the direction of $w$ (its position on the unit ball) matters for behavior. But in the Boltzmann model the product $\beta \lVert w \rVert$ sets how sharply preference probability saturates with reward gaps — the norm is absorbed into the rationality temperature. Fixing $\lVert w \rVert = 1$ makes $\beta$ alone carry the noise level.

Problem G.3 LLM-as-reward design space

You want to train a negotiation agent the lecture’s way: roll out the current policy, prompt an LLM “was Alice too stubborn in this dialogue?”, and use the answer as reward signal. (a) Give two distinct ways to turn the LLM’s output into a scalar signal, and one reason to prefer each. (b) The lecture uses the LLM signal as a regularizer added to the task utility rather than the sole reward. Give two failure modes this guards against. (c) Why does this recipe transfer poorly to a physical robot judging its own success from camera images, in Sadigh’s telling — and which lecture result shows the failure is about grounding rather than about the model lacking useful structure?

Show solution

(a) Take the yes/no token’s log probability, giving a continuous confidence-weighted score — smoother gradient signal, distinguishes “barely stubborn” from “flagrantly stubborn”; or take the discrete yes/no answer as a binary reward — simpler, robust to miscalibrated probabilities, and matches how a human evaluator would answer. (b) First, proxy exploitation: an agent whose only reward is “the LLM says this was polite” can drift into degenerate dialogues that game the judge while abandoning the task — keeping the true utility anchors task competence (the supervised-only agent that agrees to everything is exactly this failure in human-data form). Second, judge noise: the LLM’s answers are imperfect and unmodeled (the lecture notes they do not apply a noisy-rationality model to the LLM’s output); as a regularizer among other reward terms, occasional wrong judgments perturb rather than define the optimum. (c) The negotiation game is pure text — the LLM judges the very modality it was trained on. A robot’s success is a physical state that must be estimated from perception, and VLMs as success detectors on real scenes were, in her lab’s experience, not yet reliable — the grounding problem. The reward-designer quadruped work isolates this: when ground-truth state is available (simulation, or a paw motion requiring no world interaction), the same LLM machinery produces working rewards for sitting, paw-lifting, and moonwalking — so the bottleneck is state estimation and grounding, not the model’s knowledge of what the behavior should be.