HW5 · Offline RL

What you build

This assignment has you implement three offline RL algorithms and evaluate them on continuous-control tasks from the OGBench suite — cube-single (tabletop manipulation), antsoccer-arena (a quadruped pushing a ball), and antmaze-medium (maze navigation, used for debugging). All three tasks have continuous states and actions bounded in $[-1, 1]$, and each algorithm trains purely from a fixed dataset — no environment interaction during learning, which is the whole point.

The three parts escalate in sophistication:

• Part 1 — SAC+BC: a behavior-regularized actor-critic. You take SAC from the previous assignment and add a behavioral cloning penalty to the policy loss, so the policy maximizes Q values while staying near the dataset’s actions.
• Part 2 — IQL: implicit Q-learning, which sidesteps out-of-distribution actions entirely — expectile regression approximates the max over actions in the Bellman backup without ever querying the Q function off-dataset, and the policy is extracted afterward by advantage-weighted regression.
• Part 3 — FQL: flow Q-learning, which replaces the unimodal Gaussian policy with an expressive flow policy, then avoids backpropagation-through-time by distilling the flow into a one-step feedforward policy that carries the Q-maximization term.

Each part is a fill-in-the-TODOs implementation (sacbc_agent.py, iql_agent.py, fql_agent.py under src/agents/) plus a tuning campaign: for every algorithm you must find a BC/distillation coefficient $\alpha$ that clears specified success rates — e.g. above 75% on cube-single and above 5% on antsoccer-arena for SAC+BC — and report training curves across at least three $\alpha$ values.

Which lectures feed it

This is the companion to Lectures 15–16 (Offline RL I and II). Lecture 15 sets up the problem: why naive off-policy RL fails on a fixed dataset, how the learned Q function exploits out-of-distribution actions, and why distributional shift — not overfitting — is the core pathology. Rewatch it if you can’t articulate why the BC term exists at all. Lecture 16 covers the solution families this assignment draws from: policy-constraint methods (SAC+BC is the minimalist representative), implicit backups that never evaluate off-dataset actions (IQL’s expectile trick), and advantage-weighted policy extraction. Part 3’s flow policies go beyond the recorded lectures — the handout’s Section 4 is self-contained, and the FQL paper (Park et al., 2025) is the reference if you want depth.

If SAC itself feels shaky, revisit your HW3 implementation before starting: the handout deliberately builds SAC+BC on your existing SAC scaffolding.

How to attack it

Work the parts in order — each reuses machinery from the last. Before writing any code, read src/scripts/run.py, src/networks/rl_networks.py, and the config files, as the handout insists; the TODOs make little sense without the training loop’s shape in your head.

Budget compute early. This is the handout’s loudest warning: individual runs take up to about 6 hours, and the tuning sweeps multiply that. The intended workflow is running 4 agents in parallel on a single GPU (a helper script is provided) — roughly 4–6 hours per SAC+BC sweep on a T4, 3–5 for IQL, 5–7 for FQL. Use antmaze-medium as your correctness check: each algorithm has a stated success rate it should hit at 100–200K steps with defaults, so you get a debugging signal in well under an hour of training rather than after a full run.

Tuning $\alpha$ is the intellectual core, and the handout hands you the diagnostics. Watch the MSE between dataset actions and policy actions: it should fall as $\alpha$ rises, and the best $\alpha$ typically sits where the MSE is only slightly above the pure-BC floor. Watch q_min and q_max: rewards are $-1$ or $0$ per step, so Q values live in $[-\tfrac{1}{1-\gamma}, 0]$ — values escaping that box mean the backup is wrong, the classic offline-RL overestimation signature from Lecture 15. Also mind the handout’s fine print: incorporate the done signal in the backup, use the average (not minimum) of the two target Q values, clip flow-policy actions to $[-1, 1]$ before the Q function but not in FQL’s distillation term, and evaluate FQL with the one-step policy, not the flow.

Expect antsoccer-arena to look bad — single-digit success rates are normal and the thresholds reflect that. A single (best) seed per question is acceptable here, though the handout notes real offline-RL research would average over seeds.

Submission

You submit a zip (under 100MB) containing an exp/ directory with your best run per question — q1/q2/q3, each holding cube-single and antsoccer-arena run folders with logs and checkpoints, renamed exactly as the handout’s file tree shows — plus your src/ folder mirroring the starter repo’s structure. The code zip goes to Gradescope’s HW5 Code; a separate PDF report with your training curves, the $\alpha$-sweep plots, and the short SAC+BC-vs-IQL comparison goes to HW5 Report. Due April 19, 11:59 pm.