HW3 · Q-Learning & Actor-Critic

What you build

Two agents, two action-space regimes. Part one is Deep Q-Networks: a $Q$-network update with a target network and $\epsilon$-greedy action selection, where the greedy action is simply $a = \arg\max_a Q_\varphi(s, a)$ over a finite set. You then layer on double Q-learning, which fights overestimation bias by selecting the next action with the online network $Q_\varphi$ while estimating its value with the target network $Q_{\varphi'}$:

$a' = \arg\max_{a'} Q_\varphi(s', a'), \qquad Q_{\text{target}} = r + \gamma (1 - d_t)\, Q_{\varphi'}(s', a')$

Part two is Soft Actor-Critic for continuous actions, where $\max_a Q(s,a)$ has no closed form, so you learn an explicit policy trained to maximize $\mathbb{E}_{a \sim \pi(a \mid s)}[Q(s,a)]$. SAC is built up in five stages: a bootstrapped critic with hard and soft (Polyak-averaged) target updates, an entropy bonus $\beta \mathcal{H}(\pi(a \mid s))$ on both actor and critic targets, the actor update via the reparametrization trick ($a = \mu_\theta(s) + \sigma_\theta(s)\epsilon$), automatic temperature tuning by dual gradient descent on a learnable $\log \alpha$, and finally clipped double-$Q$ — two critics, target values taken as the minimum of both.

Your code lives mainly in four places: src/agents/dqn_agent.py and src/scripts/run_dqn.py for DQN, src/agents/sac_agent.py and src/scripts/run_sac.py for SAC. The handout also tells you to read src/configs/dqn_config.py and src/infrastructure/replay_buffer.py thoroughly before writing anything — do that; the provided stabilization tricks (exploration scheduling, learning-rate decay, gradient clipping) are already wired in, and you’re expected to understand why they’re there.

The lectures behind it

The handout says it covers Lectures 6–8, and each maps cleanly onto a chunk of the work. Lecture 6 (Actor-Critic Algorithms) is the conceptual frame for SAC: why you’d learn a critic alongside a policy, and where target networks and off-policy critics come from. Lecture 7 (Value Function Methods) is the theory under DQN — fitted Q-iteration, why naive Q-learning with function approximation is unstable, and what replay buffers and target networks are actually fixing. Lecture 8 (Deep RL with Q-Functions) is the practical DQN lecture: the full algorithm, double-Q and overestimation bias, and the knobs you’ll be tuning in the hyperparameter study. If your Q-values diverge or sit at zero, rewatch Lecture 8’s stability discussion; if the SAC actor update feels mysterious, Lecture 6 plus the two SAC papers linked in the handout are the references.

Order of attack, and where it gets hard

The handout’s ordering is the right one — each stage has a cheap sanity check, so resist the urge to skip ahead.

For DQN: basic Q-learning first, debugged on CartPole-v1, where you should hit a return of 500 at least once during training. Then double-Q on LunarLander-v2 (expect 200 at least once), then MsPacman-v0 from pixels (around 1500). The handout warns that default parameters should work — if LunarLander won’t reach 200, suspect your implementation before your hyperparameters. Finish with the hyperparameter study: pick one knob (learning rate, architecture, exploration schedule, or target-update frequency), run at least three extra settings, and plot all four on one graph with a caption justifying the choice.

For SAC, the staging is unusually diagnostic-friendly. The critic-only stage on InvertedPendulum-v4 shouldn’t achieve high return yet — you’re only checking that Q-values stabilize somewhere reasonable rather than exploding or flatlining at zero. After adding the entropy bonus (before any reward-maximizing actor update), the logged entropy should climb toward the maximum for a tanh-squashed 1D action space, $\log 2 \approx 0.69$ — significantly off means a bug. Only then does reparametrization make InvertedPendulum actually solve (return near 1000), unlocking the real runs: HalfCheetah (at least 6000 at some point in training), the autotuned-temperature comparison with its two-subplot figure and written questions, and the single-Q versus clipped double-Q comparison on Hopper (clipped should reach at least 1500), where you plot eval_return and q_values and connect what you see to overestimation bias.

The two classic trip-ups: forgetting .rsample() (a reparametrized sample) in the actor update, and letting the critic’s target computation track gradients it shouldn’t.

Compute, runtime, and submission

Most experiments — CartPole, LunarLander, InvertedPendulum, Hopper — are cheap and run fine locally, even on CPU. MsPacman and HalfCheetah are the expensive ones: roughly 3 hours even on a GPU, which is why the course provides Modal scripts (src/scripts/modal_run_dqn.py, modal_run_sac.py) with a --detach flag for long-running jobs and a persistent volume for logs. The handout’s advice is blunt: start early. Seeds matter too — results vary across random seeds, and some experiments may need a couple of runs; the course asks only for your best run to save compute.

Submission is a zip with an exact structure: an exp directory containing the seven named experiment-run folders (matching the tree in Section 4 up to the _sd prefix), the src folder with your .py files in the original layout, plus pyproject.toml, uv.lock, and README.md — not nested inside a parent directory, and under 100 MB. Code and logs go to Gradescope as HW3 Code; the report PDF with all your plots and written answers goes to HW3 Report. Due March 11, 11:59 pm.