HW2 · Policy Gradients

What you build

HW2 is the lecture-5 story made runnable. Inside pg_agent.py you fill in the TODO sections that turn sampled rollouts into gradient estimates: two return estimators in _calculate_q_vals — “Case 1”, the discounted full-trajectory return $\sum_{t'} \gamma^{t'} r(s_{t'}, a_{t'})$ applied to every step, and “Case 2”, the discounted reward-to-go $\sum_{t' \ge t} \gamma^{t'-t} r(s_{t'}, a_{t'})$ — plus advantage normalization, a state-dependent value-function baseline $V^\pi_\varphi(s_t)$ trained alongside the policy, and a simplified GAE-$\lambda$ in _estimate_advantage, implemented via the recursion the handout derives from the one-step TD errors $\delta_t$.

The experiments then ablate exactly these pieces. Experiment 1 runs eight CartPole configurations crossing reward-to-go, advantage normalization, and batch size (1000 vs. 4000). Experiment 2 adds the neural baseline on HalfCheetah-v4. Experiment 3 sweeps $\lambda \in \{0, 0.95, 0.98, 0.99, 1\}$ on a noisy-actions LunarLander-v2. Experiment 4 is a hyperparameter hunt: reach a return of 1000 on InvertedPendulum-v4 in as few environment steps as you can, ideally within 100K.

Which lectures feed it

Lecture 5 is the whole substance: the log-gradient identity and REINFORCE (the handout’s review section 2.1 is that derivation compressed), causality and reward-to-go, baselines and their unbiasedness proof, and the pseudo-loss implementation trick — rewatch that last part before touching pg_agent.py, since the weighted-maximum-likelihood reading is literally how the update is written. Lecture 4 supplies the framing you’ll lean on for the written questions: the RL objective, trajectory distributions, and the definitions of $Q^\pi$, $V^\pi$, and the advantage $A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)$, which is exactly what reward-to-go minus the learned baseline estimates. Two things in the handout go slightly beyond the recorded lecture 5: the two discount placements (discount from the trajectory start vs. from the current step — the handout asks you to see why one is preferred) and GAE, whose bias–variance story is developed properly in the actor-critic lecture. The handout’s section 2.2.4 is self-contained enough to implement from.

Order of attack

Follow the handout’s own sequence — it is a dependency chain. Get Case 1 and Case 2 returns working and pass Experiment 1 before writing any baseline code; the handout explicitly lets you skip the use_baseline branches at first. CartPole converging to 200 in both batch regimes is your correctness signal — if the no-tricks configuration limps and reward-to-go plus normalization does not visibly help, suspect your indexing before your hyperparameters.

The traps are mostly off-by-one shaped. The two return estimators differ only in where the sum starts and where the discount is anchored ($\gamma^{t'}$ vs. $\gamma^{t'-t}$); mixing the two gives a subtly wrong estimator that still sort of learns. GAE has an edge case at the final step ($\delta_{H-1}$ has no bootstrap term), and the recursion runs backward over the trajectory — respect episode boundaries. Advantage normalization is per-batch mean-zero, std-one with an $\varepsilon$ in the denominator; it is technically biased, and one written question asks you to reason about that, so keep the handout’s batch-size-1 thought experiment in mind.

Compute is deliberately light: each run takes seconds to at most ten minutes on a laptop CPU, and the handout recommends CPUs over cloud GPUs — the models are small enough that GPUs often lose. The expensive resource is your attention across ~16 runs, so name experiments exactly as the commands specify from the start; the autograder identifies runs by directory-name prefix.

Targets and the report

Numbers to aim at, straight from the handout: CartPole’s best small-batch and large-batch configurations both reach 200; baselined HalfCheetah clears an average return of 300 (2–3 seeds is fair, more means revisit your implementation); the best LunarLander $\lambda$ exceeds 150 at least once during training; InvertedPendulum hits 1000 within 100K environment steps. Every plot must use environment steps (Train_EnvstepsSoFar) on the x-axis, not iterations — a repeated bolded warning. The report also wants brief written answers (which estimator wins and why, what normalization and batch size did, what $\lambda=0$ and $\lambda=1$ correspond to) and the exact command lines for every run.

Submission is a zip to Gradescope: an exp/ directory whose run folders match the handout’s tree exactly, plus your src/ with the original file layout, pyproject.toml, and uv.lock, all under 100MB and not nested inside a parent folder. Code goes to HW2 Code, the PDF report to HW2 Report.