Day 36: Training Loops and Gradient Failure Diagnosis

Reference Solution:
1. Stochastic step: $\Delta b_o = -0.05 \times 4.2511 = -0.2126$, so $b_o$ would fall from $0.3$ to $\approx 0.0874$.
2. Full-batch step: $\Delta b_o = -0.05 \times 3.2111 = -0.1606$, giving $b_o \approx 0.1394$ after epoch 0 — same direction, different magnitude, because the batch gradient sums all four samples.
3. Measured: the mean loss falls monotonically from $2.106$ to $\approx 0.13$ over 40 epochs at $\eta = 0.05$ (single toy run — evidence the loop works, not a general guarantee).
4. zero_grad must run once per epoch before any backward() call; omitting it keeps every previous gradient in .grad, so the step uses a corrupted sum.

Reference observations (verified run):
1. $\eta = 0.05$: $2.106 \rightarrow 0.369 \rightarrow 0.268 \rightarrow 0.189 \rightarrow 0.129$ at epochs 0/10/20/30/39 — a steady decrease.
2. $\eta = 5.0$: $2.106 \rightarrow 6.0 \times 10^{36} \rightarrow 1.6 \times 10^{76} \rightarrow 4.4 \times 10^{115} \rightarrow 1.3 \times 10^{151}$ — unbounded divergence.
3. Each step assumes the gradient stays constant along the step direction; a large $\eta$ jumps past the basin, where the (larger) new gradient pushes the parameters even further out next epoch — exponential blow-up.
4. Illegitimate claims include: 'this learning rate is optimal', 'MLPs always converge at 0.05', or 'the loss would keep falling forever' — one toy run proves loop mechanics, not convergence guarantees.

Reference diagnoses:
1. $w_{11}.grad$: $2.0068 \rightarrow 4.0135$ (exactly $2\times$): gradients accumulate in .grad across calls on a reused graph — the second backward() adds the same vector–Jacobian product again, so every non-root gradient doubles (the root loss.grad is re-seeded to 1 on each call). Fix for a repeated backward on the same graph: call loss.zero_grad() to clear every reachable node before the next backward(). Zeroing only the parameter leaves restores correct leaf values but leaves stale .grad records on interior nodes, so whole-graph zeroing is still required for a clean reused graph. For training, rebuild the forward graph for each sample and zero the parameter leaves once before the batch.
2. With $b_1 = 8.0$: $z_1 = 9.25$, $\tanh'(z_1) \approx 3.7 \times 10^{-8}$; hidden-weight gradients collapse to the $10^{-7}$ range ($w_{11} \approx 3.3 \times 10^{-7}$) — the neuron is saturated and its weights effectively freeze. Fix: smaller initialization scale or an activation/initializer that keeps $|z|$ out of saturation.
3. z = 1.25: the extra input $999.0$ was silently discarded because zip truncates to the shortest input. Fix: assert len(x) == len(self.w) before the weighted sum (as in the Day 34 skeleton).

Reference Solution:
1. Contract: zero grads → forward → loss → backward → step, with the update p -= lr * grad. Removing parameter zeroing before a fresh batch leaves stale leaf gradients, so each epoch uses a corrupted sum. Reusing the same graph also retains interior gradients: use loss.zero_grad() to clear every reachable node before another backward; clearing leaves alone restores leaf values but leaves stale .grad records on interior nodes.
2. Most likely: learning rate too large or missing zero_grad. At fixed inputs and identical parameter values, compare the suspect gradients with one backward on a freshly rebuilt, fully zeroed graph. A mismatch identifies stale accumulation; if gradients match but the update overshoots, reduce the learning rate. Day 35's original magnitudes are not a universal threshold after parameters change. Saturation gives near-zero hidden gradients instead.
3. Quiz keys derive from measured failures: un-reset gradient accumulation, compounding overshoot, and explicit length checks. No unconditional monotonic-loss guarantee follows from a gradient step.