Day 35: Backpropagation by Hand and Gradient Flow

Instructions

Work the derivation symbolically first, then substitute numbers.

For the linear output neuron $o = w_{o1} a_1 + w_{o2} a_2 + b_o$ and loss $L = (o - y)^2$:

1. Output error term: $\delta_o = \frac{\partial L}{\partial o} = 2(o - y)$.
2. Local input gradients: $\frac{\partial o}{\partial a_1} = w_{o1}$, $\frac{\partial o}{\partial a_2} = w_{o2}$.
3. Local parameter gradients: $\frac{\partial o}{\partial w_{o1}} = a_1$, $\frac{\partial o}{\partial w_{o2}} = a_2$, $\frac{\partial o}{\partial b_o} = 1$.
4. Chain rule: every parameter gradient equals upstream $\delta_o$ times the local derivative: $\frac{\partial L}{\partial w_{o1}} = \delta_o \cdot a_1$.

Substitute the Day 34 ledger values $o = 3.1255$, $a_1 = 0.8741$, $a_2 = -0.9757$ and predict the three output-layer gradients before checking the answer.

Instructions

Continue the staged derivation into the hidden layer without assistance.

The chain rule for hidden neuron $h_1$:

1. Gradient into the activation: $\frac{\partial L}{\partial a_1} = \delta_o \cdot w_{o1} = 4.2511 \times 1.0 = 4.2511$.
2. Through the activation: $\delta_{z_1} = \frac{\partial L}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} = 4.2511 \times (1 - 0.8741^2) = 4.2511 \times 0.2360$.
3. Hidden weight gradients: $\frac{\partial L}{\partial w_{11}} = \delta_{z_1} \cdot x_1$, $\frac{\partial L}{\partial w_{12}} = \delta_{z_1} \cdot x_2$, $\frac{\partial L}{\partial b_1} = \delta_{z_1}$.

Repeat for $h_2$ with $w_{o2} = -2.0$ and $1 - a_2^2 = 0.0479$.

Then verify all nine values:

python3 - <<'EOF'
import sys; sys.path.insert(0, '.')
from projects.autograd.engine import Value
x1, x2 = Value(2.0), Value(-1.0)
W = [Value(v) for v in (0.5,-0.25,0.1, -0.5,1.0,-0.2, 1.0,-2.0,0.3)]
a1 = (x1*W[0] + x2*W[1] + W[2]).tanh()
a2 = (x1*W[3] + x2*W[4] + W[5]).tanh()
loss = (a1*W[6] + a2*W[7] + W[8] - 1.0)**2
loss.backward()
print([round(w.grad, 4) for w in W])
EOF

Instructions

Run two measurements, then do closed-book retrieval of Days 31-34.

Measurement A — saturation:

python3 - <<'EOF'
import sys; sys.path.insert(0, '.')
from projects.autograd.engine import Value
for z0 in (4.0, 8.0):
    z = Value(z0); a = z.tanh(); a.backward()
    print(z0, round(a.data, 6), z.grad)
EOF

Measurement B — depth: a chain of ten $\tanh$ neurons with weight $w = 0.5$ contributes a per-layer factor $w \cdot \tanh'(z_i) \le 0.5$, so the gradient reaching the first layer is bounded by $0.5^{10}$.

Retrieval (closed book, then check the earlier days): state why += is required in _backward, what a topological sort guarantees, the shared-node gradient $\partial f/\partial a$ for $f = (a+b)a$ at $(3,4)$, and N1's forward values.

Instructions

Independent review protocol: (a) closed-book, redo the full nine-gradient derivation for N1 on input $x = (1.0, 1.0)$ (forward values from Day 34 act-04: $a_1 = 0.3364$, $a_2 = 0.2913$, $o = 0.0538$); (b) write the VJP formulation $\mathbf{v}^T J$ and name its scalar counterpart; (c) take the objective quiz in Independent mode; (d) plan one recovery link per missed idea.