第 35 日：手算反向传播与梯度流动

$\delta_o=4.2511$；$\partial L/\partial w_{o1}=4.2511\times0.8741=3.7157$，$\partial L/\partial w_{o2}=4.2511\times(-0.9757)=-4.1480$，$\partial L/\partial b_o=4.2511$。偏置以系数1进入线性输出。到达 $a_1$ 的梯度是 $\delta_o w_{o1}=4.2511$，还需乘 $\tanh'(z_1)=1-a_1^2$。

参考梯度表（表中值来自同一引擎，手算允许显示精度的舍入）：
| param | factors | value |
| --- | --- | --- |
| $w_{o1}$ | $4.2511 \times 0.8741$ | $3.7157$ |
| $w_{o2}$ | $4.2511 \times (-0.9757)$ | $-4.1480$ |
| $b_o$ | $4.2511 \times 1$ | $4.2511$ |
| $w_{11}$ | $4.2511 \times 0.2360 \times 2.0$ | $2.0068$ |
| $w_{12}$ | $4.2511 \times 0.2360 \times (-1.0)$ | $-1.0034$ |
| $b_1$ | $4.2511 \times 0.2360 \times 1$ | $1.0034$ |
| $w_{21}$ | $-8.5022 \times 0.0479 \times 2.0$ | $-0.8149$ |
| $w_{22}$ | $-8.5022 \times 0.0479 \times (-1.0)$ | $0.4075$ |
| $b_2$ | $-8.5022 \times 0.0479 \times 1$ | $-0.4075$ |

$\tanh'(z_2)=0.0479$ 明显缩小了第二个神经元的梯度，这是饱和的预告。

$\tanh'(4)\approx1.34\times10^{-3}$、$\tanh'(8)\approx4.5\times10^{-7}$。后者传过的梯度不到百万分之一。$0.5^{10}=9.77\times10^{-4}$，十个不大于0.5的因子使传入梯度低于0.1%。+= 合计共享节点的所有消费路径；节点的 _backward 要在全部下游贡献到齐后执行。$\partial f/\partial a=10=7+3$。N1：$a_1=0.8741$、$a_2=-0.9757$、$o=3.1255$、$L=4.5179$。

参考解（均表示损失对参数的导数，不是参数新值）：
$\delta_o = -1.8925$; $\partial L/\partial w_{o1} = -0.6366$, $\partial L/\partial w_{o2} = -0.5513$, $\partial L/\partial b_o = -1.8925$; $\partial L/\partial w_{11} = \partial L/\partial w_{12} = \partial L/\partial b_1 = -1.6784$; $\partial L/\partial w_{21} = \partial L/\partial w_{22} = \partial L/\partial b_2 = 3.4638$.

四条公式：
$\delta^L = \nabla_a L \odot \sigma'(z^L)$; $\delta^l = (W^{l+1})^T\delta^{l+1}\odot\sigma'(z^l)$; $\partial L/\partial b^l = \delta^l$; $\partial L/\partial W^l = \delta^l(a^{l-1})^T$.

引擎核验九项；测验检查 delta 结构、饱和与梯度累加。