第 35 日：手算反向傳播與梯度流動

$\delta_o=4.2511$；$\partial L/\partial w_{o1}=4.2511\times0.8741=3.7157$，$\partial L/\partial w_{o2}=4.2511\times(-0.9757)=-4.1480$，$\partial L/\partial b_o=4.2511$。偏置以係數1進入線性輸出。到達 $a_1$ 的梯度是 $\delta_o w_{o1}=4.2511$，還需乘 $\tanh'(z_1)=1-a_1^2$。

參考梯度表（表中值來自同一引擎，手算允許顯示精度的舍入）：
| param | factors | value |
| --- | --- | --- |
| $w_{o1}$ | $4.2511 \times 0.8741$ | $3.7157$ |
| $w_{o2}$ | $4.2511 \times (-0.9757)$ | $-4.1480$ |
| $b_o$ | $4.2511 \times 1$ | $4.2511$ |
| $w_{11}$ | $4.2511 \times 0.2360 \times 2.0$ | $2.0068$ |
| $w_{12}$ | $4.2511 \times 0.2360 \times (-1.0)$ | $-1.0034$ |
| $b_1$ | $4.2511 \times 0.2360 \times 1$ | $1.0034$ |
| $w_{21}$ | $-8.5022 \times 0.0479 \times 2.0$ | $-0.8149$ |
| $w_{22}$ | $-8.5022 \times 0.0479 \times (-1.0)$ | $0.4075$ |
| $b_2$ | $-8.5022 \times 0.0479 \times 1$ | $-0.4075$ |

$\tanh'(z_2)=0.0479$ 明顯縮小了第二個神經元的梯度，這是飽和的預告。

$\tanh'(4)\approx1.34\times10^{-3}$、$\tanh'(8)\approx4.5\times10^{-7}$。後者傳過的梯度不到百萬分之一。$0.5^{10}=9.77\times10^{-4}$，十個不大於0.5的因子使傳入梯度低於0.1%。+= 合計共享節點的所有消費路徑；節點的 _backward 要在全部下游貢獻到齊後執行。$\partial f/\partial a=10=7+3$。N1：$a_1=0.8741$、$a_2=-0.9757$、$o=3.1255$、$L=4.5179$。

參考解（均表示損失對參數的導數，不是參數新值）：
$\delta_o = -1.8925$; $\partial L/\partial w_{o1} = -0.6366$, $\partial L/\partial w_{o2} = -0.5513$, $\partial L/\partial b_o = -1.8925$; $\partial L/\partial w_{11} = \partial L/\partial w_{12} = \partial L/\partial b_1 = -1.6784$; $\partial L/\partial w_{21} = \partial L/\partial w_{22} = \partial L/\partial b_2 = 3.4638$.

四條公式：
$\delta^L = \nabla_a L \odot \sigma'(z^L)$; $\delta^l = (W^{l+1})^T\delta^{l+1}\odot\sigma'(z^l)$; $\partial L/\partial b^l = \delta^l$; $\partial L/\partial W^l = \delta^l(a^{l-1})^T$.

引擎覈驗九項；測驗檢查 delta 結構、飽和與梯度累加。