Day 32: Topological Sorting, Shared-Node Gradient Accumulation & Gradcheck Tolerances

Consider the expression $f = (a + b) \cdot a$, where $a = 3.0, b = 4.0$.
1. Trace the depth-first search (DFS) traversal starting at root node $f$.
2. List the post-order sequence of nodes visited.
3. Verify that the reversed sequence orders $f$ first, then the sum node, and finally leaf nodes $a, b$.

Given $f(a, b) = (a + b) \cdot a$ with $a = 3.0, b = 4.0$:
1. Let $u = a + b$. Then $f = u \cdot a$. Calculate $\frac{\partial f}{\partial u}$ and $\frac{\partial f}{\partial a}$ from the multiplication step.
2. From the addition step $u = a + b$, calculate $\frac{\partial u}{\partial a}$ and its contribution to $a$.
3. Show how applying $\frac{\partial f}{\partial a} = \frac{\partial f}{\partial u} \frac{\partial u}{\partial a} + \left(\frac{\partial f}{\partial a}\right)_{\text{direct}}$ yields $10.0$.
4. Explain what happens if an implementer writes self.grad = ... instead of self.grad += ....

1. Explain why central difference $\frac{f(x+h) - f(x-h)}{2h}$ has truncation error $\mathcal{O}(h^2)$ whereas forward difference $\frac{f(x+h) - f(x)}{h}$ has error $\mathcal{O}(h)$.
2. Why can we not make $h$ arbitrarily small (e.g. $10^{-20}$)?
3. Calculate the relative error between analytical gradient $g_a = 10.0$ and numerical gradient $g_n = 9.999999999$ and verify it meets $\text{rtol} = 10^{-3}$.

Consider a computation graph where input $x = 2.0$ branches into $y_1 = x^2$ and $y_2 = 3x$, and loss is $L = y_1 + y_2$.
1. Determine the analytical derivative $\frac{\partial L}{\partial x}$.
2. Compute the central finite difference approximation at $x = 2.0$ with $h = 0.001$.
3. Compute the relative error between analytical and numerical gradients.