31 / 90

Day 31: Forward Computation Graphs, Local Derivatives & Reverse-Mode Chain Rule

Master the foundations of automatic differentiation: build scalar DAG computation graphs, derive elementary local operator derivatives, and trace the backward pass step-by-step.

Mathematics & Algorithmic Foundations 480 active minutes (8 hours focused study)
01 120 min

Computational Graphs and Forward Scalar Evaluation

Agent-Assisted Engineering Check

Prerequisites

Day 30 mathematics, basic object-oriented programming in Python, and understanding of variables as memory references.

Learning Objective

Understand how scalar arithmetic expressions are mapped into a Directed Acyclic Graph (DAG) of `Value` objects, and trace the forward pass.

Instructions

In automatic differentiation, a computation graph represents each mathematical operation as an interior node, with operands as input edges. Run the graph inspection command:

python3 projects/autograd/cli.py graph --preset neuron --format ascii

Observe how $x_1 w_1 + x_2 w_2 + b$ flows into activation $o = \tanh(n)$. Inspect the forward values.

Practice & Prediction Log

Consider the expression $f(x, y) = (x \cdot y + 2)^2$. At $x = 1.5, y = -2.0$:
1. Construct the step-by-step forward nodes: let $u = x \cdot y$, $v = u + 2$, $f = v^2$.
2. Calculate the numerical value of $u$, $v$, and $f$.
3. Write your predicted forward values in the editor below before running any script.

Independent Mastery Criterion

The learner accurately represents scalar operations as DAG nodes with tracked child dependencies (`_prev`) and forward data floats.

Authoritative Source Resources & Provenance

linking MIT

micrograd: A tiny scalar-valued autograd engine

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: micrograd/engine.py: class Value (_backward closures and backward() topological sort); README 'Example usage'

Verified Version & Date: commit a9a9f94 (Verified: 2026-10-30)

Selection Rationale:

Clear, minimal scalar DAG implementation demonstrating fundamental autograd data structures.

Source Prerequisites:

Basic Python and calculus fundamentals

Safe Fallback (Offline Resilience):

Local engine implementation in projects/autograd/engine.py and offline documentation.

Compared Candidate Sources & Selection Analysis (2)
PyTorch autograd C++ engine ↗ compared_alternative [Modified BSD]

Production scale, but too large and complex for tracing by hand in introductory lessons.

Autograd (HIPS) ↗ compared_alternative [MIT]

Vectorized autograd; micrograd's pure scalar model is pedagogical for hand backprop.

02 120 min

Local Derivatives and Elementary Operator Chain Rules

Independent Mastery Check

Prerequisites

Basic single-variable calculus (power rule, exponential, logarithm, tanh derivatives).

Learning Objective

Derive local analytical partial derivatives for all primary operators (+, -, *, /, pow, exp, log, relu, tanh) and formulate the local `_backward` closures.

Instructions

For each operator $z = f(x, y)$, the node must encapsulate a _backward() closure that multiplies the incoming upstream gradient (out.grad) by the local derivative $\frac{\partial z}{\partial x}$. Review the mathematical operator table in projects/autograd/docs/guide.en.md.

Practice & Prediction Log

Hand-derive the local derivatives for:
1. Multiplication $z = x \cdot y$: derive $\frac{\partial z}{\partial x}$ and $\frac{\partial z}{\partial y}$.
2. Power $z = x^n$ where $n$ is a scalar constant: derive $\frac{\partial z}{\partial x}$.
3. Hyperbolic tangent $z = \tanh(x)$: express $\frac{\partial z}{\partial x}$ in terms of $z$ itself.

Independent Mastery Criterion

The learner independently writes down correct local derivative formulas without table lookups or generative AI assistance.

Authoritative Source Resources & Provenance

linking SIAM Copyright / Academic Linking Only No text extraction or distribution permitted

Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation

Author: Andreas Griewank, Andrea Walther | Original Source Link ↗

Relevant Section / Range: Chapter 3: Reverse Mode and Computational Graphs

Verified Version & Date: 2nd Edition, SIAM 2008 (Verified: 2026-10-30)

Selection Rationale:

Detailed mathematical reference for algorithmic differentiation theory and complexity.

Source Prerequisites:

Multivariate calculus and linear algebra

Safe Fallback (Offline Resilience):

Self-contained mathematical derivations in activity instructions and offline guide.

Compared Candidate Sources & Selection Analysis (2)
micrograd: A tiny scalar-valued autograd engine ↗ compared_alternative [MIT License]

Executable scalar reference implementation the learner later rebuilds; the textbook supplies the formal reverse-mode theory.

Baydin, Pearlmutter, Radul, Siskind: Automatic Differentiation in Machine Learning: a Survey (JMLR 2018) ↗ compared_alternative [JMLR open access]

Broad survey taxonomy of the field; the textbook was selected for its rigorous treatment of reverse-mode complexity.

03 120 min

Reverse-Mode Chain Rule Intuition and Backward Pass Trace

Agent-Assisted Engineering Check

Prerequisites

Completion of Activity 31-01 and 31-02.

Learning Objective

Understand how the multivariable chain rule propagates gradients backwards from target loss to input weights, and trace a single artificial neuron.

Instructions

Execute the forward-backward command for the neuron preset:

python3 projects/autograd/cli.py forward-backward --preset neuron

Verify that the backward pass initializes $o.\text{grad} = 1.0$ and propagates through $n = x_1 w_1 + x_2 w_2 + b$ to yield $\frac{\partial o}{\partial w_1} = 1.0$ and $\frac{\partial o}{\partial x_1} = -1.5$.

Practice & Prediction Log

Given $o = \tanh(n)$ with $n = 0.8814$ ($o \approx 0.7071$):
1. Calculate $\frac{\partial o}{\partial n} = 1 - o^2$.
2. Given $n = x_1 w_1 + x_2 w_2 + b$, where $x_1 = 2.0, w_1 = -3.0$, calculate $\frac{\partial o}{\partial x_1}$ and $\frac{\partial o}{\partial w_1}$.
3. Compare your manual numbers with the CLI output.

Independent Mastery Criterion

The learner traces numerical chain rule products through 3+ layers of graph hierarchy without missing intermediate scaling factors.

Authoritative Source Resources & Provenance

linking MIT

CS336 Assignment 1: Basics (Spring 2025 tagged release)

Author: Stanford University CS336 Staff | Original Source Link ↗

Relevant Section / Range: README setup; tests/adapters.py; tests/test_optimizer.py; tests/test_nn_utils.py; tests/test_serialization.py

Verified Version & Date: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01

Selection Rationale:

Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.

Source Prerequisites:

Python systems programming and multivariable calculus

Safe Fallback (Offline Resilience):

Stand-alone projects/autograd engine.

Compared Candidate Sources & Selection Analysis (2)
Karpathy nanoGPT ↗ compared_alternative [MIT]

Simpler code layout, useful complementary reference for minimal PyTorch Transformer.

Hugging Face Transformers Course ↗ compared_alternative [Apache-2.0]

High-level API focus rather than building from raw tensor fundamentals.

04 120 min

Independent Mastery Check: Hand-Deriving Scalar Graphs

Independent Mastery Check

Prerequisites

Activities 31-01 through 31-03.

Learning Objective

Verify unaided mastery of scalar computational graphs and chain rule backpropagation under declared AI-free conditions.

Instructions

Complete the diagnostic check below without using AI assistants, large language models, or automated symbolic calculators. Record your reasoning in the draft workspace.

Practice & Prediction Log

For the loss function $L = (x \cdot y + 2)^2$ at $x = 1.5, y = -2.0$:
Calculate the exact analytical gradients $\frac{\partial L}{\partial x}$ and $\frac{\partial L}{\partial y}$. Show your chain rule steps.

Objective Knowledge Check: Computation Graphs & Chain Rule

Answer the following diagnostic questions to verify your understanding. Select your declared assistance condition before submitting.

Declared Assistance Condition

Note: AI-free work is an honest declared condition, not claimed surveillance.

1.

In reverse-mode automatic differentiation, what numerical value must the seed gradient loss.grad be initialized to prior to traversing the DAG backward?

2.

For an operation node $z = x \cdot y$, if z.grad is currently $5.0$, and $x = 2.0, y = -3.0$, what is the gradient contribution propagated to $x$?

3.

Why is reverse-mode automatic differentiation asymptotically superior to forward-mode for training neural networks?

Attempt History & Evidence

    Independent Mastery Criterion

    Independent calculation producing $\frac{\partial L}{\partial x} = 4.0$ and $\frac{\partial L}{\partial y} = -3.0$ with documented intermediate variable definitions.

    Authoritative Source Resources & Provenance

    linking MIT

    micrograd: A tiny scalar-valued autograd engine

    Author: Andrej Karpathy | Original Source Link ↗

    Relevant Section / Range: micrograd/engine.py: class Value (_backward closures and backward() topological sort); README 'Example usage'

    Verified Version & Date: Git commit master (Verified: 2026-10-30)

    Selection Rationale:

    Minimal, readable implementation of scalar autograd and MLP layers directly matched to our project.

    Source Prerequisites:

    Python object-oriented programming and operator overloading from Day 31.

    Safe Fallback (Offline Resilience):

    The local projects/autograd engine is runnable offline and matches micrograd semantics.

    Compared Candidate Sources & Selection Analysis (2)
    PyTorch autograd C++ engine ↗ compared_alternative [Modified BSD]

    Production scale, but too large and complex for tracing by hand in introductory lessons.

    Autograd (HIPS) ↗ compared_alternative [MIT]

    Vectorized autograd; micrograd's pure scalar model is pedagogical for hand backprop.

    linking Modified BSD

    PyTorch Documentation: Autograd Mechanics

    Author: PyTorch Contributors | Original Source Link ↗

    Relevant Section / Range: How autograd encodes the history

    Verified Version & Date: PyTorch 2.5 Documentation (Verified: 2026-10-30)

    Selection Rationale:

    Official reference for computational graph construction, backward pass execution, and in-place mutation pitfalls.

    Source Prerequisites:

    Basic understanding of computational graphs and tensors.

    Safe Fallback (Offline Resilience):

    The training loop mechanics and zero_grad requirements are illustrated with runnable Python code.

    Compared Candidate Sources & Selection Analysis (2)
    TensorFlow tf.GradientTape Documentation ↗ compared_alternative [Apache 2.0]

    Alternative tape-based execution; PyTorch dynamic graph semantics are standard in modern LLM research.

    JAX autodiff cookbook ↗ compared_alternative [Apache 2.0]

    Functional transformations (grad, vjp); PyTorch object-based graph is closer to the beginner micrograd model.