Day 34: MLP Structure and the Forward Pass

Instructions

Read the concept decomposition below, then predict before computing.

A perceptron computes $z = \sum_i w_i x_i + b$ and fires $1$ if $z > 0$ else $0$. Its step activation makes the input-output map piecewise constant: the gradient is $0$ almost everywhere and undefined at the kink, so gradient-based training cannot improve it. A differentiable neuron replaces the step with a smooth activation such as $\tanh$.

A multilayer perceptron (MLP) stacks fully-connected layers: every neuron in layer $l$ consumes every activation in layer $l-1$. A single linear unit can only draw one straight decision boundary and cannot represent XOR; two or more layers with a nonlinear activation can.

Reference network N1 (used all day): inputs $x=(2.0, -1.0)$; hidden neuron $h_1$: weights $(0.5, -0.25)$, bias $0.1$, $\tanh$; hidden neuron $h_2$: weights $(-0.5, 1.0)$, bias $-0.2$, $\tanh$; output neuron $o$: weights $(1.0, -2.0)$, bias $0.3$, linear (no activation); loss $L = (o - y)^2$ with target $y = 1.0$.

Instructions

Work the forward pass on paper or in a plain calculator before touching the engine. Fill a ledger with one row per node: node name, local inputs, computed value.

Then verify with the engine:

python3 - <<'EOF'
import sys; sys.path.insert(0, '.')
from projects.autograd.engine import Value
x1, x2 = Value(2.0), Value(-1.0)
a1 = (x1*Value(0.5) + x2*Value(-0.25) + Value(0.1)).tanh()
a2 = (x1*Value(-0.5) + x2*Value(1.0) + Value(-0.2)).tanh()
o = a1*Value(1.0) + a2*Value(-2.0) + Value(0.3)
loss = (o - 1.0)**2
print(a1.data, a2.data, o.data, loss.data)
EOF

Compare each printed value with your ledger row by row. Any disagreement means a ledger row is wrong — find the first one.

Instructions

Write a small module (for example mlp_scratch.py in a scratch directory, not inside lessons/ or projects/autograd/) with this skeleton:

from projects.autograd.engine import Value
class Module:
    def parameters(self):
        return []
class Neuron(Module):
    def __init__(self, weights, bias):
        assert len(weights) > 0
        self.w = [Value(w) for w in weights]
        self.b = Value(bias)
    def __call__(self, x):
        assert len(x) == len(self.w), 'input/weight length mismatch'
        z = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
        return z.tanh()
    def parameters(self):
        return self.w + [self.b]

Extend it with Layer (a list of neurons evaluated on the same input) and MLP (a list of layers applied in sequence). Build N1 with Linear output: give the output neuron a flag that skips tanh. Evaluate the loss on the Day 34 sample, call loss.backward(), and print all nine parameter gradients.

For a reference neuron trace, compare with:

python3 projects/autograd/cli.py forward-backward --preset neuron --json

Instructions

Without consulting earlier activities or AI tools: write the forward equations for a generic neuron, compute the output of N1 for the new input $x = (1.0, 1.0)$ (all weights and biases unchanged), and then complete the objective quiz. Record your scratch work in the draft space before submitting.