Neural Language Model Pipeline: Indices to Pre-Activations
Agent-Assisted Engineering Check
Prerequisites
Day 40 embedding lookup and context concatenation; Days 34-36 MLP mechanics.
Learning Objective
Walk the full pipeline indices -> C rows -> concatenated x -> hidden pre-activations, and compute the input and pre-activation vectors by hand.
Instructions
Required work: use the Day 40 table C and context ('a','b'), so x = [0.2, -0.1, -0.4, 0.5]. Weights stay fixed today: W1 = [[0.5,-0.2,0.1],[0.3,0.4,-0.3],[-0.1,0.2,0.6],[0.2,-0.5,0.4]], b1 = [0.1,-0.1,0.0]. Compute each hidden pre-activation pre_j = sum_i x_i * W1[i][j] + b1[j]. Record your predicted values before inspecting the trace. Optional work: predict which single input coordinate has the largest influence on pre_1.
Practice & Prediction Log
1. Compute pre_0, pre_1, pre_2 for the x and W1 above, showing every product.
2. Predict whether any pre-activation is exactly zero.
3. Name the pipeline stages in order from raw symbols to pre.
pre_0 = 0.2*0.5 + (-0.1)*0.3 + (-0.4)*(-0.1) + 0.5*0.2 + 0.1. Keep signs attached to each product before summing.
pre_0 = 0.10 - 0.03 + 0.04 + 0.10 + 0.10 = 0.31; pre_1 = -0.04 - 0.04 - 0.08 - 0.25 - 0.10 = -0.51; pre_2 = 0.02 + 0.03 - 0.24 + 0.20 + 0.00 = 0.01. None is exactly zero, though pre_2 is small enough that tanh will be nearly linear there. Stages: symbols -> indices -> C rows -> concat -> x*W1 + b1 -> tanh -> *W2 + b2 -> softmax.
Hand-computed pre-activations match [0.31, -0.51, 0.01] with each product shown, and the learner narrates the pipeline stages in order without notes.
Authoritative Source Resources & Provenance
makemore: Character-level language models on names.txt
Relevant Section / Range: Segment wiring the MLP forward pass: C[X] -> hidden tanh layer -> output logits
Verified Version & Date: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
Compared Candidate Sources & Selection Analysis (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
Relevant Section / Range: Section 2 (the mapping g: context feature vectors to next-word probabilities)
Verified Version & Date: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
Compared Candidate Sources & Selection Analysis (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.