Cross-Entropy Loss: From Probabilities to a Scalar Surprisal
Agent-Assisted Engineering Check
Prerequisites
Day 41 forward pass and softmax; natural logarithms.
Learning Objective
Define the negative log-likelihood loss, compute it for a worked distribution, and interpret it against the uniform baseline.
Instructions
Required work: for target index t the loss is L = -ln p_t. Yesterday's forward pass produced p = [0.229291, 0.209107, 0.306433, 0.255169] for context ('a','b') whose true next symbol is 'c' (index 3). Compute L, then compare it with the uniform baseline ln(4) = 1.386294. Decide whether this model is better than chance. Optional work: express L in bits by switching to log2.
Practice & Prediction Log
1. L = -ln(0.255169) = ?
2. For logits z = [1.0, 0.5, -0.5] over 3 classes with target index 0, compute the softmax probabilities and the NLL.
3. What does perplexity exp(L) mean in this example?
e^1 = 2.7183, e^0.5 = 1.6487, e^-0.5 = 0.6065; their sum is 4.9735. Perplexity is the reciprocal of the target probability.
L = -ln(0.255169) = 1.365831 nats - barely below uniform 1.386294, so the untrained model sits at about chance level. Second example: p = [0.5465, 0.3315, 0.1220], NLL = -ln(0.5465) = 0.6041. Perplexity = 1/0.255169 ~= 3.92: the model behaves as if choosing among about 3.9 equally likely options instead of 4.
Both losses computed to 4+ decimals, the uniform-baseline comparison stated, and perplexity interpreted as an effective vocabulary size.
Authoritative Source Resources & Provenance
makemore: Character-level language models on names.txt
Relevant Section / Range: Loss computation segment: negative log-likelihood over the minibatch
Verified Version & Date: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
Compared Candidate Sources & Selection Analysis (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
Relevant Section / Range: Section 2 (training criterion: maximizing the log-likelihood, equivalently minimizing NLL)
Verified Version & Date: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
Compared Candidate Sources & Selection Analysis (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.