From One-Hot Vectors to Embedding Lookup
Agent-Assisted Engineering Check
Prerequisites
Day 39 tokenization and n-gram statistics; basic matrix multiplication from the mathematics block.
Learning Objective
Derive why an embedding lookup equals a one-hot vector times the embedding matrix, and predict table shapes and parameter counts for real configurations.
Instructions
Required work: use the tiny vocabulary V = {'.', 'a', 'b', 'c'} (indices 0-3) and embedding table C with rows '.'=[0.1,0.3], 'a'=[0.2,-0.1], 'b'=[-0.4,0.5], 'c'=[0.0,0.2], shape (V,d) = (4,2). Show that multiplying one-hot e_2 = [0,0,1,0] by C selects row 2, giving [-0.4, 0.5]. Explain why code uses integer lookup C[i] instead of materializing a V-dimensional one-hot vector. Optional work: state what changes if the same character is embedded by two different tables.
Practice & Prediction Log
1. Compute e_3 * C for the table above and state which row is selected.
2. For the names vocabulary V=27 with d=10, state the shape and parameter count of C.
3. In one sentence, why is C[i] cheaper than building e_i and multiplying?
A one-hot vector is all zeros except one 1; every zero entry erases one row of C in the product, so only the 1-indexed row survives. Parameters = rows * columns.
e_3 = [0,0,0,1] selects row 3, giving [0.0, 0.2]. For V=27, d=10 the table C has shape (27,10) and 270 parameters. Lookup C[i] costs O(d) work and memory; the one-hot route spends O(V) effort only to discard V-1 rows, and wastes memory storing mostly zeros.
The learner writes the identity e_i * C = C[i,:], computes both worked lookups correctly, and derives the 270-parameter count for the (27,10) table without prompting.
Authoritative Source Resources & Provenance
makemore: Character-level language models on names.txt
Relevant Section / Range: makemore MLP lecture segment: indexing the embedding table C with one-hot vectors
Verified Version & Date: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
Compared Candidate Sources & Selection Analysis (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
Relevant Section / Range: Sections 1-2 (feature vectors and the joint probability model)
Verified Version & Date: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
Compared Candidate Sources & Selection Analysis (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.