从 one-hot 向量到嵌入查表
Agent 辅助工程检查
前置要求
第 39 天的分词与 n-gram 统计；数学模块中的矩阵乘法基础。
学习目标
推导为什么嵌入查表等价于 one-hot 向量乘以嵌入矩阵，并对真实配置预测表的形状与参数数量。
操作指引
必做任务：使用小词汇表 V = {'.', 'a', 'b', 'c'}（索引 0-3）与嵌入表 C，其各行分别为 '.'=[0.1,0.3]、'a'=[0.2,-0.1]、'b'=[-0.4,0.5]、'c'=[0.0,0.2]，形状 (V,d) = (4,2)。证明 one-hot 向量 e_2 = [0,0,1,0] 与 C 相乘会选出第 2 行，得到 [-0.4, 0.5]。解释为什么代码用整数索引查表 C[i] 而不去物化一个 V 维 one-hot 向量。选做任务：说明若同一个字符由两张不同的表嵌入，会有什么变化。
自主演练与记录
1. 用上表计算 e_3 * C，并说明选中了哪一行。
2. 对姓名词汇表 V=27、d=10，给出 C 的形状与参数数量。
3. 用一句话说明为什么 C[i] 比先构造 e_i 再做乘法更省。
one-hot 向量除一个 1 之外全为 0；乘积中每个 0 都会抹掉 C 的一行，只有下标为 1 的那一行保留下来。参数数 = 行数 × 列数。
e_3 = [0,0,0,1] 选中第 3 行，得到 [0.0, 0.2]。V=27、d=10 时，C 的形状为 (27,10)，共 270 个参数。查表 C[i] 只需 O(d) 的计算与内存；one-hot 方式要花 O(V) 的代价却只为丢弃 V-1 行，还浪费内存存放几乎全零的向量。
学习者能写出恒等式 e_i * C = C[i,:]，正确算出两个示例查表结果，并在无提示下推出 (27,10) 表的 270 个参数。
权威源资源与权利证明
makemore: Character-level language models on names.txt
对应章节 / 时间戳: makemore MLP lecture segment: indexing the embedding table C with one-hot vectors
验证版本与日期: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
对比候选资源与选型分析 (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
对应章节 / 时间戳: Sections 1-2 (feature vectors and the joint probability model)
验证版本与日期: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
对比候选资源与选型分析 (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.