從 one-hot 向量到嵌入查表
Agent 輔助工程檢查
前置要求
第 39 天的斷詞與 n-gram 統計；數學模組中的矩陣乘法基礎。
學習目標
推導為什麼嵌入查表等價於 one-hot 向量乘以嵌入矩陣，並對真實配置預測表的形狀與參數數量。
操作指引
必做任務：使用小詞彙表 V = {'.', 'a', 'b', 'c'}（索引 0-3）與嵌入表 C，其各列分別為 '.'=[0.1,0.3]、'a'=[0.2,-0.1]、'b'=[-0.4,0.5]、'c'=[0.0,0.2]，形狀 (V,d) = (4,2)。證明 one-hot 向量 e_2 = [0,0,1,0] 與 C 相乘會選出第 2 列，得到 [-0.4, 0.5]。解釋為什麼程式以整數索引查表 C[i]，而不是物化一個 V 維 one-hot 向量。選做任務：說明若同一個字元由兩張不同的表嵌入，會有什麼變化。
自主實作與記錄
1. 用上表計算 e_3 * C，並說明選中了哪一列。
2. 對姓名詞彙表 V=27、d=10，給出 C 的形狀與參數數量。
3. 用一句話說明為什麼 C[i] 比先構造 e_i 再做乘法更省。
one-hot 向量除一個 1 之外全為 0；乘積中每個 0 都會抹掉 C 的一列，只有下標為 1 的那一列保留下來。參數數 = 列數 × 欄數。
e_3 = [0,0,0,1] 選中第 3 列，得到 [0.0, 0.2]。V=27、d=10 時，C 的形狀為 (27,10)，共 270 個參數。查表 C[i] 只需 O(d) 的計算與記憶體；one-hot 方式要花 O(V) 的代價卻只為丟棄 V-1 列，還浪費記憶體存放幾乎全零的向量。
學習者能寫出恆等式 e_i * C = C[i,:]，正確算出兩個範例查表結果，並在無提示下推出 (27,10) 表的 270 個參數。
權威來源資源與權利證明
makemore: Character-level language models on names.txt
對應章節 / 時間戳: makemore MLP lecture segment: indexing the embedding table C with one-hot vectors
驗證版本與日期: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
對比候選資源與選型分析 (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
對應章節 / 時間戳: Sections 1-2 (feature vectors and the joint probability model)
驗證版本與日期: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
對比候選資源與選型分析 (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.