神經語言模型流水線：從索引到預激活
Agent 輔助工程檢查
前置要求
第 40 天的嵌入查表與上下文串接；第 34-36 天的 MLP 機制。
學習目標
走完 索引 → C 的列 → 串接 x → 隱藏層預激活 的完整流水線，並手算輸入向量與預激活向量。
操作指引
必做任務：沿用第 40 天的表 C 與上下文 ('a','b')，故 x = [0.2, -0.1, -0.4, 0.5]。今天權重固定：W1 = [[0.5,-0.2,0.1],[0.3,0.4,-0.3],[-0.1,0.2,0.6],[0.2,-0.5,0.4]]，b1 = [0.1,-0.1,0.0]。計算各隱藏層預激活 pre_j = sum_i x_i * W1[i][j] + b1[j]。先記錄你的預測值再查看推演。選做任務：預測哪個輸入座標對 pre_1 影響最大。
自主實作與記錄
1. 用上面的 x 與 W1 計算 pre_0、pre_1、pre_2，寫出每一個乘積。
2. 預測是否有某個預激活恰好為 0。
3. 按順序說出從原始符號到 pre 的流水線各階段。
pre_0 = 0.2*0.5 + (-0.1)*0.3 + (-0.4)*(-0.1) + 0.5*0.2 + 0.1。求和前保持每個乘積的符號。
pre_0 = 0.10 - 0.03 + 0.04 + 0.10 + 0.10 = 0.31；pre_1 = -0.04 - 0.04 - 0.08 - 0.25 - 0.10 = -0.51；pre_2 = 0.02 + 0.03 - 0.24 + 0.20 + 0.00 = 0.01。沒有一個恰好為 0，但 pre_2 足夠小，tanh 在那裡近似線性。各階段：符號 → 索引 → C 的列 → 串接 → x*W1 + b1 → tanh → *W2 + b2 → softmax。
手算預激活寫出每個乘積並與 [0.31, -0.51, 0.01] 一致，學習者能不看筆記按順序敘述流水線各階段。
權威來源資源與權利證明
makemore: Character-level language models on names.txt
對應章節 / 時間戳: Segment wiring the MLP forward pass: C[X] -> hidden tanh layer -> output logits
驗證版本與日期: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
對比候選資源與選型分析 (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
對應章節 / 時間戳: Section 2 (the mapping g: context feature vectors to next-word probabilities)
驗證版本與日期: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
對比候選資源與選型分析 (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.