交叉熵損失：從機率到純量驚異度
Agent 輔助工程檢查
前置要求
第 41 天的前向傳播與 softmax；自然對數。
學習目標
定義負對數似然損失，對範例分布完成計算，並相對均勻基線做出解釋。
操作指引
必做任務：目標索引 t 的損失為 L = -ln p_t。昨天的前向傳播對上下文 ('a','b')（真實下一符號為 'c'，索引 3）輸出 p = [0.229291, 0.209107, 0.306433, 0.255169]。計算 L，並與均勻基線 ln(4) = 1.386294 比較，判斷該模型是否優於隨機猜測。選做任務：換用 log2 把 L 表示成位元。
自主實作與記錄
1. L = -ln(0.255169) = ？
2. 對 3 類別的 logit z = [1.0, 0.5, -0.5]、目標索引 0，計算 softmax 機率與 NLL。
3. 本例中困惑度 exp(L) 意味著什麼？
e^1 = 2.7183，e^0.5 = 1.6487，e^-0.5 = 0.6065，三者之和為 4.9735。困惑度是目標機率的倒數。
L = -ln(0.255169) = 1.365831 nats——僅略低於均勻基線 1.386294，未訓練模型大約就是隨機水準。第二個例子：p = [0.5465, 0.3315, 0.1220]，NLL = -ln(0.5465) = 0.6041。困惑度 = 1/0.255169 ~= 3.92：模型的表現等效於在約 3.9 個等可能選項中挑選，而不是 4 個。
兩個損失都算到小數點後 4 位以上，給出與均勻基線的比較，並把困惑度正確解釋為有效詞彙量。
權威來源資源與權利證明
makemore: Character-level language models on names.txt
對應章節 / 時間戳: Loss computation segment: negative log-likelihood over the minibatch
驗證版本與日期: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
對比候選資源與選型分析 (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
對應章節 / 時間戳: Section 2 (training criterion: maximizing the log-likelihood, equivalently minimizing NLL)
驗證版本與日期: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
對比候選資源與選型分析 (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.