交叉熵损失：从概率到标量惊异度
Agent 辅助工程检查
前置要求
第 41 天的前向传播与 softmax；自然对数。
学习目标
定义负对数似然损失，对示例分布完成计算，并相对均匀基线做出解释。
操作指引
必做任务：目标索引 t 的损失为 L = -ln p_t。昨天的前向传播对上下文 ('a','b')（真实下一符号为 'c'，索引 3）输出 p = [0.229291, 0.209107, 0.306433, 0.255169]。计算 L，并与均匀基线 ln(4) = 1.386294 比较，判断该模型是否优于随机猜测。选做任务：换用 log2 把 L 表示成比特。
自主演练与记录
1. L = -ln(0.255169) = ？
2. 对 3 类别的 logit z = [1.0, 0.5, -0.5]、目标索引 0，计算 softmax 概率与 NLL。
3. 本例中困惑度 exp(L) 意味着什么？
e^1 = 2.7183，e^0.5 = 1.6487，e^-0.5 = 0.6065，三者之和为 4.9735。困惑度是目标概率的倒数。
L = -ln(0.255169) = 1.365831 nats——仅略低于均匀基线 1.386294，未训练模型大约就是随机水平。第二个例子：p = [0.5465, 0.3315, 0.1220]，NLL = -ln(0.5465) = 0.6041。困惑度 = 1/0.255169 ~= 3.92：模型的表现等效于在约 3.9 个等可能选项中挑选，而不是 4 个。
两个损失都算到小数点后 4 位以上，给出与均匀基线的比较，并把困惑度正确解释为有效词汇量。
权威源资源与权利证明
makemore: Character-level language models on names.txt
对应章节 / 时间戳: Loss computation segment: negative log-likelihood over the minibatch
验证版本与日期: Repository state verified 2026-10-30; lecture published 2022
Standard beginner walkthrough of the Bengio-style MLP language model; the MIT-licensed reference implementation pairs with a stepwise derivation that matches the Days 40-42 worked numbers exactly.
Python basics, matrix multiplication, and the Day 37-39 tokenization and n-gram context
All forward-pass arithmetic, shapes, parameter counts, and loss values are reproduced as original hand-worked examples in the lesson text; no video, subtitle, or transcript is required.
对比候选资源与选型分析 (3)
The original architecture paper; selected jointly as the theoretical citation while Makemore supplies the stepwise implementation walkthrough.
Strong PyTorch rigor but assumes fluent tensor programming; deferred to the Days 43-60 pretraining blocks rather than the first embedding contact.
Strong visual intuition for network mechanics but covers image classification rather than embedding tables and next-token language modeling; retained as optional supplementary viewing.
A Neural Probabilistic Language Model
对应章节 / 时间戳: Section 2 (training criterion: maximizing the log-likelihood, equivalently minimizing NLL)
验证版本与日期: JMLR Volume 3 (2003); verified 2026-10-30
The original paper introducing jointly learned word feature vectors and the neural probabilistic language model; grounds the curriculum's embedding and MLP derivations in the primary literature.
Probability basics and linear algebra
Every worked derivation in the lessons is self-contained and offline-complete; the paper is further reading only.
对比候选资源与选型分析 (3)
Selected jointly: Makemore reproduces this architecture at character scale with MIT-licensed code; the paper supplies the original formalism and the n-gram comparison framing.
Strong textbook coverage of n-grams through neural LMs; kept as secondary reading because the primary derivation follows the original paper's notation.
Foundational for embedding geometry but trains on context-prediction objectives rather than the full next-token probabilistic model studied this week; revisited in later representation blocks.