第 53 日：模型评估规范：验证集损失差距、过拟合与困惑度 (Perplexity)

01 120 分钟

评估流程规范：训练损失与验证损失差距分析

Agent 辅助工程分析

前置要求

第 51 日预训练循环与第 42 日交叉熵损失。

学习目标

测量训练步数推进中训练集与验证集损失的分离曲线，界定过拟合开始发生的拐点。

操作指引

定义 estimate_loss() 函数：在 torch.no_grad() 与 model.eval() 环境下遍历 eval_iters 个批次。记录典型曲线：0步 (4.10, 4.12)；500步 (2.20, 2.25)；1500步 (1.10, 1.95)。指出 1000 步左右验证集损失停止下降的泛化平台期。

自主演练与记录

1. 为何评估循环必须在 torch.no_grad() 和 model.eval() 下执行？
2. 训练集损失持续走低而验证集损失开始抬头表明什么？
3. 挑选最优模型检查点（Checkpoint）的唯一标准是什么？

标记为已完成

独立掌握标准

掌握标准评估代码范式，明确验证集损失优选原则。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head (returns cross-entropy loss when targets are given)

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

02 120 分钟

测试集困惑度 (Perplexity / PPL) 严谨量测

Agent 辅助工程分析

前置要求

活动 01 及第 42 日困惑度基础。

学习目标

计算测试集平均交叉熵损失，并通过自然指数换算得到独立困惑度 PPL = exp(loss)。

操作指引

对于未参与训练的 100,000 词元测试集：(1) 平均交叉熵为 L_val = 1.6094 nats；(2) 困惑度 PPL = exp(1.6094) = 5.0；(3) 与 V=50,000 的随机猜测基线 (ln 50000 ≈ 10.82, PPL=50,000) 形成鲜明对比。

自主演练与记录

1. 若验证集平均损失为 2.3026 nats，对应困惑度是多少？
2. PPL=10 在直观上代表模型对下一词预测的不确定度处于什么水平？
3. 为什么在评测中只汇报训练集困惑度是一种极具欺骗性的行为？

标记为已完成

独立掌握标准

准确推导交叉熵与困惑度换算，深刻解释其信息论内涵。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head (returns cross-entropy loss when targets are given)

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking Free author-published draft chapter Draft text is not reproduced; linking only

Speech and Language Processing, 3rd ed. draft, Chapter 3: N-gram Language Models

作者 / 机构: Daniel Jurafsky, James H. Martin | 原始出处链接 ↗

对应章节 / 时间戳: Sections 3.1-3.4: n-gram counts, maximum-likelihood estimation, Laplace smoothing, and perplexity

验证版本与日期: Draft of 2026-08-19 (Verified: 2026-10-30)

入选理由:

Standard reference text for n-gram definitions, count normalization, smoothing, and perplexity used across Day 38.

来源前置条件:

Basic probability

安全回退方案（离线弹性）:

All formulas and worked tables in the curriculum are self-contained and verified by execution.

对比候选资源与选型分析 (2)

Chen & Goodman: An Empirical Study of Smoothing Techniques for Language Modeling ↗ compared_alternative [CC BY-NC-SA 3.0 (ACL Anthology, pre-2016 material)]

Detailed smoothing survey but far beyond the add-one scope of a first n-gram day; optional further reading.

MIT OCW 6.863J Natural Language Processing ↗ compared_alternative [CC BY-NC-SA 4.0]

Solid academic lecture treatment but less current and less integrated with executable count tables than the SLP3 draft.

03 120 分钟

过拟合、数据死记硬背与数据泄漏深度诊断

Agent 辅助工程分析

前置要求

活动 01 与 02。

学习目标

排查三类隐蔽的数据泄漏：词元级乱序切分泄漏、测试集与训练集短语重叠、微型数据未加正则化。

操作指引

分析三种场景：(A) 在切分成序列之前对词元乱序（Shuffle）（破坏自回归因果性，并使相邻词跨训练/验证划分泄漏）；(B) 提示与验证集之间存在重叠的 n-gram；(C) 在极小数据集上将 Dropout 设为零。

自主演练与记录

1. 为何在切分训练/验证集前对词元数组全局打乱会导致灾难性数据泄漏？
2. 如何区分真正的泛化能力与对高频死板套话的机械背诵？
3. 正确的切分顺序是：先切分原始文档再独立分词，还是先全文拼接分词再随机切片？

标记为已完成

独立掌握标准

指出词元级泄漏的致命机理，建立标准的文档级切分防御规范。

权威源资源与权利证明

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: N/A

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

04 120 分钟

独立模型泛化与检查点审查考核

完全独立闭卷考核

前置要求

完成活动 01 至 03。

学习目标

面对一个具有明显过拟合特征的评测日志，独立计算指标并给出客观诚实的评估报告。

操作指引

某 10M 参数模型在小样本上的训练日志显示：训练损失 0.85，验证损失 1.75。分析其泛化状态并给出工程改进策略。

自主演练与记录

1. 该模型处于欠拟合、理想拟合还是严重过拟合？
2. 应该采取哪些具体的正则化手段（权重衰减、Dropout、早停机制）？
3. 写出一句应写入公开评测报告的诚实陈述。

标记为已完成

独立掌握标准

面对不理想数据不粉饰、不掩盖，给出严密的诊断与改进方案。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: N/A

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.