第 29 天：评估——数据划分、泄漏与指标

01 120 分钟

训练集、验证集与测试集：先固定比较方式

仅在记录预测与观察证据之后允许求助。

前置要求

完成第 28 天统计；掌握预测—回顾流程。

学习目标

解释训练/验证/测试划分在评估中的作用，以及为什么测试集只读取一次；在任何调参之前先固定评估协议。

操作指引

评估回答的问题是『模型能否泛化』，使用的数据模型在训练中从未见过。训练集拟合参数；验证集指导选择（选哪个模型、哪些设置）；测试集是最终的、未动过的读取。1,000 个样本按 60/20/20 划分为 600 训练、200 验证、200 测试。只有当每个候选都在同一划分上用同一指标打分时，比较才有效——先固定协议再调参，而不是看到结果之后再改。

自主演练与记录

某学习者有 1,000 个标注样本。(a) 按 60/20/20 分配训练/验证/测试。(b) 模型 M1 是在测试集上试评 40 次后选出的——指出协议违规及其后果。(c) 说明比较 M1 与 M2 得分前必须保持一致的条件。

标记为已完成

独立掌握标准

说明各划分在评估中的作用，指出测试集复用违规，并在调参前写出固定协议清单。

权威源资源与权利证明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 机构: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出处链接 ↗

对应章节 / 时间戳: Section: Model Selection, Underfitting, and Overfitting

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

来源前置条件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（离线弹性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

对比候选资源与选型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Training and Test Sets; Validation

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

02 120 分钟

基线与数据泄漏

独立掌握检查（无 AI 辅助）。

前置要求

完成活动 29-01；本块不使用 AI 辅助。

学习目标

计算平凡基线，并识别至少两种会让评估报告出虚假成功的泄漏机制。

操作指引

基线是最廉价的诚实预测器：分类任务中即多数类。若 1,000 个样本中 900 个是类别 0，永远预测类别 0 就得 0.90——任何模型都要对照 0.90 而非 0.0 来评判。泄漏是评估数据的信息流入训练过程：划分前用全部行计算归一化统计量、重复行跨越划分、或由标签派生的特征。它们都会在没有真实能力的情况下抬高评估结果。

自主演练与记录

(a) 数据集 1,000 个样本、900 个属类别 0——多数类基线是多少？某模型报告 0.92：真实提升是多少？(b) 某流水线先用全部 1,000 行的均值做特征标准化，再按 60/20/20 划分——0.92 的分数可信吗？为什么？(c) 若 50 条测试行同时（重复地）出现在 600 行训练集中，评估会怎样？

标记为已完成

独立掌握标准

正确算出基线，指出至少两种泄漏机制及修复（仅在训练集拟合、去重、删除标签派生特征），并报告诚实的较低数字而非被抬高的数字。

权威源资源与权利证明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 机构: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出处链接 ↗

对应章节 / 时间戳: Section: Model Selection, Underfitting, and Overfitting; generalization discussion

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

来源前置条件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（离线弹性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

对比候选资源与选型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Overfitting; Generalization

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

03 120 分钟

指标与分布偏移

仅在记录预测与观察证据之后允许求助。

前置要求

完成活动 29-02；先记录预测。

学习目标

把准确率、交叉熵损失与困惑度理解为同一评估证据的不同分辨率，并诊断由分布偏移而非模型缺陷导致的分数下降。

操作指引

准确率统计答对个数；交叉熵（正确标签被赋予概率 p 的平均 −ln p）同时度量置信度——p = 0.9 代价 0.105，p = 0.5 代价 0.693，p = 0.1 代价 2.303。困惑度是 exp（平均交叉熵）：等价的等可能选择数——总给正确词元 p = 0.5 的模型困惑度为 2，在 4 个选项上均匀猜测的困惑度为 4。分布偏移打破『训练≈测试』假设：白天照片训练的检测器在夜间评估，是面对新分布的新评估，而不必然是模型缺陷。

自主演练与记录

(a) 对正确标签概率 [0.9, 0.8, 0.7, 0.6]，计算平均交叉熵与困惑度。(b) 某模型在测试集上准确率 18/20 = 0.90，但夜间照片只有 11/20——给出两种不同解释及各自的后续检查。(c) 困惑度 2.0 与 10.0 哪个更好？各意味着什么？

标记为已完成

独立掌握标准

NLL/困惑度计算正确，说明准确率与损失的区别，并能区分偏移与缺陷、各给出具体检查。

权威源资源与权利证明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 机构: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出处链接 ↗

对应章节 / 时间戳: Sections: Loss functions and cross-entropy; dataset shift discussion

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

来源前置条件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（离线弹性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

对比候选资源与选型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Classification accuracy; static versus dynamic evaluation

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

04 120 分钟

考核：设计一个有效的小实验

独立掌握检查（无 AI 辅助）。

前置要求

完成活动 29-01 至 29-03；独立闭卷。

学习目标

为一个微型分类任务设计完整、诚实的评估，并为每个选择辩护。

操作指引

场景：800 个标注样本、两个候选模型、类别按 80/20 分布。端到端设计评估并写成协议：带数量的划分、基线、主指标加一个辅助指标、泄漏检查、测试集何时读取，以及如何报告阴性结果。不使用 AI 辅助；重点是一份站得住脚的协议，而不是侥幸的数字。

自主演练与记录

为该 800 样本任务写协议（建议 480/160/160），包含多数类基线值、除准确率外要报告的指标、三项泄漏检查，以及当花哨模型仅与基线持平时你会写下的句子。

标记为已完成

独立掌握标准

协议完整且自洽，写明基线 0.80 与至少一个非准确率指标，包含三项泄漏检查与单次测试读取，阴性结果句不粉饰。

权威源资源与权利证明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 机构: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出处链接 ↗

对应章节 / 时间戳: Section: Model Selection, Underfitting, and Overfitting

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

来源前置条件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（离线弹性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

对比候选资源与选型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 机构: Google for Developers | 原始出处链接 ↗

对应章节 / 时间戳: Modules: Training and Test Sets; Overfitting

验证版本与日期: Current online edition (Verified: 2026-10-30)

入选理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

来源前置条件:

Basic algebra; short self-contained modules.

安全回退方案（离线弹性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

对比候选资源与选型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.