第 29 天：評估——資料劃分、洩漏與指標

01 120 分鐘

訓練集、驗證集與測試集：先固定比較方式

僅在記錄預測與觀察證據之後允許求助。

前置要求

完成第 28 天統計；掌握預測—回顧流程。

學習目標

解釋訓練/驗證/測試劃分在評估中的作用，以及為什麼測試集只讀取一次；在任何調參之前先固定評估協議。

操作指引

評估回答的問題是『模型能否泛化』，使用的資料模型在訓練中從未見過。訓練集擬合參數；驗證集指導選擇（選哪個模型、哪些設定）；測試集是最終的、未動過的讀取。1,000 個樣本按 60/20/20 劃分為 600 訓練、200 驗證、200 測試。只有當每個候選都在同一劃分上用同一指標打分時，比較才有效——先固定協議再調參，而不是看到結果之後再改。

自主實作與記錄

某學習者有 1,000 個標註樣本。(a) 按 60/20/20 分配訓練/驗證/測試。(b) 模型 M1 是在測試集上試評 40 次後選出的——指出協議違規及其後果。(c) 說明比較 M1 與 M2 得分前必須保持一致的條件。

標記為已完成

獨立掌握標準

說明各劃分在評估中的作用，指出測試集複用違規，並在調參前寫出固定協議清單。

權威來源資源與權利證明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 機構: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出處連結 ↗

對應章節 / 時間戳: Section: Model Selection, Underfitting, and Overfitting

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

來源前置條件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（離線彈性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

對比候選資源與選型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Training and Test Sets; Validation

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

02 120 分鐘

基線與資料洩漏

獨立掌握檢查（無 AI 輔助）。

前置要求

完成活動 29-01；本區塊不使用 AI 輔助。

學習目標

計算平凡基線，並識別至少兩種會讓評估報告出虛假成功的洩漏機制。

操作指引

基線是最廉價的誠實預測器：分類任務中即多數類。若 1,000 個樣本中 900 個是類別 0，永遠預測類別 0 就得 0.90——任何模型都要對照 0.90 而非 0.0 來評判。洩漏是評估資料的資訊流入訓練過程：劃分前用全部列計算正規化統計量、重複列跨越劃分、或由標籤派生的特徵。它們都會在沒有真實能力的情況下抬高評估結果。

自主實作與記錄

(a) 資料集 1,000 個樣本、900 個屬類別 0——多數類基線是多少？某模型報告 0.92：真實提升是多少？(b) 某流水線先用全部 1,000 列的均值做特徵標準化，再按 60/20/20 劃分——0.92 的分數可信嗎？為什麼？(c) 若 50 筆測試列同時（重複地）出現在 600 列訓練集中，評估會怎樣？

標記為已完成

獨立掌握標準

正確算出基線，指出至少兩種洩漏機制及修復（僅在訓練集擬合、去重、刪除標籤派生特徵），並報告誠實的較低數字而非被抬高的數字。

權威來源資源與權利證明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 機構: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出處連結 ↗

對應章節 / 時間戳: Section: Model Selection, Underfitting, and Overfitting; generalization discussion

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

來源前置條件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（離線彈性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

對比候選資源與選型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Overfitting; Generalization

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

03 120 分鐘

指標與分布偏移

僅在記錄預測與觀察證據之後允許求助。

前置要求

完成活動 29-02；先記錄預測。

學習目標

把準確率、交叉熵損失與困惑度理解為同一評估證據的不同解析度，並診斷由分布偏移而非模型缺陷導致的分數下降。

操作指引

準確率統計答對個數；交叉熵（正確標籤被賦予機率 p 的平均 −ln p）同時度量置信度——p = 0.9 代價 0.105，p = 0.5 代價 0.693，p = 0.1 代價 2.303。困惑度是 exp（平均交叉熵）：等價的等可能選擇數——總給正確詞元 p = 0.5 的模型困惑度為 2，在 4 個選項上均勻猜測的困惑度為 4。分布偏移打破『訓練≈測試』假設：白天照片訓練的偵測器在夜間評估，是面對新分布的新評估，而不必然是模型缺陷。

自主實作與記錄

(a) 對正確標籤機率 [0.9, 0.8, 0.7, 0.6]，計算平均交叉熵與困惑度。(b) 某模型在測試集上準確率 18/20 = 0.90，但夜間照片只有 11/20——給出兩種不同解釋及各自的後續檢查。(c) 困惑度 2.0 與 10.0 哪個更好？各意味著什麼？

標記為已完成

獨立掌握標準

NLL/困惑度計算正確，說明準確率與損失的區別，並能區分偏移與缺陷、各給出具體檢查。

權威來源資源與權利證明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 機構: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出處連結 ↗

對應章節 / 時間戳: Sections: Loss functions and cross-entropy; dataset shift discussion

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

來源前置條件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（離線彈性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

對比候選資源與選型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Classification accuracy; static versus dynamic evaluation

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

04 120 分鐘

考核：設計一個有效的小實驗

獨立掌握檢查（無 AI 輔助）。

前置要求

完成活動 29-01 至 29-03；獨立閉卷。

學習目標

為一個微型分類任務設計完整、誠實的評估，並為每個選擇辯護。

操作指引

場景：800 個標註樣本、兩個候選模型、類別按 80/20 分布。端到端設計評估並寫成協議：帶數量的劃分、基線、主指標加一個輔助指標、洩漏檢查、測試集何時讀取，以及如何報告陰性結果。不使用 AI 輔助；重點是一份站得住腳的協議，而不是僥倖的數字。

自主實作與記錄

為該 800 樣本任務寫協議（建議 480/160/160），包含多數類基線值、除準確率外要報告的指標、三項洩漏檢查，以及當花哨模型僅與基線持平時你會寫下的句子。

標記為已完成

獨立掌握標準

協議完整且自洽，寫明基線 0.80 與至少一個非準確率指標，包含三項洩漏檢查與單次測試讀取，陰性結果句不粉飾。

權威來源資源與權利證明

linking CC BY-SA 4.0 (book text) / modified MIT (sample code) share-alike

Dive into Deep Learning (d2l.ai)

作者 / 機構: Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola | 原始出處連結 ↗

對應章節 / 時間戳: Section: Model Selection, Underfitting, and Overfitting

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.

來源前置條件:

Linear algebra, calculus, and probability at the Days 19-28 level.

安全回退方案（離線彈性）:

All required formulas and worked numbers are reproduced as original explanations; the book is further reading.

對比候選資源與選型分析 (2)

Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.

fast.ai: Practical Deep Learning for Coders ↗ compared_alternative [Course site copyright / Apache-2.0 (fastai code)]

Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Training and Test Sets; Overfitting

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.