第 53 日：模型評估規範：驗證集損失差距、過擬合與困惑度 (Perplexity)

01 120 分鐘

評估流程規範：訓練損失與驗證損失差距分析

Agent 輔助工程分析

前置要求

第 51 日預訓練循環與第 42 日交叉熵損失。

學習目標

測量訓練步數推進中訓練集與驗證集損失的分離曲線，界定過擬合開始發生的拐點。

操作指引

定義 estimate_loss() 函數：在 torch.no_grad() 與 model.eval() 環境下遍歷 eval_iters 個批次。記錄典型曲線：0步 (4.10, 4.12)；500步 (2.20, 2.25)；1500步 (1.10, 1.95)。指出 1000 步左右驗證集損失停止下降的泛化平台期。

自主實作與記錄

1. 為何評估循環必須在 torch.no_grad() 和 model.eval() 下執行？
2. 訓練集損失持續走低而驗證集損失開始抬頭表明什麼？
3. 挑選最優模型檢查點（Checkpoint）的唯一標準是什麼？

標記為已完成

獨立掌握標準

掌握標準評估代碼範式，明確驗證集損失優選原則。

權威來源資源與權利證明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head (returns cross-entropy loss when targets are given)

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

02 120 分鐘

測試集困惑度 (Perplexity / PPL) 嚴謹量測

Agent 輔助工程分析

前置要求

活動 01 及第 42 日困惑度基礎。

學習目標

計算測試集平均交叉熵損失，並通過自然指數換算得到獨立困惑度 PPL = exp(loss)。

操作指引

對於未參與訓練的 100,000 詞元測試集：(1) 平均交叉熵為 L_val = 1.6094 nats；(2) 困惑度 PPL = exp(1.6094) = 5.0；(3) 與 V=50,000 的隨機猜測基線 (ln 50000 ≈ 10.82, PPL=50,000) 形成鮮明對比。

自主實作與記錄

1. 若驗證集平均損失為 2.3026 nats，對應困惑度是多少？
2. PPL=10 在直觀上代表模型對下一詞預測的不確定度處於什麼水平？
3. 為什麼在評測中只匯報訓練集困惑度是一種極具欺騙性的行為？

標記為已完成

獨立掌握標準

準確推導交叉熵與困惑度換算，深刻解釋其信息論內涵。

權威來源資源與權利證明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head (returns cross-entropy loss when targets are given)

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking Free author-published draft chapter Draft text is not reproduced; linking only

Speech and Language Processing, 3rd ed. draft, Chapter 3: N-gram Language Models

作者 / 機構: Daniel Jurafsky, James H. Martin | 原始出處連結 ↗

對應章節 / 時間戳: Sections 3.1-3.4: n-gram counts, maximum-likelihood estimation, Laplace smoothing, and perplexity

驗證版本與日期: Draft of 2026-08-19 (Verified: 2026-10-30)

入選理由:

Standard reference text for n-gram definitions, count normalization, smoothing, and perplexity used across Day 38.

來源前置條件:

Basic probability

安全回退方案（離線彈性）:

All formulas and worked tables in the curriculum are self-contained and verified by execution.

對比候選資源與選型分析 (2)

Chen & Goodman: An Empirical Study of Smoothing Techniques for Language Modeling ↗ compared_alternative [CC BY-NC-SA 3.0 (ACL Anthology, pre-2016 material)]

Detailed smoothing survey but far beyond the add-one scope of a first n-gram day; optional further reading.

MIT OCW 6.863J Natural Language Processing ↗ compared_alternative [CC BY-NC-SA 4.0]

Solid academic lecture treatment but less current and less integrated with executable count tables than the SLP3 draft.

03 120 分鐘

過擬合、數據死記硬背與數據泄漏深度診斷

Agent 輔助工程分析

前置要求

活動 01 與 02。

學習目標

排查三類隱蔽的數據泄漏：詞元級亂序切分泄漏、測試集與訓練集短語重疊、微型數據未加正則化。

操作指引

分析三種場景：(A) 在切分成序列之前對詞元亂序（Shuffle）（破壞自迴歸因果性，並使相鄰詞跨訓練/驗證劃分洩漏）；(B) 提示與驗證集之間存在重疊的 n-gram；(C) 在極小資料集上將 Dropout 設為零。

自主實作與記錄

1. 為何在切分訓練/驗證集前對詞元數組全局打亂會導致災難性數據泄漏？
2. 如何區分真正的泛化能力與對高頻死板套話的機械背誦？
3. 正確的切分順序是：先切分原始文檔再獨立分詞，還是先全文拼接分詞再隨機切片？

標記為已完成

獨立掌握標準

指出詞元級泄漏的致命機理，建立標準的文檔級切分防禦規範。

權威來源資源與權利證明

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: N/A

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

04 120 分鐘

獨立模型泛化與檢查點審查考核

完全獨立閉卷考核

前置要求

完成活動 01 至 03。

學習目標

面對一個具有明顯過擬合特徵的評測日誌，獨立計算指標並給出客觀誠實的評估報告。

操作指引

某 10M 參數模型在小樣本上的訓練日誌顯示：訓練損失 0.85，驗證損失 1.75。分析其泛化狀態並給出工程改進策略。

自主實作與記錄

1. 該模型處於欠擬合、理想擬合還是嚴重過擬合？
2. 應該採取哪些具體的正則化手段（權重衰減、Dropout、早停機制）？
3. 寫出一句應寫入公開評測報告的誠實陳述。

標記為已完成

獨立掌握標準

面對不理想數據不粉飾、不掩蓋，給出嚴密的診斷與改進方案。

權威來源資源與權利證明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: N/A

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking CC BY 4.0

Google Machine Learning Crash Course

作者 / 機構: Google for Developers | 原始出處連結 ↗

對應章節 / 時間戳: Modules: Training and Test Sets; Overfitting; Classification accuracy

驗證版本與日期: Current online edition (Verified: 2026-10-30)

入選理由:

Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.

來源前置條件:

Basic algebra; short self-contained modules.

安全回退方案（離線彈性）:

Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.

對比候選資源與選型分析 (2)

Dive into Deep Learning (d2l.ai) ↗ compared_selected_for_day29 [CC BY-SA 4.0 (book text) / modified MIT (sample code)]

Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.

scikit-learn: Model Evaluation documentation ↗ compared_alternative [BSD-3-Clause]

Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.