第 54 日：故障陳列室：生成退化死循環、模型幻覺與評測綜合考核

01 120 分鐘

定性故障陳列室：文本退化死循環與事實性幻覺

Agent 輔助工程分析

前置要求

第 52 日採樣策略與第 53 日評估流程規範。

學習目標

歸納自回歸文本生成的三大典型定性故障：複讀機式死循環、極端熵值亂碼、以及語法流利的事實性幻覺。

操作指引

剖析三大案例：(1) 貪心死循環：'The president said that the president said that...'; (2) 極高溫度 (T=3.0) 產生的無序字符亂碼；(3) 自信的幻覺：語法無瑕疵但完全捏造虛假歷史事實。

自主實作與記錄

1. 導致貪心解碼陷入無限死循環的數學反饋迴路是什麼？
2. 為什麼將採樣溫度調高到 2.0 以上會破壞語法結構與可讀性？
3. 為什麼 50M 參數以下的小模型絕不能作為事實知識庫使用？

標記為已完成

獨立掌握標準

能準確歸納生成故障的數學機理，指認小模型的參數容量天花板。

權威來源資源與權利證明

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 機構: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出處連結 ↗

對應章節 / 時間戳: Section 5.3 Repetition and Appendix B Example Generations

驗證版本與日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入選理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

來源前置條件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（離線彈性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

對比候選資源與選型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

linking MIT

nanoGPT: sample.py autoregressive generation script

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: sample.py: temperature and top_k settings passed to model.generate (model.py: GPT.generate)

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

PyTorch inference, torch.multinomial, and Day 48 decoder-only forward pass.

安全回退方案（離線彈性）:

The generation loop is implemented in runnable local scripts.

對比候選資源與選型分析 (2)

llama.cpp sampling implementation ↗ compared_alternative [MIT]

C++ implementation with high performance; nanoGPT is readable Python matched to our curriculum.

PyTorch text generation tutorial ↗ compared_alternative [Modified BSD]

Encoder-decoder model; nanoGPT decoder-only autoregressive loop is the direct fit for modern LLMs.

02 120 分鐘

小模型能力邊界：淺層語言模式與真實邏輯知識的界限

Agent 輔助工程分析

前置要求

活動 01。

學習目標

建立清晰的技術紅線：明確教學小模型所學會的（詞形、語法、音步節奏）與規模化大模型所需能力（推理、世界知識）之間的本質鴻溝。

操作指引

在莎士比亞數據集上訓練的 10M 模型能流利模擬中古英語詞彙和抑揚格五步音步，但無法回答基本的算術或現代常識。堅決拒絕將玩具模型擬合誇大為 'AGI' 或通用推理能力。

自主實作與記錄

1. 10M 字符級或小詞表模型能成功捕獲哪些語言學特徵？
2. 若小模型輸出 '2 + 2 = 5'，這屬於代碼 bug 還是預期內的能力邊界？
3. 在展示模型生成結果時必須包含什麼樣的免責聲明？

標記為已完成

獨立掌握標準

清晰闡明小模型的能力邊界，堅決避免擬人化或誇大宣傳。

權威來源資源與權利證明

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 機構: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出處連結 ↗

對應章節 / 時間戳: N/A

驗證版本與日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入選理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

來源前置條件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（離線彈性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

對比候選資源與選型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: N/A

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

03 120 分鐘

消融實驗設計：模型規模、上下文長度與採樣參數聯動

Agent 輔助工程分析

前置要求

活動 01 與 02。

學習目標

設計受控的消融實驗矩陣，系統對比不同上下文窗口長度 (T=32 vs 128) 與溫度參數對生成文本連貫性的影響。

操作指引

設計包含 6 種組合的評測矩陣，記錄定性連貫性得分 (1-5) 與定量重複率指標 (Distinct-2)。

自主實作與記錄

1. 為什麼上下文窗口僅為 T=32 的模型在長篇生成中必然發生嚴重離題？
2. 經驗上被證明最平衡的開放式生成參數組合是什麼？
3. 給出衡量生成文本詞彙多樣性的定量計算公式。

標記為已完成

獨立掌握標準

能設計嚴密的消融實驗方案並運用定量指標衡量生成質量。

權威來源資源與權利證明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

對應章節 / 時間戳: Section 6.2 Model Variations (Table 3)

驗證版本與日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入選理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

來源前置條件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（離線彈性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

對比候選資源與選型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 機構: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.1 Nucleus Sampling (with Section 2.1 Text Generation Decoding Strategies)

驗證版本與日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入選理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

來源前置條件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（離線彈性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

對比候選資源與選型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

04 120 分鐘

文本生成、過擬合與評估綜合掌握度終結考核

完全獨立閉卷考核

前置要求

完成活動 01 至 03。

學習目標

完成閉卷綜合結業考試，全面回答採樣公式、困惑度換算、防泄漏紀律與模型局限性四大核心板塊。

操作指引

解答涵蓋核採樣閾值方程、測試集困惑度精確計算、三大大數據集切分金科玉律以及低訓練損失陷阱的綜合問答。

自主實作與記錄

1. 寫出 Top-p 核採樣的累積概率選擇方程。
2. 將驗證集平均損失 1.7918 nats 換算為精確的困惑度數值。
3. 列舉防止數據泄漏的三大黃金法則。
4. 為什麼極低的訓練損失絕不能保證生成文本的連貫與高質量？

標記為已完成

獨立掌握標準

全面準確回答四大核心問題，展現對生成與評測體系的完整獨立掌握。

權威來源資源與權利證明

linking MIT

nanoGPT: sample.py autoregressive generation script

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: sample.py: temperature and top_k settings passed to model.generate (model.py: GPT.generate)

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

PyTorch inference, torch.multinomial, and Day 48 decoder-only forward pass.

安全回退方案（離線彈性）:

The generation loop is implemented in runnable local scripts.

對比候選資源與選型分析 (2)

llama.cpp sampling implementation ↗ compared_alternative [MIT]

C++ implementation with high performance; nanoGPT is readable Python matched to our curriculum.

PyTorch text generation tutorial ↗ compared_alternative [Modified BSD]

Encoder-decoder model; nanoGPT decoder-only autoregressive loop is the direct fit for modern LLMs.

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 機構: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.1 Nucleus Sampling (with Section 2.1 Text Generation Decoding Strategies)

驗證版本與日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入選理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

來源前置條件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（離線彈性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

對比候選資源與選型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.