第 54 日：故障陈列室：生成退化死循环、模型幻觉与评测综合考核

01 120 分钟

定性故障陈列室：文本退化死循环与事实性幻觉

Agent 辅助工程分析

前置要求

第 52 日采样策略与第 53 日评估流程规范。

学习目标

归纳自回归文本生成的三大典型定性故障：复读机式死循环、极端熵值乱码、以及语法流利的事实性幻觉。

操作指引

剖析三大案例：(1) 贪心死循环：'The president said that the president said that...'; (2) 极高温度 (T=3.0) 产生的无序字符乱码；(3) 自信的幻觉：语法无瑕疵但完全捏造虚假历史事实。

自主演练与记录

1. 导致贪心解码陷入无限死循环的数学反馈回路是什么？
2. 为什么将采样温度调高到 2.0 以上会破坏语法结构与可读性？
3. 为什么 50M 参数以下的小模型绝不能作为事实知识库使用？

标记为已完成

独立掌握标准

能准确归纳生成故障的数学机理，指认小模型的参数容量天花板。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 机构: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出处链接 ↗

对应章节 / 时间戳: Section 5.3 Repetition and Appendix B Example Generations

验证版本与日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入选理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

来源前置条件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（离线弹性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

对比候选资源与选型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

linking MIT

nanoGPT: sample.py autoregressive generation script

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: sample.py: temperature and top_k settings passed to model.generate (model.py: GPT.generate)

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

PyTorch inference, torch.multinomial, and Day 48 decoder-only forward pass.

安全回退方案（离线弹性）:

The generation loop is implemented in runnable local scripts.

对比候选资源与选型分析 (2)

llama.cpp sampling implementation ↗ compared_alternative [MIT]

C++ implementation with high performance; nanoGPT is readable Python matched to our curriculum.

PyTorch text generation tutorial ↗ compared_alternative [Modified BSD]

Encoder-decoder model; nanoGPT decoder-only autoregressive loop is the direct fit for modern LLMs.

02 120 分钟

小模型能力边界：浅层语言模式与真实逻辑知识的界限

Agent 辅助工程分析

前置要求

活动 01。

学习目标

建立清晰的技术红线：明确教学小模型所学会的（词形、语法、音步节奏）与规模化大模型所需能力（推理、世界知识）之间的本质鸿沟。

操作指引

在莎士比亚数据集上训练的 10M 模型能流利模拟中古英语词汇和抑扬格五步音步，但无法回答基本的算术或现代常识。坚决拒绝将玩具模型拟合夸大为 'AGI' 或通用推理能力。

自主演练与记录

1. 10M 字符级或小词表模型能成功捕获哪些语言学特征？
2. 若小模型输出 '2 + 2 = 5'，这属于代码 bug 还是预期内的能力边界？
3. 在展示模型生成结果时必须包含什么样的免责声明？

标记为已完成

独立掌握标准

清晰阐明小模型的能力边界，坚决避免拟人化或夸大宣传。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 机构: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出处链接 ↗

对应章节 / 时间戳: N/A

验证版本与日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入选理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

来源前置条件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（离线弹性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

对比候选资源与选型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: N/A

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

03 120 分钟

消融实验设计：模型规模、上下文长度与采样参数联动

Agent 辅助工程分析

前置要求

活动 01 与 02。

学习目标

设计受控的消融实验矩阵，系统对比不同上下文窗口长度 (T=32 vs 128) 与温度参数对生成文本连贯性的影响。

操作指引

设计包含 6 种组合的评测矩阵，记录定性连贯性得分 (1-5) 与定量重复率指标 (Distinct-2)。

自主演练与记录

1. 为什么上下文窗口仅为 T=32 的模型在长篇生成中必然发生严重离题？
2. 经验上被证明最平衡的开放式生成参数组合是什么？
3. 给出衡量生成文本词汇多样性的定量计算公式。

标记为已完成

独立掌握标准

能设计严密的消融实验方案并运用定量指标衡量生成质量。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 6.2 Model Variations (Table 3)

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 机构: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.1 Nucleus Sampling (with Section 2.1 Text Generation Decoding Strategies)

验证版本与日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入选理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

来源前置条件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（离线弹性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

对比候选资源与选型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.

04 120 分钟

文本生成、过拟合与评估综合掌握度终结考核

完全独立闭卷考核

前置要求

完成活动 01 至 03。

学习目标

完成闭卷综合结业考试，全面回答采样公式、困惑度换算、防泄漏纪律与模型局限性四大核心板块。

操作指引

解答涵盖核采样阈值方程、测试集困惑度精确计算、三大数据集切分金科玉律以及低训练损失陷阱的综合问答。

自主演练与记录

1. 写出 Top-p 核采样的累积概率选择方程。
2. 将验证集平均损失 1.7918 nats 换算为精确的困惑度数值。
3. 列举防止数据泄漏的三大黄金法则。
4. 为什么极低的训练损失绝不能保证生成文本的连贯与高质量？

标记为已完成

独立掌握标准

全面准确回答四大核心问题，展现对生成与评测体系的完整独立掌握。

权威源资源与权利证明

linking MIT

nanoGPT: sample.py autoregressive generation script

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: sample.py: temperature and top_k settings passed to model.generate (model.py: GPT.generate)

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

PyTorch inference, torch.multinomial, and Day 48 decoder-only forward pass.

安全回退方案（离线弹性）:

The generation loop is implemented in runnable local scripts.

对比候选资源与选型分析 (2)

llama.cpp sampling implementation ↗ compared_alternative [MIT]

C++ implementation with high performance; nanoGPT is readable Python matched to our curriculum.

PyTorch text generation tutorial ↗ compared_alternative [Modified BSD]

Encoder-decoder model; nanoGPT decoder-only autoregressive loop is the direct fit for modern LLMs.

linking arXiv non-exclusive license / Open Access

The Curious Case of Neural Text Degeneration

作者 / 机构: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.1 Nucleus Sampling (with Section 2.1 Text Generation Decoding Strategies)

验证版本与日期: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)

入选理由:

Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).

来源前置条件:

Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.

安全回退方案（离线弹性）:

Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.

对比候选资源与选型分析 (2)

Fan et al. Hierarchical Neural Story Generation (Top-k) ↗ compared_alternative [Open Access]

Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.

Hugging Face Generation Strategies Guide ↗ compared_alternative [Apache 2.0]

Practical library guide; Holtzman provides the foundational mathematical reasoning.