45 / 90

第 45 天：多头注意力机制与数值稳定性

将多头注意力解构为线性子空间投影 W^Q, W^K, W^V, W^O。掌握跨 h 个并行注意力头的张量重塑 (B, h, T, d_k)，验证严谨的参数总账，并解决注意力实现中的边界数值不稳定性。

模型与训练 480 分钟主动学习（8 小时专注研读）
01 120 分钟

多头子空间：为什么以及如何分割 d_model

助教辅助工程检查

前置要求

第 43-44 天的缩放点积注意力与因果掩码机制。

学习目标

阐明多头注意力的表征必要性，形式化线性投影 W^Q, W^K, W^V, W^O，并推导微型模型与标准 GPT 规模的参数总账。

操作指引

必做任务：解释为什么单头注意力会将多种截然不同的关联（如句法主谓依存与长程语义共指）压缩为单一的加权平均。形式化 MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O，其中每个 head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)。设 d_model = 4, h = 2，因此 d_k = d_v = 2。推导参数总账：W^Q, W^K, W^V, W^O 均为 (4, 4) 矩阵，各含 16 个参数（无偏置共 64 参数，含偏置共 80 参数）。计算 GPT-2 Small 的参数量：d_model = 768, h = 12, d_k = 64（权重 4 * 768*768 = 2,359,296，偏置 4*768 = 3,072，总计 2,362,368）。

自主演练与记录

1. 为什么单个注意力头无法在关注相邻词元的同时关注远距离句法依存？
2. 写出 d_model = 4, h = 2 在含偏置和不含偏置情况下的精确参数总账。
3. 计算 GPT-2 Small 注意力层的参数总账（d_model = 768, h = 12）。

独立掌握标准

准确阐述多头子空间路由的表征意义，并无误推导出微型模型与 GPT-2 规模的完整参数台账。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.2.2: Multi-Head Attention architecture, linear projections, and parameter equations

验证版本与日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

来源前置条件:

Linear algebra, matrix multiplication, softmax probability distributions

安全回退方案（离线弹性）:

Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

对比候选资源与选型分析 (3)
Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: MultiHeadedAttention module implementation: linear projections and subspace splitting

验证版本与日期: Updated 2022 edition (Verified: 2026-10-30)

入选理由:

Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.

来源前置条件:

Intermediate Python, PyTorch broadcasting and tensor slicing semantics

安全回退方案（离线弹性）:

The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.

对比候选资源与选型分析 (3)
Jay Alammar: The Illustrated Transformer ↗ compared_alternative [CC BY-NC 4.0]

Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.

Stanford CS224N: Natural Language Processing with Deep Learning (Lecture 10) ↗ compared_alternative [Stanford University Copyright]

Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.

Vaswani et al.: Attention Is All You Need ↗ compared_selected_co_primary [arXiv Open Access]

The original primary paper; Annotated Transformer annotates it directly.

02 120 分钟

张量重塑、转置与批处理注意力 (B, h, T, d_k)

独立掌握检查

前置要求

完成第 45 天活动 1。

学习目标

完整追踪注意力前向计算中的张量阶数与维度变换：投影、分头转置 (B, T, h, d_k) -> (B, h, T, d_k)、批处理矩阵乘法及拼接。

操作指引

必做任务：追踪输入张量 X 形状为 (B, T, d_model) = (1, 2, 4) 在 h=2, d_k=2 多头注意力中的流动过程。步骤 1：线性投影 Q = X W^Q, K = X W^K, V = X W^V，形状为 (1, 2, 4)。步骤 2：view 重塑为 (B, T, h, d_k) = (1, 2, 2, 2)。步骤 3：转置维度 1 和 2 为 (B, h, T, d_k) = (1, 2, 2, 2)。步骤 4：分头矩阵乘法 Q @ K^T 得到 (1, 2, 2, 2)。步骤 5：除以 sqrt(2) 并施加因果掩码。步骤 6：与 V 相乘得 (1, 2, 2, 2)。步骤 7：转置回 (1, 2, 2, 2) 并展平拼接为 (1, 2, 4)。步骤 8：通过 W^O 投影为 (1, 2, 4)。使用 X = [[1, 0, 1, 0], [0, 2, 0, 1]] 与单位投影权重手算数值。

自主演练与记录

1. 列出从输入 X 到最终输出 Out 的每一步张量形状。
2. 对 X = [[1, 0, 1, 0], [0, 2, 0, 1]] 及单位阵投影，手算头 0 与头 1 的因果注意力矩阵。
3. 计算经 W^O 投影后的最终拼接输出。

独立掌握标准

完整推导所有张量形状变化，并准确计算出双头注意力的具体输出数值。

权威源资源与权利证明

linking MIT

nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: view and transpose operations: B, T, nh, hs -> B, nh, T, hs

验证版本与日期: GitHub master commit (Verified: 2026-10-30)

入选理由:

Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

来源前置条件:

Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

安全回退方案（离线弹性）:

Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

对比候选资源与选型分析 (3)
Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: MultiHeadedAttention forward pass tensor reshaping and transposition trace

验证版本与日期: Updated 2022 edition (Verified: 2026-10-30)

入选理由:

Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.

来源前置条件:

Intermediate Python, PyTorch broadcasting and tensor slicing semantics

安全回退方案（离线弹性）:

The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.

对比候选资源与选型分析 (3)
Jay Alammar: The Illustrated Transformer ↗ compared_alternative [CC BY-NC 4.0]

Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.

Stanford CS224N: Natural Language Processing with Deep Learning (Lecture 10) ↗ compared_alternative [Stanford University Copyright]

Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.

Vaswani et al.: Attention Is All You Need ↗ compared_selected_co_primary [arXiv Open Access]

The original primary paper; Annotated Transformer annotates it directly.

03 120 分钟

注意力中的数值稳定性：最大值平移、精度边界与掩码陷阱

助教辅助工程检查

前置要求

完成第 45 天活动 2。

学习目标

剖析并解决注意力实现中的浮点数值漏洞：最大值平移 Softmax 不变性证明、全被掩码行的崩溃防范，以及 fp16/fp32 动态范围限制。

操作指引

必做任务：深入分析三大关键数值边界场景。场景 1：未平移指数溢出。在 float32 中，exp(z) 在 z >= 88.7 时上溢为 inf；在 float16 中，z >= 11.1 即发生溢出。代数证明平移不变性 softmax(z)_i = softmax(z - c)_i。场景 2：全掩码行崩溃（All-masked row）。当某一行全为 -inf 时（如带有填充 padding 的交叉注意力），max(z) = -inf，导致 -inf - (-inf) = NaN。给出避免产生 NaN 的安全实现方式。场景 3：混合精度下的掩码常数选取。解释为什么在 float16 下 -1e4 比 -1e9 更安全。

自主演练与记录

1. 代数证明对于任意标量 c，均有 softmax(z - c) = softmax(z)。
2. 在 PyTorch float16 中计算 torch.exp(torch.tensor(15.0, dtype=torch.float16)) 会发生什么？
3. 当遇到完全被掩码的行时，工程上如何确保模型不输出 NaN？

独立掌握标准

给出严谨的平移不变性代数证明，并准确诊断 float16 上溢与全掩码行工程处理方案。

权威源资源与权利证明

linking MIT

nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: FlashAttention / scaled_dot_product_attention numerical stability and fallback path

验证版本与日期: GitHub master commit (Verified: 2026-10-30)

入选理由:

Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

来源前置条件:

Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

安全回退方案（离线弹性）:

Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

对比候选资源与选型分析 (3)
Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.2.1: Softmax numerical behavior and score magnitude scaling

验证版本与日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

来源前置条件:

Linear algebra, matrix multiplication, softmax probability distributions

安全回退方案（离线弹性）:

Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

对比候选资源与选型分析 (3)
Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

04 120 分钟

独立掌握检查：多头台账、因果不变性与稳定性检验

独立掌握检查

前置要求

活动 45-01 至 45-03。

学习目标

在声明无 AI 辅助条件下，独立完成中型模型多头参数核算、端到端前向数值追踪、跨头因果不变性证明与数值稳定性检验。

操作指引

必做任务：闭卷独立作答。1. 计算中型 GPT（d_model = 1024, h = 16）单层注意力块的精确参数量（包含与不含偏置的 W^Q, W^K, W^V, W^O）。2. 证明为什么当每个单头满足因果性时，输出线性投影 W^O 绝不会跨词元泄露未来信息。3. 给定未缩放对数几率 [500.0, 502.0]，计算其精确的最大值平移 Softmax 概率。4. 完成客观选择题。

自主演练与记录

1. d_model = 1024, h = 16 注意力层的总参数量（含与不含偏置）？
2. 证明为什么 W^O 绝不会破坏各单头的因果不变性？
3. 计算对数几率 [500.0, 502.0] 的精确 Softmax 输出。
4. 完成 3 道单项选择题。

客观理解度测试：多头注意力与数值稳定性

请回答以下诊断题目检验掌握情况。提交前请诚实声明你的辅助条件。

声明辅助条件

说明：无 AI 辅助属于诚实声明条件，非平台监控或强制检测手段。

1.

在模型嵌入维度 d_model = 768、注意力头数 h = 12 的多头注意力块中，每个单头的特征维度 d_k 是多少？

2.

对于包含投影矩阵 W^Q, W^K, W^V, W^O in R^{d_model x d_model}（不含偏置）的标准多头注意力层，当 d_model = 768 时总权重参数量是多少？

3.

在采用最大值平移的 Softmax 实现 softmax(z)_i = exp(z_i - max(z)) / sum_j exp(z_j - max(z)) 中，减去 max(z) 的核心依据是什么？

作答应证与历史记录

    独立掌握标准

    精确计算中型模型参数总账（4,194,304 / 4,198,400），严密证明 W^O 的逐位置因果保持性，客观测试题获 100% 满分。

    权威源资源与权利证明

    linking arXiv.org perpetual non-exclusive license / Open Access

    Attention Is All You Need

    作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

    对应章节 / 时间戳: Section 3.2.2: Multi-Head Attention parameter count equations and projection matrices

    验证版本与日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

    入选理由:

    The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

    来源前置条件:

    Linear algebra, matrix multiplication, softmax probability distributions

    安全回退方案（离线弹性）:

    Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

    对比候选资源与选型分析 (3)
    Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

    Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

    Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

    Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

    Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

    Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

    linking MIT

    nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

    作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

    对应章节 / 时间戳: model.py: CausalSelfAttention parameter accounting and projection weights verification

    验证版本与日期: GitHub master commit (Verified: 2026-10-30)

    入选理由:

    Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

    来源前置条件:

    Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

    安全回退方案（离线弹性）:

    Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

    对比候选资源与选型分析 (3)
    Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

    Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

    Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

    Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

    Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

    Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.