45 / 90

第 45 天：多頭注意力機制與數值穩定性

將多頭注意力解構為線性子空間投影 W^Q, W^K, W^V, W^O。掌握跨 h 個並行注意力頭的張量重塑 (B, h, T, d_k)，驗證嚴謹的參數總帳，並解決注意力實現中的邊界數值不穩定性。

模型與訓練 480 分鐘主動學習（8 小時專注研讀）
01 120 分鐘

多頭子空間：為什麼以及如何分割 d_model

助教輔助工程檢查

前置要求

第 43-44 天的縮放點積注意力與因果遮罩機制。

學習目標

闡明多頭注意力的表徵必要性，形式化線性投影 W^Q, W^K, W^V, W^O，並推導微型模型與標準 GPT 規模的參數總帳。

操作指引

必做任務：解釋為什麼單頭注意力會將多種截然不同的關聯（如句法主謂依存與長程語義共指）壓縮為單一的加權平均。形式化 MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O，其中每個 head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)。設 d_model = 4, h = 2，因此 d_k = d_v = 2。推導參數總帳：W^Q, W^K, W^V, W^O 均為 (4, 4) 矩陣，各含 16 個參數（無偏置共 64 參數，含偏置共 80 參數）。計算 GPT-2 Small 的參數量：d_model = 768, h = 12, d_k = 64（權重 4 * 768*768 = 2,359,296，偏置 4*768 = 3,072，總計 2,362,368）。

自主實作與記錄

1. 為什麼單個注意力頭無法在關注相鄰標記的同時關注遠距離句法依存？
2. 寫出 d_model = 4, h = 2 在含偏置和不含偏置情況下的精確參數總帳。
3. 計算 GPT-2 Small 注意力層的參數總帳（d_model = 768, h = 12）。

獨立掌握標準

準確闡述多頭子空間路由的表徵意義，並無誤推導出微型模型與 GPT-2 規模的完整參數台帳。

權威來源資源與權利證明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.2.2: Multi-Head Attention architecture, linear projections, and parameter equations

驗證版本與日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

入選理由:

The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

來源前置條件:

Linear algebra, matrix multiplication, softmax probability distributions

安全回退方案（離線彈性）:

Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

對比候選資源與選型分析 (3)
Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

linking MIT

The Annotated Transformer

作者 / 機構: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出處連結 ↗

對應章節 / 時間戳: MultiHeadedAttention module implementation: linear projections and subspace splitting

驗證版本與日期: Updated 2022 edition (Verified: 2026-10-30)

入選理由:

Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.

來源前置條件:

Intermediate Python, PyTorch broadcasting and tensor slicing semantics

安全回退方案（離線彈性）:

The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.

對比候選資源與選型分析 (3)
Jay Alammar: The Illustrated Transformer ↗ compared_alternative [CC BY-NC 4.0]

Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.

Stanford CS224N: Natural Language Processing with Deep Learning (Lecture 10) ↗ compared_alternative [Stanford University Copyright]

Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.

Vaswani et al.: Attention Is All You Need ↗ compared_selected_co_primary [arXiv Open Access]

The original primary paper; Annotated Transformer annotates it directly.

02 120 分鐘

張量重塑、轉置與批處理注意力 (B, h, T, d_k)

獨立掌握檢查

前置要求

完成第 45 天活動 1。

學習目標

完整追蹤注意力前向計算中的張量階數與維度變換：投影、分頭轉置 (B, T, h, d_k) -> (B, h, T, d_k)、批處理矩陣乘法及拼接。

操作指引

必做任務：追蹤輸入張量 X 形狀為 (B, T, d_model) = (1, 2, 4) 在 h=2, d_k=2 多頭注意力中的流動過程。步驟 1：線性投影 Q = X W^Q, K = X W^K, V = X W^V，形狀為 (1, 2, 4)。步驟 2：view 重塑為 (B, T, h, d_k) = (1, 2, 2, 2)。步驟 3：轉置維度 1 和 2 為 (B, h, T, d_k) = (1, 2, 2, 2)。步驟 4：分頭矩陣乘法 Q @ K^T 得到 (1, 2, 2, 2)。步驟 5：除以 sqrt(2) 並施加因果遮罩。步驟 6：與 V 相乘得 (1, 2, 2, 2)。步驟 7：轉置回 (1, 2, 2, 2) 並展平拼接為 (1, 2, 4)。步驟 8：透過 W^O 投影為 (1, 2, 4)。使用 X = [[1, 0, 1, 0], [0, 2, 0, 1]] 與單位投影權重手算數值。

自主實作與記錄

1. 列出從輸入 X 到最終輸出 Out 的每一步張量形狀。
2. 對 X = [[1, 0, 1, 0], [0, 2, 0, 1]] 及單位陣投影，手算頭 0 與頭 1 的因果注意力矩陣。
3. 計算經 W^O 投影後的最終拼接輸出。

獨立掌握標準

完整推導所有張量形狀變化，並準確計算出雙頭注意力的具體輸出數值。

權威來源資源與權利證明

linking MIT

nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: view and transpose operations: B, T, nh, hs -> B, nh, T, hs

驗證版本與日期: GitHub master commit (Verified: 2026-10-30)

入選理由:

Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

來源前置條件:

Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

安全回退方案（離線彈性）:

Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

對比候選資源與選型分析 (3)
Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.

linking MIT

The Annotated Transformer

作者 / 機構: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出處連結 ↗

對應章節 / 時間戳: MultiHeadedAttention forward pass tensor reshaping and transposition trace

驗證版本與日期: Updated 2022 edition (Verified: 2026-10-30)

入選理由:

Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.

來源前置條件:

Intermediate Python, PyTorch broadcasting and tensor slicing semantics

安全回退方案（離線彈性）:

The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.

對比候選資源與選型分析 (3)
Jay Alammar: The Illustrated Transformer ↗ compared_alternative [CC BY-NC 4.0]

Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.

Stanford CS224N: Natural Language Processing with Deep Learning (Lecture 10) ↗ compared_alternative [Stanford University Copyright]

Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.

Vaswani et al.: Attention Is All You Need ↗ compared_selected_co_primary [arXiv Open Access]

The original primary paper; Annotated Transformer annotates it directly.

03 120 分鐘

注意力中的數值穩定性：最大值平移、精度邊界與遮罩陷阱

助教輔助工程檢查

前置要求

完成第 45 天活動 2。

學習目標

剖析並解決注意力實現中的浮點數值漏洞：最大值平移 Softmax 不變性證明、全被遮罩行的崩潰防範，以及 fp16/fp32 動態範圍限制。

操作指引

必做任務：深入分析三大關鍵數值邊界場景。場景 1：未平移指數溢出。在 float32 中，exp(z) 在 z >= 88.7 時上溢為 inf；在 float16 中，z >= 11.1 即發生溢出。代數證明平移不變性 softmax(z)_i = softmax(z - c)_i。場景 2：全遮罩行崩潰（All-masked row）。當某一行全為 -inf 時（如帶有填充 padding 的交叉注意力），max(z) = -inf，導致 -inf - (-inf) = NaN。給出避免產生 NaN 的安全實現方式。場景 3：混合精度下的遮罩常數選取。解釋為什麼在 float16 下 -1e4 比 -1e9 更安全。

自主實作與記錄

1. 代數證明對於任意標量 c，均有 softmax(z - c) = softmax(z)。
2. 在 PyTorch float16 中計算 torch.exp(torch.tensor(15.0, dtype=torch.float16)) 會發生什麼？
3. 當遇到完全被遮罩的行時，工程上如何確保模型不輸出 NaN？

獨立掌握標準

給出嚴謹的平移不變性代數證明，並準確診斷 float16 上溢與全遮罩行工程處理解決方案。

權威來源資源與權利證明

linking MIT

nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: FlashAttention / scaled_dot_product_attention numerical stability and fallback path

驗證版本與日期: GitHub master commit (Verified: 2026-10-30)

入選理由:

Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

來源前置條件:

Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

安全回退方案（離線彈性）:

Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

對比候選資源與選型分析 (3)
Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.2.1: Softmax numerical behavior and score magnitude scaling

驗證版本與日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

入選理由:

The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

來源前置條件:

Linear algebra, matrix multiplication, softmax probability distributions

安全回退方案（離線彈性）:

Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

對比候選資源與選型分析 (3)
Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

04 120 分鐘

獨立掌握檢查：多頭台帳、因果不變性與穩定性檢驗

獨立掌握檢查

前置要求

活動 45-01 至 45-03。

學習目標

在聲明無 AI 輔助條件下，獨立完成中型模型多頭參數核算、端到端前向數值追蹤、跨頭因果不變性證明與數值穩定性檢驗。

操作指引

必做任務：閉卷獨立作答。1. 計算中型 GPT（d_model = 1024, h = 16）單層注意力塊的精確參數量（包含與不含偏置的 W^Q, W^K, W^V, W^O）。2. 證明為什麼當每個單頭滿足因果性時，輸出線性投影 W^O 絕不會跨標記洩漏未來資訊。3. 給定未縮放對數幾率 [500.0, 502.0]，計算其精確的最大值平移 Softmax 機率。4. 完成客觀選擇題。

自主實作與記錄

1. d_model = 1024, h = 16 注意力層的總參數量（含與不含偏置）？
2. 證明為什麼 W^O 絕不會破壞各單頭的因果不變性？
3. 計算對數幾率 [500.0, 502.0] 的精確 Softmax 輸出。
4. 完成 3 道單項選擇題。

客觀理解度測試：多頭注意力與數值穩定性

請回答以下診斷題目檢驗掌握情況。提交前請誠實聲明你的輔助條件。

聲明輔助條件

說明：無 AI 輔助屬於誠實聲明條件，非平台監控或強制檢測手段。

1.

在模型嵌入維度 d_model = 768、注意力頭數 h = 12 的多頭注意力塊中，每個單頭的特徵維度 d_k 是多少？

2.

對於包含投影矩陣 W^Q, W^K, W^V, W^O in R^{d_model x d_model}（不含偏置）的標準多頭注意力層，當 d_model = 768 時總權重參數量是多少？

3.

在採用最大值平移的 Softmax 實現 softmax(z)_i = exp(z_i - max(z)) / sum_j exp(z_j - max(z)) 中，減去 max(z) 的核心依據是什麼？

作答應證與歷史記錄

    獨立掌握標準

    精確計算中型模型參數總帳（4,194,304 / 4,198,400），嚴密證明 W^O 的逐位置因果保持性，客觀測試題獲 100% 滿分。

    權威來源資源與權利證明

    linking arXiv.org perpetual non-exclusive license / Open Access

    Attention Is All You Need

    作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

    對應章節 / 時間戳: Section 3.2.2: Multi-Head Attention parameter count equations and projection matrices

    驗證版本與日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)

    入選理由:

    The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.

    來源前置條件:

    Linear algebra, matrix multiplication, softmax probability distributions

    安全回退方案（離線彈性）:

    Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.

    對比候選資源與選型分析 (3)
    Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015) ↗ compared_alternative [arXiv Open Access]

    Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.

    Shaw et al.: Self-Attention with Relative Position Representations (NAACL 2018) ↗ compared_deferred [arXiv Open Access]

    Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.

    Karpathy nanoGPT (causal self-attention module) ↗ compared_selected_co_primary [MIT]

    Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.

    linking MIT

    nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs

    作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

    對應章節 / 時間戳: model.py: CausalSelfAttention parameter accounting and projection weights verification

    驗證版本與日期: GitHub master commit (Verified: 2026-10-30)

    入選理由:

    Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.

    來源前置條件:

    Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics

    安全回退方案（離線彈性）:

    Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.

    對比候選資源與選型分析 (3)
    Hugging Face transformers (GPT2Attention) ↗ compared_rejected_for_foundations [Apache-2.0]

    Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.

    Harvard NLP: The Annotated Transformer ↗ compared_selected_co_primary [MIT]

    Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.

    Karpathy: build-nanogpt (YouTube & GitHub 2024) ↗ compared_alternative [MIT]

    Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.