46 / 90

第 46 日：Transformer 模組組裝：LayerNorm、RMSNorm 與殘差連接

使用小數值矩陣精確推導 LayerNorm 與 RMSNorm 歸一化，解釋為何殘差相加能創造梯度高速公路，並深入對比 Pre-LN 與 Post-LN 的收斂穩定性。

模型、預訓練與微調 (LoRA) 480 分鐘專注學習（8 小時）
01 120 分鐘

Layer Normalization 與 RMSNorm：數學原理與數值穩定性

Agent 輔助工程分析

前置要求

第 28 日方差與標準差；第 22 日向量代數基礎。

學習目標

針對輸入向量 x = [2.0, -1.0, 0.0, 3.0] 逐步計算 LayerNorm 與 RMSNorm，闡釋現代架構偏好 RMSNorm 的根本原因。

操作指引

對於 d=4 的向量 x = [2.0, -1.0, 0.0, 3.0]：(1) 計算均值 mu = 1.0、方差 sigma^2 = 2.5，以及 eps=1e-5 下的 LayerNorm 輸出。(2) 計算均方根 RMS(x) = sqrt(mean(x^2)) = sqrt(3.5) ≈ 1.8708，及 RMSNorm 輸出 x / RMS(x)。(3) 比較計算成本：RMSNorm 省去了在顯存上對均值的遍歷歸約。

自主實作與記錄

1. 在 gamma=1, beta=0 下計算 LayerNorm(x)，保留 4 位小數。
2. 在 gamma=1 下計算 RMSNorm(x)，保留 4 位小數。
3. 為何 RMSNorm 在 GPU 核函數執行時具有更顯著的顯存帶寬優勢？

獨立掌握標準

學習者能夠手算完成兩類歸一化，並能準確論述 RMSNorm 的訪存帶寬優勢。

權威來源資源與權利證明

linking arXiv non-exclusive license / Open Access

Root Mean Square Layer Normalization

作者 / 機構: Biao Zhang, Rico Sennrich (University of Edinburgh) | 原始出處連結 ↗

對應章節 / 時間戳: Section 4 RMSNorm (Equation 4) and Section 4.1 Invariance Analysis

驗證版本與日期: NeurIPS 2019 / arXiv:1910.07467v1 (Verified: 2026-10-30)

入選理由:

Primary paper showing that mean-centering in LayerNorm can be discarded without performance loss, speeding up modern LLMs (LLaMA, Gemma).

來源前置條件:

Variance, standard deviation, and LayerNorm basics.

安全回退方案（離線彈性）:

RMSNorm formulas and numerical comparisons are derived completely in the lesson.

對比候選資源與選型分析 (2)
Ba et al. Layer Normalization (2016) ↗ compared_selected_for_day46 [Open Access]

Original LayerNorm formulation with mean-centering; RMSNorm is compared directly against it as an optimization.

Ioffe & Szegedy Batch Normalization (2015) ↗ compared_alternative [Open Access]

Batch-dependent statistics; unsuitable for autoregressive sequence modeling.

linking MIT

The Annotated Transformer

作者 / 機構: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出處連結 ↗

對應章節 / 時間戳: Encoder and Decoder Stacks (class LayerNorm, class SublayerConnection)

驗證版本與日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入選理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

來源前置條件:

PyTorch tensor operations and attention theory.

安全回退方案（離線彈性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

對比候選資源與選型分析 (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

02 120 分鐘

殘差連接與恆等梯度高速公路

Agent 輔助工程分析

前置要求

活動 01 及第 35 日反向傳播鏈式法則。

學習目標

推導殘差加法 y = x + F(x) 產生無衰減恆等梯度項 dL/dx = dL/dy + dL/dy * dF/dx 的數學機制。

操作指引

寫出 y = x + F(x) 關於 x 的向量導數。證明即使 dF/dx 極小或趨近飽和為 0，dL/dx 依然保留恆等項 dL/dy，從而徹底消除百層深層 Transformer 中的梯度消失風險。

自主實作與記錄

1. 寫出 y = x + F(x) 下 dL/dx 的鏈式法則展開式。
2. 若 dF/dx 的特徵值模長均 < 0.1，為何反向梯度不會衰減殆盡？
3. 說明 x + F(x) 加法能夠執行所要求的張量維度一致性條件。

獨立掌握標準

推導出梯度高速公路公式並指明嚴格的張量形狀不變性。

權威來源資源與權利證明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.1 Encoder and Decoder Stacks (residual connection around each sub-layer, then layer normalization)

驗證版本與日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入選理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

來源前置條件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（離線彈性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

對比候選資源與選型分析 (2)
Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)
GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

03 120 分鐘

Pre-LN 與 Post-LN 架構對比分析

Agent 輔助工程分析

前置要求

活動 01 與 02。

學習目標

對比 Pre-LN (x = x + Attn(LN(x))) 與原版 Post-LN (x = LN(x + Attn(x)))。解釋 Xiong et al.（2020）關於初始化和學習率預熱的結論，並將 GPT-2、Llama 2 作為有文獻依據的預歸一化實例（Llama 2 使用 RMSNorm），而非所有模型都採用該結構的證據。

操作指引

追蹤兩種配置下的激活與梯度流，並說明兩者訓練穩定性的差異。Xiong et al.（2020）證明：在初始化時，Post-LN Transformer 靠近輸出層的參數梯度期望值很大，因此需要學習率預熱（warm-up）階段才能穩定訓練；Pre-LN 在初始化時梯度表現良好，在該論文的實驗中，不使用預熱訓練的 Pre-LN Transformer 也取得了相當的結果。GPT-2（Section 2.3）把層歸一化移到每個子區塊的輸入端，Llama 2（Section 2.2）採用 RMSNorm 預歸一化。這些只是例子，並非全面調查：不要聲稱所有現代大模型都使用 Pre-LN。

自主實作與記錄

1. 給出 Pre-LN 與 Post-LN 模組的數據流公式。
2. 在 Post-LN 中，為何殘差路徑無法作為純淨的恆等高速通道？
3. 根據 Xiong et al.（2020），為什麼 Pre-LN 可以不用 Post-LN 所需的學習率預熱階段進行訓練？

獨立掌握標準

清晰闡明兩者結構差異，準確將訓練穩定性歸因於未被歸一化破壞的主幹殘差。

權威來源資源與權利證明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 機構: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出處連結 ↗

對應章節 / 時間戳: Section 3.1 Encoder and Decoder Stacks (LayerNorm(x + Sublayer(x)), the post-LN placement)

驗證版本與日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入選理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

來源前置條件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（離線彈性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

對比候選資源與選型分析 (2)
Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

The Annotated Transformer

作者 / 機構: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出處連結 ↗

對應章節 / 時間戳: Encoder and Decoder Stacks (class SublayerConnection: x + dropout(sublayer(norm(x))))

驗證版本與日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入選理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

來源前置條件:

PyTorch tensor operations and attention theory.

安全回退方案（離線彈性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

對比候選資源與選型分析 (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

linking arXiv.org perpetual non-exclusive license / Open Access

On Layer Normalization in the Transformer Architecture

作者 / 機構: Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu | 原始出處連結 ↗

對應章節 / 時間戳: Abstract; Section 3.3 Understanding the Transformer at initialization (Theorem 1, gradients of the last layer); Section 4 Experiments (Pre-LN without warm-up)

驗證版本與日期: arXiv:2002.04745v2 (Verified: 2026-11-01)

入選理由:

Supports the activity's stability claim: Post-LN has large expected gradients near the output at initialization and needs warm-up; Pre-LN trained without warm-up reached comparable results in the paper's experiments.

來源前置條件:

LayerNorm and residual connections from Days 46-01 and 46-02; gradients from Days 31-36.

安全回退方案（離線彈性）:

The lesson states the Pre-LN and Post-LN formulas and the warm-up finding in its own words.

對比候選資源與選型分析 (2)
Liu et al.: Understanding the Difficulty of Training Transformers (EMNLP 2020) ↗ compared_deferred [arXiv Open Access]

Argues that Post-LN instability comes from amplified dependence on the residual branch and proposes Admin initialization; deeper than this activity needs, so deferred.

Wang et al.: Learning Deep Transformer Models for Machine Translation (ACL 2019) ↗ compared_alternative [arXiv Open Access]

Reports that pre-norm Transformers train deeper stacks than post-norm ones, but is empirical and specific to translation; Xiong et al. give the gradient analysis this activity asks learners to explain.

linking Open Access / OpenAI Public Publication

Language Models are Unsupervised Multitask Learners

作者 / 機構: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever | 原始出處連結 ↗

對應章節 / 時間戳: Section 2.3 Model (layer normalization moved to the input of each sub-block)

驗證版本與日期: OpenAI Technical Report 2019 (Verified: 2026-10-30)

入選理由:

Example of a widely used decoder-only LM that places layer normalization before each sub-block.

來源前置條件:

Autoregressive conditional likelihood, cross-entropy loss, perplexity exponentiation.

安全回退方案（離線彈性）:

Conditional log-likelihood equations, length normalization factors, and held-out cross-entropy calculations are fully derived in lesson material.

對比候選資源與選型分析 (2)
Kaplan et al. Scaling Laws for Neural Language Models ↗ compared_alternative [Open Access]

Focuses on empirical power-law loss curves; Radford et al. establishes the zero-shot log-likelihood task evaluation protocol.

Hoffmann et al. Training Compute-Optimal Large Language Models (Chinchilla) ↗ compared_alternative [Open Access]

Optimal compute scaling; Radford provides the fundamental zero-shot conditional probability formulation.

linking Open Access

Llama 2: Open Foundation and Fine-Tuned Chat Models

作者 / 機構: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. | 原始出處連結 ↗

對應章節 / 時間戳: Section 2.2 Training Details (pre-normalization using RMSNorm)

驗證版本與日期: arXiv:2307.09288v2 (Verified: 2026-10-30)

入選理由:

Example of a widely used decoder-only LLM that normalizes before each sub-layer, with RMSNorm in place of LayerNorm.

來源前置條件:

Transformer architecture, RLHF/SFT pipelines.

安全回退方案（離線彈性）:

The LLaMA-2 prompt structure, token boundary rules, and comparative analysis are fully worked out in curriculum lessons.

對比候選資源與選型分析 (2)
Touvron et al. LLaMA: Open and Efficient Foundation Language Models ↗ compared_alternative [Open Access]

Original foundation model paper without conversational chat tuning details; LLaMA-2 formalizes dialogue formatting.

Jiang et al. Mistral 7B ↗ compared_alternative [Open Access]

Adopts similar [INST] formatting; LLaMA-2 provides the foundational ablation on system prompt integration.

04 120 分鐘

獨立 Transformer 模組組裝考核

完全獨立閉卷考核

前置要求

完成活動 01 至 03。

學習目標

獨立實現規範的 Pre-LN Transformer 注意力模組，驗證輸入輸出張量形狀嚴格保持。

操作指引

編寫 Block 類，組合 Norm、自注意力層與殘差加法。斷言形狀為 (2, 8, 64) 的輸入張量在通過模組後輸出形狀依然精確為 (2, 8, 64)。

自主實作與記錄

1. 給出注意力子層的 forward 方法代碼。
2. 驗證輸出張量與預期手算結果一致。
3. 若注意力輸出投影維度錯寫為 d != d_model，會引發什麼運行時錯誤？

獨立掌握標準

模組化實現通過所有形狀不變性與斷言測試。

權威來源資源與權利證明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

驗證版本與日期: Git master (Verified: 2026-10-30)

入選理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

來源前置條件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（離線彈性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

對比候選資源與選型分析 (2)
GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking MIT

The Annotated Transformer

作者 / 機構: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出處連結 ↗

對應章節 / 時間戳: Encoder and Decoder Stacks (class EncoderLayer, class DecoderLayer)

驗證版本與日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入選理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

來源前置條件:

PyTorch tensor operations and attention theory.

安全回退方案（離線彈性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

對比候選資源與選型分析 (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.