46 / 90

第 46 日：Transformer 模块组装：LayerNorm、RMSNorm 与残差连接

使用小数值矩阵精确推导 LayerNorm 与 RMSNorm 归一化，解释为何残差相加能创造梯度高速公路，并深入对比 Pre-LN 与 Post-LN 的收敛稳定性。

模型、预训练与微调 (LoRA) 480 分钟专注学习（8 小时）

01 120 分钟

Layer Normalization 与 RMSNorm：数学原理与数值稳定性

Agent 辅助工程分析

前置要求

第 28 日方差与标准差；第 22 日向量代数基础。

学习目标

针对输入向量 x = [2.0, -1.0, 0.0, 3.0] 逐步计算 LayerNorm 与 RMSNorm，阐释现代架构偏好 RMSNorm 的根本原因。

操作指引

对于 d=4 的向量 x = [2.0, -1.0, 0.0, 3.0]：(1) 计算均值 mu = 1.0、方差 sigma^2 = 2.5，以及 eps=1e-5 下的 LayerNorm 输出。(2) 计算均方根 RMS(x) = sqrt(mean(x^2)) = sqrt(3.5) ≈ 1.8708，及 RMSNorm 输出 x / RMS(x)。(3) 比较计算成本：RMSNorm 省去了在显存上对均值的遍历归约。

自主演练与记录

1. 在 gamma=1, beta=0 下计算 LayerNorm(x)，保留 4 位小数。
2. 在 gamma=1 下计算 RMSNorm(x)，保留 4 位小数。
3. 为何 RMSNorm 在 GPU 核函数执行时具有更显著的显存带宽优势？

标记为已完成

独立掌握标准

学习者能够手算完成两类归一化，并能准确论述 RMSNorm 的访存带宽优势。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

Root Mean Square Layer Normalization

作者 / 机构: Biao Zhang, Rico Sennrich (University of Edinburgh) | 原始出处链接 ↗

对应章节 / 时间戳: Section 4 RMSNorm (Equation 4) and Section 4.1 Invariance Analysis

验证版本与日期: NeurIPS 2019 / arXiv:1910.07467v1 (Verified: 2026-10-30)

入选理由:

Primary paper showing that mean-centering in LayerNorm can be discarded without performance loss, speeding up modern LLMs (LLaMA, Gemma).

来源前置条件:

Variance, standard deviation, and LayerNorm basics.

安全回退方案（离线弹性）:

RMSNorm formulas and numerical comparisons are derived completely in the lesson.

对比候选资源与选型分析 (2)

Ba et al. Layer Normalization (2016) ↗ compared_selected_for_day46 [Open Access]

Original LayerNorm formulation with mean-centering; RMSNorm is compared directly against it as an optimization.

Ioffe & Szegedy Batch Normalization (2015) ↗ compared_alternative [Open Access]

Batch-dependent statistics; unsuitable for autoregressive sequence modeling.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: Encoder and Decoder Stacks (class LayerNorm, class SublayerConnection)

验证版本与日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入选理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

来源前置条件:

PyTorch tensor operations and attention theory.

安全回退方案（离线弹性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

对比候选资源与选型分析 (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

02 120 分钟

残差连接与恒等梯度高速公路

Agent 辅助工程分析

前置要求

活动 01 及第 35 日反向传播链式法则。

学习目标

推导残差加法 y = x + F(x) 产生无衰减恒等梯度项 dL/dx = dL/dy + dL/dy * dF/dx 的数学机制。

操作指引

写出 y = x + F(x) 关于 x 的向量导数。证明即使 dF/dx 极小或趋近饱和为 0，dL/dx 依然保留恒等项 dL/dy，从而彻底消除百层深层 Transformer 中的梯度消失风险。

自主演练与记录

1. 写出 y = x + F(x) 下 dL/dx 的链式法则展开式。
2. 若 dF/dx 的特征值模长均 < 0.1，为何反向梯度不会衰减殆尽？
3. 说明 x + F(x) 加法能够执行所要求的张量维度一致性条件。

标记为已完成

独立掌握标准

推导出梯度高速公路公式并指明严格的张量形状不变性。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.1 Encoder and Decoder Stacks (residual connection around each sub-layer, then layer normalization)

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

03 120 分钟

Pre-LN 与 Post-LN 架构对比分析

Agent 辅助工程分析

前置要求

活动 01 与 02。

学习目标

对比 Pre-LN (x = x + Attn(LN(x))) 与原版 Post-LN (x = LN(x + Attn(x)))。解释 Xiong et al.（2020）关于初始化和学习率预热的结论，并将 GPT-2、Llama 2 作为有文献依据的预归一化实例（Llama 2 使用 RMSNorm），而非所有模型都采用该结构的证据。

操作指引

追踪两种配置下的激活与梯度流，并说明两者训练稳定性的差异。Xiong et al.（2020）证明：在初始化时，Post-LN Transformer 靠近输出层的参数梯度期望值很大，因此需要学习率预热（warm-up）阶段才能稳定训练；Pre-LN 在初始化时梯度表现良好，在该论文的实验中，不使用预热训练的 Pre-LN Transformer 也取得了相当的结果。GPT-2（Section 2.3）把层归一化移到每个子块的输入端，Llama 2（Section 2.2）采用 RMSNorm 预归一化。这些只是例子，并非全面调查：不要声称所有现代大模型都使用 Pre-LN。

自主演练与记录

1. 给出 Pre-LN 与 Post-LN 模块的数据流公式。
2. 在 Post-LN 中，为何残差路径无法作为纯净的恒等高速通道？
3. 根据 Xiong et al.（2020），为什么 Pre-LN 可以不用 Post-LN 所需的学习率预热阶段进行训练？

标记为已完成

独立掌握标准

清晰阐明两者结构差异，准确将训练稳定性归因于未被归一化破坏的主干残差。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.1 Encoder and Decoder Stacks (LayerNorm(x + Sublayer(x)), the post-LN placement)

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: Encoder and Decoder Stacks (class SublayerConnection: x + dropout(sublayer(norm(x))))

验证版本与日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入选理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

来源前置条件:

PyTorch tensor operations and attention theory.

安全回退方案（离线弹性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

对比候选资源与选型分析 (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

linking arXiv.org perpetual non-exclusive license / Open Access

On Layer Normalization in the Transformer Architecture

作者 / 机构: Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu | 原始出处链接 ↗

对应章节 / 时间戳: Abstract; Section 3.3 Understanding the Transformer at initialization (Theorem 1, gradients of the last layer); Section 4 Experiments (Pre-LN without warm-up)

验证版本与日期: arXiv:2002.04745v2 (Verified: 2026-11-01)

入选理由:

Supports the activity's stability claim: Post-LN has large expected gradients near the output at initialization and needs warm-up; Pre-LN trained without warm-up reached comparable results in the paper's experiments.

来源前置条件:

LayerNorm and residual connections from Days 46-01 and 46-02; gradients from Days 31-36.

安全回退方案（离线弹性）:

The lesson states the Pre-LN and Post-LN formulas and the warm-up finding in its own words.

对比候选资源与选型分析 (2)

Liu et al.: Understanding the Difficulty of Training Transformers (EMNLP 2020) ↗ compared_deferred [arXiv Open Access]

Argues that Post-LN instability comes from amplified dependence on the residual branch and proposes Admin initialization; deeper than this activity needs, so deferred.

Wang et al.: Learning Deep Transformer Models for Machine Translation (ACL 2019) ↗ compared_alternative [arXiv Open Access]

Reports that pre-norm Transformers train deeper stacks than post-norm ones, but is empirical and specific to translation; Xiong et al. give the gradient analysis this activity asks learners to explain.

linking Open Access / OpenAI Public Publication

Language Models are Unsupervised Multitask Learners

作者 / 机构: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever | 原始出处链接 ↗

对应章节 / 时间戳: Section 2.3 Model (layer normalization moved to the input of each sub-block)

验证版本与日期: OpenAI Technical Report 2019 (Verified: 2026-10-30)

入选理由:

Example of a widely used decoder-only LM that places layer normalization before each sub-block.

来源前置条件:

Autoregressive conditional likelihood, cross-entropy loss, perplexity exponentiation.

安全回退方案（离线弹性）:

Conditional log-likelihood equations, length normalization factors, and held-out cross-entropy calculations are fully derived in lesson material.

对比候选资源与选型分析 (2)

Kaplan et al. Scaling Laws for Neural Language Models ↗ compared_alternative [Open Access]

Focuses on empirical power-law loss curves; Radford et al. establishes the zero-shot log-likelihood task evaluation protocol.

Hoffmann et al. Training Compute-Optimal Large Language Models (Chinchilla) ↗ compared_alternative [Open Access]

Optimal compute scaling; Radford provides the fundamental zero-shot conditional probability formulation.

linking Open Access

Llama 2: Open Foundation and Fine-Tuned Chat Models

作者 / 机构: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. | 原始出处链接 ↗

对应章节 / 时间戳: Section 2.2 Training Details (pre-normalization using RMSNorm)

验证版本与日期: arXiv:2307.09288v2 (Verified: 2026-10-30)

入选理由:

Example of a widely used decoder-only LLM that normalizes before each sub-layer, with RMSNorm in place of LayerNorm.

来源前置条件:

Transformer architecture, RLHF/SFT pipelines.

安全回退方案（离线弹性）:

The LLaMA-2 prompt structure, token boundary rules, and comparative analysis are fully worked out in curriculum lessons.

对比候选资源与选型分析 (2)

Touvron et al. LLaMA: Open and Efficient Foundation Language Models ↗ compared_alternative [Open Access]

Original foundation model paper without conversational chat tuning details; LLaMA-2 formalizes dialogue formatting.

Jiang et al. Mistral 7B ↗ compared_alternative [Open Access]

Adopts similar [INST] formatting; LLaMA-2 provides the foundational ablation on system prompt integration.

04 120 分钟

独立 Transformer 模块组装考核

完全独立闭卷考核

前置要求

完成活动 01 至 03。

学习目标

独立实现规范的 Pre-LN Transformer 注意力模块，验证输入输出张量形状严格保持。

操作指引

编写 Block 类，组合 Norm、自注意力层与残差加法。断言形状为 (2, 8, 64) 的输入张量在通过模块后输出形状依然精确为 (2, 8, 64)。

自主演练与记录

1. 给出注意力子层的 forward 方法代码。
2. 验证输出张量与预期手算结果一致。
3. 若注意力输出投影维度错写为 d != d_model，会引发什么运行时错误？

标记为已完成

独立掌握标准

模块化实现通过所有形状不变性与断言测试。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: Encoder and Decoder Stacks (class EncoderLayer, class DecoderLayer)

验证版本与日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入选理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

来源前置条件:

PyTorch tensor operations and attention theory.

安全回退方案（离线弹性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

对比候选资源与选型分析 (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.