第 48 日：完整 Decoder-Only Transformer：位置编码、权重绑定与 Logits 投影

01 120 分钟

位置信息表示：可学习编码、正弦编码与旋转位置编码 (RoPE)

Agent 辅助工程分析

前置要求

第 43 日自注意力与第 47 日 Transformer 模块堆叠。

学习目标

解释为何在无位置信息时自注意力具有排列等变性，对比可学习绝对编码、正弦绝对编码与旋转位置编码 (RoPE)。

操作指引

证明调换词元 1 与词元 2 将导致无位置编码的自注意力输出发生等变交换。定义加性位置编码 x = tok_emb + pos_emb，并对比 RoPE 在复数平面上的乘性旋转机制。

自主演练与记录

1. 证明朴素点积注意力具有排列等变性。
2. GPT-2 如何将位置向量注入到词元嵌入中？
3. RoPE 相比可学习绝对位置编码具有什么关键的外推优势？

标记为已完成

独立掌握标准

清晰阐明排列等变性，准确对比加性编码与 RoPE 的几何本质。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.5 Positional Encoding

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: Positional Encoding (class PositionalEncoding)

验证版本与日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入选理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

来源前置条件:

PyTorch tensor operations and attention theory.

安全回退方案（离线弹性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

对比候选资源与选型分析 (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

02 120 分钟

端到端 Decoder-Only Transformer 前向全流程

Agent 辅助工程分析

前置要求

活动 01 及第 46-47 日所有模块。

学习目标

追踪从整数词元索引张量 (B, T) 到下一词预测 Logits 张量 (B, T, V) 的完整张量生命周期。

操作指引

梳理五大核心阶段：(1) 输入词元与位置编码相加 -> (B, T, d)；(2) 连续通过 N 个 Pre-LN Transformer Block -> (B, T, d)；(3) 最终 LayerNorm 归一化 -> (B, T, d)；(4) LM Head 投影 -> (B, T, V)；(5) 提取最后一个时间步切片计算损失。

自主演练与记录

1. 给出 PyTorch 中 forward 方法的张量变迁签名与核心代码。
2. 对于 B=1, T=3, d=4, V=6，输出 logits 的精确张量形状是什么？
3. 如何从 logits 中提取时间步 t=3 的下一词预测分布？

标记为已完成

独立掌握标准

完整准确推导出从整数输入到最终 Logits 的全套张量变迁流程。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3 Model Architecture (3.1-3.5)

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

03 120 分钟

权重绑定 (Weight Tying) 与输出 Logit 投影头

Agent 辅助工程分析

前置要求

活动 02 与第 40 日嵌入表原理。

学习目标

解释权重绑定假设：让词元输入嵌入表与输出语言模型投影头共享同一套权重矩阵。

操作指引

分析 self.lm_head.weight = self.transformer.wte.weight。计算参数节省量（如 GPT-2 节省 50,257 * 768 ≈ 38.6M 参数）。论述词元作为输入特征与其作为输出类别的语义对称性。

自主演练与记录

1. 权重绑定在数学上对词元输入和输出空间作出了什么对称性假设？
2. 在 V=32,000, d=4096 的模型中，权重绑定能直接省下多少参数？
3. 为什么权重绑定有助于防止低频生僻词的表征漂移？

标记为已完成

独立掌握标准

能准确计算参数节省规模并深刻解释语义对称性与梯度共享机理。

权威源资源与权利证明

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class GPT __init__ weight tying (transformer.wte.weight = lm_head.weight)

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking MIT

The Annotated Transformer

作者 / 机构: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | 原始出处链接 ↗

对应章节 / 时间戳: Embeddings and Softmax (class Generator)

验证版本与日期: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

入选理由:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

来源前置条件:

PyTorch tensor operations and attention theory.

安全回退方案（离线弹性）:

All block assembly and normalization logic are provided in runnable local Python snippets.

对比候选资源与选型分析 (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

04 120 分钟

Transformer 架构综合掌握度终结考核

完全独立闭卷考核

前置要求

完成活动 01 至 03。

学习目标

完成闭卷综合考核：独立手算一个 2 词元微型序列通过 1 层玩具 Transformer 的端到端全流程。

操作指引

在 V=4, d_model=2, n_heads=1, N_layers=1 的极小配置下，手算输入序列 [0, 2] 的嵌入、因果自注意力、残差相加、归一化及最终 logits。

自主演练与记录

1. 用 5 个简洁的核心数学等式总结端到端前向推理过程。
2. 解释为何在因果解码下词元 0 绝不能关注到词元 2。
3. 写出当监督目标为 [2, 1] 时的交叉熵损失计算公式。

标记为已完成

独立掌握标准

独立完成全部端到端公式闭卷默写，准确掌握因果约束与损失计算。

权威源资源与权利证明

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

作者 / 机构: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | 原始出处链接 ↗

对应章节 / 时间戳: Section 3 Model Architecture (3.1-3.5)

验证版本与日期: arXiv:1706.03762v7 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

来源前置条件:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

安全回退方案（离线弹性）:

All mathematical formulas and structural definitions are self-contained in the lesson text.

对比候选资源与选型分析 (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

nanoGPT: model.py architecture implementation

作者 / 机构: Andrej Karpathy | 原始出处链接 ↗

对应章节 / 时间戳: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head

验证版本与日期: Git master (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

来源前置条件:

Python, PyTorch, and Day 43-45 attention fundamentals.

安全回退方案（离线弹性）:

Local models and classes in the curriculum mirror nanoGPT semantics directly.

对比候选资源与选型分析 (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.