自回归约束与下三角因果掩码构建
助教辅助工程检查
前置要求
第 43 天的注意力机制与缩放除法推导。
学习目标
阐明解码器语言模型的自回归因式分解约束，构建下三角因果掩码矩阵 M，并推导掩码后的分数矩阵 S_masked = S + M。
操作指引
必做任务：解释为什么因果解码器必须满足 P(x_1, ..., x_T) = prod_{t=1}^T P(x_t | x_{
自主演练与记录
1. 写出 3x3 因果掩码矩阵 M 的完整元素。
2. 写出加性掩码后的分数矩阵 S_masked = S + M。
3. 如果代码中错误地将主对角线上方设为 0、下方设为 -inf，会产生什么后果？
M 的主对角线及其下方（下三角）全为 0，主对角线上方全为 -inf。对于第 0 行，位置 1 和 2 均为 -inf。
掩码矩阵 M：
[[0.0, -inf, -inf],
[0.0, 0.0, -inf],
[0.0, 0.0, 0.0]]。
掩码后分数矩阵 S_masked：
[[2.0, -inf, -inf],
[0.0, 3.0, -inf],
[1.0, 1.5, 2.0]]。
若上下颠倒，模型将仅能关注未来而无法看到历史上下文，导致反向因果泄露，在自回归生成时彻底崩溃。
准确写出下三角因果掩码 M 并完成 3x3 掩码分数矩阵 S_masked 的推导。
权威源资源与权利证明
Attention Is All You Need
对应章节 / 时间戳: Section 3.2.3: Masked Multi-Head Attention in the decoder preventing leftward information flow
验证版本与日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
对比候选资源与选型分析 (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
The Annotated Transformer
对应章节 / 时间戳: subsequent_mask implementation using torch.triu and triangular boolean masking
验证版本与日期: Updated 2022 edition (Verified: 2026-10-30)
Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.
Intermediate Python, PyTorch broadcasting and tensor slicing semantics
The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.
对比候选资源与选型分析 (3)
Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.
Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.
The original primary paper; Annotated Transformer annotates it directly.