自迴歸約束與下三角因果遮罩構建
助教輔助工程檢查
前置要求
第 43 天的注意力機制與縮放除法推導。
學習目標
闡明解碼器語言模型的自迴歸因式分解約束，構建下三角因果遮罩矩陣 M，並推導遮罩後的分數矩陣 S_masked = S + M。
操作指引
必做任務：解釋為什麼因果解碼器必須滿足 P(x_1, ..., x_T) = prod_{t=1}^T P(x_t | x_{
自主實作與記錄
1. 寫出 3x3 因果遮罩矩陣 M 的完整元素。
2. 寫出加性遮罩後的分數矩陣 S_masked = S + M。
3. 如果代碼中錯誤地將主對角線上方程設為 0、下方設為 -inf，會產生什麼後果？
M 的主對角線及其下方（下三角）全為 0，主對角線上方程全為 -inf。對於第 0 行，位置 1 和 2 均為 -inf。
遮罩矩陣 M：
[[0.0, -inf, -inf],
[0.0, 0.0, -inf],
[0.0, 0.0, 0.0]]。
遮罩後分數矩陣 S_masked：
[[2.0, -inf, -inf],
[0.0, 3.0, -inf],
[1.0, 1.5, 2.0]]。
若上下顛倒，模型將僅能關注未來而無法看到歷史上下文，導致反向因果洩漏，在自迴歸生成時徹底崩潰。
準確寫出下三角因果遮罩 M 並完成 3x3 遮罩分數矩陣 S_masked 的推導。
權威來源資源與權利證明
Attention Is All You Need
對應章節 / 時間戳: Section 3.2.3: Masked Multi-Head Attention in the decoder preventing leftward information flow
驗證版本與日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
對比候選資源與選型分析 (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
The Annotated Transformer
對應章節 / 時間戳: subsequent_mask implementation using torch.triu and triangular boolean masking
驗證版本與日期: Updated 2022 edition (Verified: 2026-10-30)
Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.
Intermediate Python, PyTorch broadcasting and tensor slicing semantics
The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.
對比候選資源與選型分析 (3)
Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.
Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.
The original primary paper; Annotated Transformer annotates it directly.