Autoregressive Constraint and Lower-Triangular Causal Masking
Agent-Assisted Engineering Check
Prerequisites
Day 43 attention mechanisms and scaling derivations.
Learning Objective
Define the autoregressive constraint for decoder language models, construct the lower-triangular causal mask matrix M, and derive masked scores S_masked = S + M.
Instructions
Required work: explain why a causal decoder must satisfy P(x_1, ..., x_T) = prod_{t=1}^T P(x_t | x_{
Practice & Prediction Log
1. Write out the 3x3 causal mask matrix M.
2. Write out the masked score matrix S_masked = S + M.
3. What happens if an implementation accidentally puts 0 above the diagonal and -inf below?
M has zeros on and below the main diagonal (lower triangle) and -inf above the main diagonal. For row 0, entries 1 and 2 are -inf.
M matrix:
[[0.0, -inf, -inf],
[0.0, 0.0, -inf],
[0.0, 0.0, 0.0]].
Masked scores S_masked:
[[2.0, -inf, -inf],
[0.0, 3.0, -inf],
[1.0, 1.5, 2.0]].
If inverted (0 above and -inf below), the model attends exclusively to future tokens and is blind to past context, creating an anti-causal leak that fails autoregressive generation.
Exact formulation of lower-triangular mask M and derivation of the 3x3 S_masked matrix.
Authoritative Source Resources & Provenance
Attention Is All You Need
Relevant Section / Range: Section 3.2.3: Masked Multi-Head Attention in the decoder preventing leftward information flow
Verified Version & Date: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
Compared Candidate Sources & Selection Analysis (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
The Annotated Transformer
Relevant Section / Range: subsequent_mask implementation using torch.triu and triangular boolean masking
Verified Version & Date: Updated 2022 edition (Verified: 2026-10-30)
Widely used pedagogical commentary that pairs every paragraph of the original Vaswani paper with an executable, testable code cell.
Intermediate Python, PyTorch broadcasting and tensor slicing semantics
The curriculum provides standalone hand-worked numerical tables, explicit causal mask verification scripts, and offline analytical proofs.
Compared Candidate Sources & Selection Analysis (3)
Spectacular visual diagrams and intuitive color-coding, but lacks executable tensor code and formal numerical shape validation.
Rigorous academic slides and video lectures, but access to assignment grading is restricted to Stanford enrolled students.
The original primary paper; Annotated Transformer annotates it directly.