查詢、鍵與值：向量角色與注意力公式形式化
助教輔助工程檢查
前置要求
第 41-42 天的前向傳播、矩陣乘法與逐行 Softmax 機制。
學習目標
將注意力機制分解為用查詢檢索鍵以聚合值的路由過程，推導張量維度，並完成原始點積分數矩陣的數值計算。
操作指引
學習來源：在本活動現有的閱讀與練習時段內，參考 Stanford CS224N: Natural Language Processing with Deep Learning 的 Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides，用自己的話解釋一處與本地任務的聯繫。保留現有練習與驗收檢查；若外部資料無法存取，使用課程自寫的解釋。必做任務：形式化縮放點積注意力公式 Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V。給定序列長度 T=3、鍵維度 d_k=4、值維度 d_v=2。設定 Q 各行為 q_0=[2.0, 0.0, 2.0, 0.0], q_1=[0.0, 2.0, 0.0, 2.0], q_2=[1.0, 1.0, 1.0, 1.0]；K 各行為 k_0=[1.0, 0.0, 1.0, 0.0], k_1=[0.0, 2.0, 0.0, 1.0], k_2=[2.0, 1.0, 0.0, 1.0]；V 各行為 v_0=[1.0, 0.0], v_1=[0.0, 2.0], v_2=[1.0, 1.0]。逐步手算原始分數矩陣 S_raw = Q K^T。闡述查詢（當前位置在找什麼）、鍵（當前位置提供什麼資訊）與值（傳輸的具體表徵內容）的幾何與物理意義。
自主實作與記錄
1. 計算 S_raw = Q K^T 的 3x3 矩陣全部數值。
2. 明確給出 Q、K、V、Q K^T 以及最終輸出矩陣的張量形狀。
3. 用兩句話說明為什麼注意力是一種數據驅動的關聯檢索，而非靜態固定權重層？
S_raw 的第 i 行第 j 列是 q_i 與 k_j 的內積。例如 S_raw[0,0] = 2*1 + 0*0 + 2*1 + 0*0 = 4。
S_raw 矩陣：第 0 行 [4.0, 0.0, 4.0]；第 1 行 [0.0, 6.0, 4.0]；第 2 行 [2.0, 3.0, 4.0]。張量形狀：Q 為 (3,4)，K 為 (3,4)，V 為 (3,2)，Q K^T 為 (3,3)，輸出為 (3,2)。傳統線性層權重是訓練後凍結的固定參數，而注意力權重是由當前動態上下文即時計算出的自適應路由係數。
準確列出矩陣乘法各步驟，得到完全正確的 3x3 S_raw 矩陣及各張量形狀。
權威來源資源與權利證明
Stanford CS224N: Natural Language Processing with Deep Learning
對應章節 / 時間戳: Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides
驗證版本與日期: Public edition checked 2026-11-01; moving pages may change
Selected attention/Transformer lecture slides provide a coherent teaching check on Q/K/V and architecture; original papers remain definition references and local code exercises remain required.
Linear algebra, gradients, basic neural language models and PyTorch.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
對比候選資源與選型分析 (2)
Runnable code commentary remains the implementation spine; its sublayer convention must not be mistaken for the original paper.
Concise diagrams are useful, but provide neither the wider lecture sequence nor checked implementation exercises.
Attention Is All You Need
對應章節 / 時間戳: Section 3.2.1: Scaled Dot-Product Attention equation and query/key/value matrix definitions
驗證版本與日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
對比候選資源與選型分析 (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs
對應章節 / 時間戳: model.py: CausalSelfAttention forward pass query/key/value projection extraction
驗證版本與日期: GitHub master commit (Verified: 2026-10-30)
Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.
Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics
Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.
對比候選資源與選型分析 (3)
Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.
Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.
Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.