查询、键与值：向量角色与注意力公式形式化
助教辅助工程检查
前置要求
第 41-42 天的前向传播、矩阵乘法与逐行 Softmax 机制。
学习目标
将注意力机制分解为用查询检索键以聚合值的路由过程，推导张量维度，并完成原始点积分数矩阵的数值计算。
操作指引
学习来源：在本活动现有的阅读与练习时段内，参考 Stanford CS224N: Natural Language Processing with Deep Learning 的 Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides，用自己的话解释一处与本地任务的联系。保留现有练习与验收检查；若外部资料无法访问，使用课程自写的解释。必做任务：形式化缩放点积注意力公式 Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V。给定序列长度 T=3、键维度 d_k=4、值维度 d_v=2。设定 Q 各行为 q_0=[2.0, 0.0, 2.0, 0.0], q_1=[0.0, 2.0, 0.0, 2.0], q_2=[1.0, 1.0, 1.0, 1.0]；K 各行为 k_0=[1.0, 0.0, 1.0, 0.0], k_1=[0.0, 2.0, 0.0, 1.0], k_2=[2.0, 1.0, 0.0, 1.0]；V 各行为 v_0=[1.0, 0.0], v_1=[0.0, 2.0], v_2=[1.0, 1.0]。逐步手算原始分数矩阵 S_raw = Q K^T。阐述查询（当前位置在找什么）、键（当前位置提供什么信息）与值（传输的具体表征内容）的几何与物理意义。
自主演练与记录
1. 计算 S_raw = Q K^T 的 3x3 矩阵全部数值。
2. 明确给出 Q、K、V、Q K^T 以及最终输出矩阵的张量形状。
3. 用两句话说明为什么注意力是一种数据驱动的关联检索，而非静态固定权重层？
S_raw 的第 i 行第 j 列是 q_i 与 k_j 的内积。例如 S_raw[0,0] = 2*1 + 0*0 + 2*1 + 0*0 = 4。
S_raw 矩阵：第 0 行 [4.0, 0.0, 4.0]；第 1 行 [0.0, 6.0, 4.0]；第 2 行 [2.0, 3.0, 4.0]。张量形状：Q 为 (3,4)，K 为 (3,4)，V 为 (3,2)，Q K^T 为 (3,3)，输出为 (3,2)。传统线性层权重是训练后冻结的固定参数，而注意力权重是由当前动态上下文实时计算出的自适应路由系数。
准确列出矩阵乘法各步骤，得到完全正确的 3x3 S_raw 矩阵及各张量形状。
权威源资源与权利证明
Stanford CS224N: Natural Language Processing with Deep Learning
对应章节 / 时间戳: Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides
验证版本与日期: Public edition checked 2026-11-01; moving pages may change
Selected attention/Transformer lecture slides provide a coherent teaching check on Q/K/V and architecture; original papers remain definition references and local code exercises remain required.
Linear algebra, gradients, basic neural language models and PyTorch.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
对比候选资源与选型分析 (2)
Runnable code commentary remains the implementation spine; its sublayer convention must not be mistaken for the original paper.
Concise diagrams are useful, but provide neither the wider lecture sequence nor checked implementation exercises.
Attention Is All You Need
对应章节 / 时间戳: Section 3.2.1: Scaled Dot-Product Attention equation and query/key/value matrix definitions
验证版本与日期: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
对比候选资源与选型分析 (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs
对应章节 / 时间戳: model.py: CausalSelfAttention forward pass query/key/value projection extraction
验证版本与日期: GitHub master commit (Verified: 2026-10-30)
Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.
Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics
Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.
对比候选资源与选型分析 (3)
Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.
Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.
Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.