Query, Key, and Value: Vector Roles and Attention Formulation
Agent-Assisted Engineering Check
Prerequisites
Day 41-42 forward pass, matrix multiplication, and row-wise softmax mechanics.
Learning Objective
Deconstruct attention into query routing against keys to retrieve values, derive matrix dimensions, and compute raw dot-product score matrices.
Instructions
Study source: within this activity's existing reading/practice time, use Stanford CS224N: Natural Language Processing with Deep Learning, Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides, then explain one connection to the local task in your own words. Keep the local exercise and acceptance checks; use its original explanation if the external source is unavailable. Required work: formulate scaled dot-product attention Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V. Given sequence length T=3, key dimension d_k=4, and value dimension d_v=2, define Q with rows q_0=[2.0, 0.0, 2.0, 0.0], q_1=[0.0, 2.0, 0.0, 2.0], q_2=[1.0, 1.0, 1.0, 1.0]; K with rows k_0=[1.0, 0.0, 1.0, 0.0], k_1=[0.0, 2.0, 0.0, 1.0], k_2=[2.0, 1.0, 0.0, 1.0]; and V with rows v_0=[1.0, 0.0], v_1=[0.0, 2.0], v_2=[1.0, 1.0]. Compute the raw dot-product matrix S_raw = Q K^T step by step. Explain the conceptual role of Query (what position seeks), Key (what position offers), and Value (content transmitted).
Practice & Prediction Log
1. Compute the 3x3 entries of S_raw = Q K^T for the matrices above.
2. State the tensor shapes of Q, K, V, Q K^T, and the final output.
3. Why is attention an associative retrieval mechanism rather than a fixed linear layer?
Row i of S_raw is computed by dotting q_i with each key column k_j (equivalent to row j of K). For example, S_raw[0,0] = 2*1 + 0*0 + 2*1 + 0*0 = 4.
S_raw row 0: [4.0, 0.0, 4.0]; row 1: [0.0, 6.0, 4.0]; row 2: [2.0, 3.0, 4.0]. Shapes: Q is (3,4), K is (3,4), V is (3,2), Q K^T is (3,3), and output is (3,2). Unlike fixed weights, attention weights are dynamic data-dependent routing coefficients computed from the current context.
Accurate step-by-step matrix multiplication yielding the exact 3x3 S_raw matrix and correct shape annotations.
Authoritative Source Resources & Provenance
Stanford CS224N: Natural Language Processing with Deep Learning
Relevant Section / Range: Winter 2026 Lecture 05 Transformers (2026-01-20); attention and self-attention slides
Verified Version & Date: Public edition checked 2026-11-01; moving pages may change
Selected attention/Transformer lecture slides provide a coherent teaching check on Q/K/V and architecture; original papers remain definition references and local code exercises remain required.
Linear algebra, gradients, basic neural language models and PyTorch.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
Compared Candidate Sources & Selection Analysis (2)
Runnable code commentary remains the implementation spine; its sublayer convention must not be mistaken for the original paper.
Concise diagrams are useful, but provide neither the wider lecture sequence nor checked implementation exercises.
Attention Is All You Need
Relevant Section / Range: Section 3.2.1: Scaled Dot-Product Attention equation and query/key/value matrix definitions
Verified Version & Date: NeurIPS 2017 / arXiv:1706.03762v7 (Verified: 2026-10-30)
The foundational primary literature establishing scaled dot-product attention, multi-head projection mechanisms, and autoregressive causal masking.
Linear algebra, matrix multiplication, softmax probability distributions
Complete original algebraic derivations, matrix examples, variance proofs, and tensor diagrams are self-contained in the curriculum text without external document dependency.
Compared Candidate Sources & Selection Analysis (3)
Pioneered additive attention in RNNs, but Vaswani scaled dot-product attention replaced additive alignment in modern LLM architectures due to parallel matrix multiplication speed.
Introduces relative positional biases into attention; deferred to subsequent positional embedding modules after absolute scaled attention is thoroughly mastered.
Selected jointly as the primary code reference: translates Section 3.2 of the Vaswani paper into minimal, idiomatic PyTorch tensor operations.
nanoGPT: Simplest, Fastest Repository for Training and Finetuning Medium GPTs
Relevant Section / Range: model.py: CausalSelfAttention forward pass query/key/value projection extraction
Verified Version & Date: GitHub master commit (Verified: 2026-10-30)
Detailed modern reference implementation of clean, idiomatic batched causal self-attention in fewer than 60 lines of code.
Python, NumPy/PyTorch tensor fundamentals, and feed-forward network basics
Self-contained step-by-step tensor derivations, numeric matrix traces, and pure Python/NumPy implementations are directly provided in curriculum exercises.
Compared Candidate Sources & Selection Analysis (3)
Includes extensive production abstraction layers, configuration polymorphism, and cache tracking that obscure direct mathematical tensor operations for beginner learners.
Selected jointly: provides line-by-line literary commentary mapping equations directly to PyTorch tensors.
Expanded video walkthrough; software is MIT-licensed while video audio/subtitles remain linking-only without transcript import.