Day 48: Complete Decoder-Only Transformer: Position, Weight Tying, and Logits

01 120 min

Positional Information: Absolute, Sinusoidal, and Rotary (RoPE)

Agent-Assisted Engineering Check

Prerequisites

Day 43 Attention and Day 47 Transformer Stacking.

Learning Objective

Explain why self-attention is permutation-equivariant without position encoding and compare learned vs sinusoidal vs Rotary Position Embeddings.

Instructions

Demonstrate that swapping token 1 and token 2 in self-attention produces correspondingly permuted outputs without position indicators. Define additive position embeddings x = tok_emb + pos_emb vs multiplicative RoPE rotation in complex space.

Practice & Prediction Log

1. Prove that vanilla dot-product attention has permutation equivariance.
2. In standard GPT-2, how are position vectors added to token vectors?
3. What key extrapolation advantage does RoPE provide over learned absolute embeddings?

Mark as Completed

Independent Mastery Criterion

Articulates permutation equivariance and contrasts additive position vs RoPE.

Authoritative Source Resources & Provenance

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

Author: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | Original Source Link ↗

Relevant Section / Range: Section 3.5 Positional Encoding

Verified Version & Date: arXiv:1706.03762v7 (Verified: 2026-10-30)

Selection Rationale:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

Source Prerequisites:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

Safe Fallback (Offline Resilience):

All mathematical formulas and structural definitions are self-contained in the lesson text.

Compared Candidate Sources & Selection Analysis (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

The Annotated Transformer

Author: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | Original Source Link ↗

Relevant Section / Range: Positional Encoding (class PositionalEncoding)

Verified Version & Date: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

Selection Rationale:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

Source Prerequisites:

PyTorch tensor operations and attention theory.

Safe Fallback (Offline Resilience):

All block assembly and normalization logic are provided in runnable local Python snippets.

Compared Candidate Sources & Selection Analysis (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

02 120 min

End-to-End Decoder-Only Transformer Forward Pass

Agent-Assisted Engineering Check

Prerequisites

Activity 48-01 and Days 46-47 blocks.

Learning Objective

Trace the complete tensor lifecycle from integer token indices (B, T) to next-token logits (B, T, V).

Instructions

Walk the 5 pipeline stages: (1) idx in (B, T) -> Token + Pos Embedding -> (B, T, d); (2) Stack of N Blocks -> (B, T, d); (3) Final LayerNorm -> (B, T, d); (4) Output Head projection -> (B, T, V); (5) Logits extraction for next-token loss.

Practice & Prediction Log

1. Write the PyTorch forward signature and line-by-line tensor transformations.
2. For B=1, T=3, d=4, V=6, state the exact shape of the output logits tensor.
3. How is the prediction for the next token at position t=3 obtained from the logits?

Mark as Completed

Independent Mastery Criterion

Complete and accurate tensor flow derivation from integer tokens to final logits.

Authoritative Source Resources & Provenance

linking MIT

nanoGPT: model.py architecture implementation

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head

Verified Version & Date: Git master (Verified: 2026-10-30)

Selection Rationale:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

Source Prerequisites:

Python, PyTorch, and Day 43-45 attention fundamentals.

Safe Fallback (Offline Resilience):

Local models and classes in the curriculum mirror nanoGPT semantics directly.

Compared Candidate Sources & Selection Analysis (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

Author: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | Original Source Link ↗

Relevant Section / Range: Section 3 Model Architecture (3.1-3.5)

Verified Version & Date: arXiv:1706.03762v7 (Verified: 2026-10-30)

Selection Rationale:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

Source Prerequisites:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

Safe Fallback (Offline Resilience):

All mathematical formulas and structural definitions are self-contained in the lesson text.

Compared Candidate Sources & Selection Analysis (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

03 120 min

Weight Tying and Output Logit Head Projection

Agent-Assisted Engineering Check

Prerequisites

Activity 48-02 and Day 40 embedding tables.

Learning Objective

Explain the weight tying hypothesis: sharing parameters between the token embedding table and the output language model head.

Instructions

Analyze self.lm_head.weight = self.transformer.wte.weight. Show parameter savings: saving V * d_model parameters (e.g. 50,257 * 768 = 38.6M parameters in GPT-2). Explain the semantic symmetry between token input embedding and logit scoring.

Practice & Prediction Log

1. What does weight tying mathematically assert about the input and output token spaces?
2. In a model with V=32,000 and d=4096, how many parameters does weight tying eliminate?
3. Why does tying weights help prevent rare tokens from having misaligned embeddings?

Mark as Completed

Independent Mastery Criterion

Explains semantic and parameter-saving benefits with exact numbers.

Authoritative Source Resources & Provenance

linking MIT

nanoGPT: model.py architecture implementation

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: model.py: class GPT __init__ weight tying (transformer.wte.weight = lm_head.weight)

Verified Version & Date: Git master (Verified: 2026-10-30)

Selection Rationale:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

Source Prerequisites:

Python, PyTorch, and Day 43-45 attention fundamentals.

Safe Fallback (Offline Resilience):

Local models and classes in the curriculum mirror nanoGPT semantics directly.

Compared Candidate Sources & Selection Analysis (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking MIT

The Annotated Transformer

Author: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | Original Source Link ↗

Relevant Section / Range: Embeddings and Softmax (class Generator)

Verified Version & Date: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

Selection Rationale:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

Source Prerequisites:

PyTorch tensor operations and attention theory.

Safe Fallback (Offline Resilience):

All block assembly and normalization logic are provided in runnable local Python snippets.

Compared Candidate Sources & Selection Analysis (2)

HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

04 120 min

Comprehensive Transformer Forward Pass Mastery Check

Independent Mastery Check

Prerequisites

Complete activities 48-01 through 48-03.

Learning Objective

Pass a closed-book comprehensive mastery exam: trace a 2-token sequence through a toy Transformer end-to-end.

Instructions

Given a toy vocabulary V=4, d_model=2, n_heads=1, N_layers=1: calculate embeddings, causal self-attention, residual addition, LayerNorm, and final logits for input sequence [0, 2].

Practice & Prediction Log

1. Provide the end-to-end forward algorithm in 5 clean equations.
2. Explain why token 0 cannot attend to token 2 in causal decoding.
3. Formulate the cross-entropy loss calculation given target token [2, 1].

Mark as Completed

Independent Mastery Criterion

Independent mastery of the entire Transformer decoder pipeline verified.

Authoritative Source Resources & Provenance

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

Author: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | Original Source Link ↗

Relevant Section / Range: Section 3 Model Architecture (3.1-3.5)

Verified Version & Date: arXiv:1706.03762v7 (Verified: 2026-10-30)

Selection Rationale:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

Source Prerequisites:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

Safe Fallback (Offline Resilience):

All mathematical formulas and structural definitions are self-contained in the lesson text.

Compared Candidate Sources & Selection Analysis (2)

Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

nanoGPT: model.py architecture implementation

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head

Verified Version & Date: Git master (Verified: 2026-10-30)

Selection Rationale:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

Source Prerequisites:

Python, PyTorch, and Day 43-45 attention fundamentals.

Safe Fallback (Offline Resilience):

Local models and classes in the curriculum mirror nanoGPT semantics directly.

Compared Candidate Sources & Selection Analysis (2)

GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.