46 / 90

Day 46: Transformer Block Assembly: LayerNorm, RMSNorm, and Residuals

Derive LayerNorm and RMSNorm normalizations with exact small tensors, explain why residual additions create gradient highways, and compare Pre-LN vs Post-LN stability.

Models, Pretraining & LoRA 480 active minutes (8 hours focused study)
01 120 min

Layer Normalization and RMSNorm: Mathematics and Numerical Stability

Agent-Assisted Engineering Check

Prerequisites

Variance and standard deviation from Day 28; vector arithmetic from Day 22.

Learning Objective

Compute LayerNorm and RMSNorm step by step for an activation vector x = [2.0, -1.0, 0.0, 3.0] and explain why modern architectures prefer RMSNorm.

Instructions

For x = [2.0, -1.0, 0.0, 3.0] of dimension d=4: (1) Compute mean mu = 1.0, variance sigma^2 = 2.5, and LN output with eps=1e-5. (2) Compute RMS(x) = sqrt(mean(x^2)) = sqrt(3.5) approx 1.8708, and RMSNorm output x / RMS(x). (3) Contrast computational cost: RMSNorm eliminates the mean-centering pass over memory.

Practice & Prediction Log

1. Calculate LayerNorm(x) to 4 decimal places with gamma=1, beta=0.
2. Calculate RMSNorm(x) to 4 decimal places with gamma=1.
3. Why is RMSNorm strictly cheaper in GPU kernel execution?

Independent Mastery Criterion

Learner correctly computes both normalizations by hand and articulates the memory bandwidth advantage of RMSNorm.

Authoritative Source Resources & Provenance

linking arXiv non-exclusive license / Open Access

Root Mean Square Layer Normalization

Author: Biao Zhang, Rico Sennrich (University of Edinburgh) | Original Source Link ↗

Relevant Section / Range: Section 4 RMSNorm (Equation 4) and Section 4.1 Invariance Analysis

Verified Version & Date: NeurIPS 2019 / arXiv:1910.07467v1 (Verified: 2026-10-30)

Selection Rationale:

Primary paper showing that mean-centering in LayerNorm can be discarded without performance loss, speeding up modern LLMs (LLaMA, Gemma).

Source Prerequisites:

Variance, standard deviation, and LayerNorm basics.

Safe Fallback (Offline Resilience):

RMSNorm formulas and numerical comparisons are derived completely in the lesson.

Compared Candidate Sources & Selection Analysis (2)
Ba et al. Layer Normalization (2016) ↗ compared_selected_for_day46 [Open Access]

Original LayerNorm formulation with mean-centering; RMSNorm is compared directly against it as an optimization.

Ioffe & Szegedy Batch Normalization (2015) ↗ compared_alternative [Open Access]

Batch-dependent statistics; unsuitable for autoregressive sequence modeling.

linking MIT

The Annotated Transformer

Author: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | Original Source Link ↗

Relevant Section / Range: Encoder and Decoder Stacks (class LayerNorm, class SublayerConnection)

Verified Version & Date: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

Selection Rationale:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

Source Prerequisites:

PyTorch tensor operations and attention theory.

Safe Fallback (Offline Resilience):

All block assembly and normalization logic are provided in runnable local Python snippets.

Compared Candidate Sources & Selection Analysis (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

02 120 min

Residual Connections and Identity Gradient Highways

Agent-Assisted Engineering Check

Prerequisites

Activity 46-01 and backpropagation chain rule from Day 35.

Learning Objective

Derive why the residual addition y = x + F(x) provides an unattenuated identity gradient highway dL/dx = dL/dy + dL/dy * dF/dx.

Instructions

Write out the vector derivative of y = x + F(x) with respect to x. Show that even if dF/dx is small or saturated near zero, dL/dx retains the direct dL/dy identity term, preventing vanishing gradients in deep 100+ layer Transformers.

Practice & Prediction Log

1. Write the chain rule expansion for dL/dx where y = x + F(x).
2. If dF/dx has eigenvalues with magnitude < 0.1, what prevents the gradient from vanishing?
3. State the dimensional invariant required for the addition x + F(x) to be defined.

Independent Mastery Criterion

Derive the gradient highway equation and state the strict shape invariant.

Authoritative Source Resources & Provenance

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

Author: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | Original Source Link ↗

Relevant Section / Range: Section 3.1 Encoder and Decoder Stacks (residual connection around each sub-layer, then layer normalization)

Verified Version & Date: arXiv:1706.03762v7 (Verified: 2026-10-30)

Selection Rationale:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

Source Prerequisites:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

Safe Fallback (Offline Resilience):

All mathematical formulas and structural definitions are self-contained in the lesson text.

Compared Candidate Sources & Selection Analysis (2)
Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

nanoGPT: model.py architecture implementation

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

Verified Version & Date: Git master (Verified: 2026-10-30)

Selection Rationale:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

Source Prerequisites:

Python, PyTorch, and Day 43-45 attention fundamentals.

Safe Fallback (Offline Resilience):

Local models and classes in the curriculum mirror nanoGPT semantics directly.

Compared Candidate Sources & Selection Analysis (2)
GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

03 120 min

Pre-LN vs Post-LN Architecture Analysis

Agent-Assisted Engineering Check

Prerequisites

Activities 46-01 and 46-02.

Learning Objective

Contrast Pre-LN (x = x + Attn(LN(x))) with the original Post-LN (x = LN(x + Attn(x))). Explain the initialization and warm-up findings of Xiong et al. (2020), and identify GPT-2 and Llama 2 as documented examples of pre-normalization (Llama 2 uses RMSNorm), not evidence of universal adoption.

Instructions

Trace activations and gradients through both configurations and explain the difference in training stability. Xiong et al. (2020) show that at initialization the Post-LN Transformer has large expected gradients for the parameters near the output layer, which is why it needs a learning-rate warm-up stage to train stably; in Pre-LN the gradients are well-behaved at initialization, and in their experiments Pre-LN Transformers trained without warm-up reached comparable results. GPT-2 (Section 2.3) moved layer normalization to the input of each sub-block, and Llama 2 (Section 2.2) uses pre-normalization with RMSNorm. These are examples, not a survey: do not claim that every modern LLM uses Pre-LN.

Practice & Prediction Log

1. Diagram the data flow formulas for both Pre-LN and Post-LN Transformer blocks.
2. In Post-LN, why does the residual path fail to act as a pure identity highway?
3. According to Xiong et al. (2020), why can Pre-LN be trained without the learning-rate warm-up stage that Post-LN needs?

Independent Mastery Criterion

Clear structural comparison and correct attribution of training stability to the unnormalized residual backbone.

Authoritative Source Resources & Provenance

linking arXiv.org perpetual non-exclusive license / Open Access

Attention Is All You Need

Author: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin | Original Source Link ↗

Relevant Section / Range: Section 3.1 Encoder and Decoder Stacks (LayerNorm(x + Sublayer(x)), the post-LN placement)

Verified Version & Date: arXiv:1706.03762v7 (Verified: 2026-10-30)

Selection Rationale:

Foundational primary literature defining the Transformer architecture, multi-head attention, and scaled dot-product attention.

Source Prerequisites:

Matrix multiplication and basic neural network concepts from Days 22-24 and 34-36.

Safe Fallback (Offline Resilience):

All mathematical formulas and structural definitions are self-contained in the lesson text.

Compared Candidate Sources & Selection Analysis (2)
Illustrated Transformer (Jay Alammar) ↗ compared_alternative [CC BY-NC-SA 4.0]

High-level visual intuition, but Vaswani et al. provides the primary mathematical specifications needed for implementation.

Annotated Transformer (Harvard NLP) ↗ compared_selected_for_day46_48 [MIT]

Co-primary: PyTorch line-by-line implementation alongside the paper equations.

linking MIT

The Annotated Transformer

Author: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | Original Source Link ↗

Relevant Section / Range: Encoder and Decoder Stacks (class SublayerConnection: x + dropout(sublayer(norm(x))))

Verified Version & Date: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

Selection Rationale:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

Source Prerequisites:

PyTorch tensor operations and attention theory.

Safe Fallback (Offline Resilience):

All block assembly and normalization logic are provided in runnable local Python snippets.

Compared Candidate Sources & Selection Analysis (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.

linking arXiv.org perpetual non-exclusive license / Open Access

On Layer Normalization in the Transformer Architecture

Author: Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu | Original Source Link ↗

Relevant Section / Range: Abstract; Section 3.3 Understanding the Transformer at initialization (Theorem 1, gradients of the last layer); Section 4 Experiments (Pre-LN without warm-up)

Verified Version & Date: arXiv:2002.04745v2 (Verified: 2026-11-01)

Selection Rationale:

Supports the activity's stability claim: Post-LN has large expected gradients near the output at initialization and needs warm-up; Pre-LN trained without warm-up reached comparable results in the paper's experiments.

Source Prerequisites:

LayerNorm and residual connections from Days 46-01 and 46-02; gradients from Days 31-36.

Safe Fallback (Offline Resilience):

The lesson states the Pre-LN and Post-LN formulas and the warm-up finding in its own words.

Compared Candidate Sources & Selection Analysis (2)
Liu et al.: Understanding the Difficulty of Training Transformers (EMNLP 2020) ↗ compared_deferred [arXiv Open Access]

Argues that Post-LN instability comes from amplified dependence on the residual branch and proposes Admin initialization; deeper than this activity needs, so deferred.

Wang et al.: Learning Deep Transformer Models for Machine Translation (ACL 2019) ↗ compared_alternative [arXiv Open Access]

Reports that pre-norm Transformers train deeper stacks than post-norm ones, but is empirical and specific to translation; Xiong et al. give the gradient analysis this activity asks learners to explain.

linking Open Access / OpenAI Public Publication

Language Models are Unsupervised Multitask Learners

Author: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever | Original Source Link ↗

Relevant Section / Range: Section 2.3 Model (layer normalization moved to the input of each sub-block)

Verified Version & Date: OpenAI Technical Report 2019 (Verified: 2026-10-30)

Selection Rationale:

Example of a widely used decoder-only LM that places layer normalization before each sub-block.

Source Prerequisites:

Autoregressive conditional likelihood, cross-entropy loss, perplexity exponentiation.

Safe Fallback (Offline Resilience):

Conditional log-likelihood equations, length normalization factors, and held-out cross-entropy calculations are fully derived in lesson material.

Compared Candidate Sources & Selection Analysis (2)
Kaplan et al. Scaling Laws for Neural Language Models ↗ compared_alternative [Open Access]

Focuses on empirical power-law loss curves; Radford et al. establishes the zero-shot log-likelihood task evaluation protocol.

Hoffmann et al. Training Compute-Optimal Large Language Models (Chinchilla) ↗ compared_alternative [Open Access]

Optimal compute scaling; Radford provides the fundamental zero-shot conditional probability formulation.

linking Open Access

Llama 2: Open Foundation and Fine-Tuned Chat Models

Author: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. | Original Source Link ↗

Relevant Section / Range: Section 2.2 Training Details (pre-normalization using RMSNorm)

Verified Version & Date: arXiv:2307.09288v2 (Verified: 2026-10-30)

Selection Rationale:

Example of a widely used decoder-only LLM that normalizes before each sub-layer, with RMSNorm in place of LayerNorm.

Source Prerequisites:

Transformer architecture, RLHF/SFT pipelines.

Safe Fallback (Offline Resilience):

The LLaMA-2 prompt structure, token boundary rules, and comparative analysis are fully worked out in curriculum lessons.

Compared Candidate Sources & Selection Analysis (2)
Touvron et al. LLaMA: Open and Efficient Foundation Language Models ↗ compared_alternative [Open Access]

Original foundation model paper without conversational chat tuning details; LLaMA-2 formalizes dialogue formatting.

Jiang et al. Mistral 7B ↗ compared_alternative [Open Access]

Adopts similar [INST] formatting; LLaMA-2 provides the foundational ablation on system prompt integration.

04 120 min

Independent Transformer Block Assembly Check

Independent Mastery Check

Prerequisites

Complete activities 46-01 through 46-03.

Learning Objective

Independently write a clean Pre-LN Transformer Attention Block in pure Python / PyTorch and verify input/output shape preservation.

Instructions

Implement class Block(nn.Module) with RMSNorm/LayerNorm, CausalSelfAttention, and residual addition. Assert input tensor of shape (2, 8, 64) produces output tensor of shape (2, 8, 64) with exact match.

Practice & Prediction Log

1. Provide the forward method code for the attention sublayer.
2. Verify that torch.allclose(out[:,0,:], expected) matches your hand derivation.
3. What error occurs if attention projection projects to a dimension d != d_model?

Independent Mastery Criterion

Working modular implementation passing all shape invariance and numerical checks.

Authoritative Source Resources & Provenance

linking MIT

nanoGPT: model.py architecture implementation

Author: Andrej Karpathy | Original Source Link ↗

Relevant Section / Range: model.py: class Block (x = x + attn(ln_1(x)); x = x + mlp(ln_2(x)))

Verified Version & Date: Git master (Verified: 2026-10-30)

Selection Rationale:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

Source Prerequisites:

Python, PyTorch, and Day 43-45 attention fundamentals.

Safe Fallback (Offline Resilience):

Local models and classes in the curriculum mirror nanoGPT semantics directly.

Compared Candidate Sources & Selection Analysis (2)
GPT-2 TensorFlow source (OpenAI) ↗ compared_alternative [Modified MIT License]

Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.

Llama 2 / 3 PyTorch source (Meta) ↗ compared_alternative [Llama 2 Community License]

Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.

linking MIT

The Annotated Transformer

Author: v2022: Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, Stella Biderman; original: Sasha Rush (Harvard NLP) | Original Source Link ↗

Relevant Section / Range: Encoder and Decoder Stacks (class EncoderLayer, class DecoderLayer)

Verified Version & Date: Updated PyTorch 2.0 edition (Verified: 2026-10-30)

Selection Rationale:

Clear reference implementation of Transformer modules, residual connections, and layer normalization.

Source Prerequisites:

PyTorch tensor operations and attention theory.

Safe Fallback (Offline Resilience):

All block assembly and normalization logic are provided in runnable local Python snippets.

Compared Candidate Sources & Selection Analysis (2)
HuggingFace Transformers library source ↗ compared_alternative [Apache 2.0]

Industrial scale with hundreds of model variations; Annotated Transformer is pedagogically focused on the core architecture.

PyTorch nn.Transformer documentation ↗ compared_alternative [Modified BSD]

Standard library implementation, but internal C++ bindings obscure the first-principles data flow.