AdamW Decoupled Weight Decay Mathematical Derivation
Agent-Assisted Engineering Check
Prerequisites
Gradient descent and momentum from Days 31-33; L2 regularization concepts.
Learning Objective
Derive the AdamW update equation w_{t+1} = w_t (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps) and explain why L2 regularization fails in Adam.
Instructions
In standard Adam with L2 regularization, the loss gradient is modified by adding lambda * w. When scaled by the adaptive second moment sqrt(v_t), weights with large historical gradients experience suppressed regularization, while weights with small gradients undergo excessive decay. Loshchilov & Hutter (2019) resolved this by decoupling weight decay directly from gradient scaling. Compute the step 1 update for w_0 = 1.0, lambda = 0.1, eta = 0.01, g_1 = 0.5, beta_1 = 0.9, beta_2 = 0.999, eps = 1e-8.
Practice & Prediction Log
1. Write the update equation for Adam with L2 regularization vs AdamW.
2. Compute step 1 values for m_hat_1, v_hat_1, and updated weight w_1 under AdamW.
3. Why does decoupled weight decay significantly improve Transformer generalization?
m_1 = (1 - 0.9) * 0.5 = 0.05 => m_hat_1 = 0.05 / 0.1 = 0.5. v_1 = (1 - 0.999) * 0.25 = 0.00025 => v_hat_1 = 0.00025 / 0.001 = 0.25. sqrt(v_hat_1) = 0.5.
1. Adam with L2 adds lambda * w to gradient g_t before moment accumulation. AdamW updates weights directly via w_{t+1} = w_t * (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps).
2. Hand calculation:
- m_hat_1 = 0.5, v_hat_1 = 0.25, sqrt(v_hat_1) = 0.5
- Decay factor: 1.0 * (1 - 0.01 * 0.1) = 0.9990
- Gradient step: 0.01 * (0.5 / 0.5) = 0.0100
- Result: w_1 = 0.9990 - 0.0100 = 0.9890.
3. AdamW provides uniform, scale-invariant decay across all parameters, preventing weights in frequent token paths from blowing up without being regularized.
Learner derives both update formulas and verifies step 1 numerical values by hand.
Authoritative Source Resources & Provenance
Decoupled Weight Decay Regularization
Relevant Section / Range: Section 2 Decoupling the Weight Decay from the Gradient-Based Update (Algorithm 2, AdamW)
Verified Version & Date: ICLR 2019 / arXiv:1711.05101v3 (Verified: 2026-10-30)
Foundational primary literature defining AdamW, demonstrating that L2 regularization does not equal decoupled weight decay in adaptive gradient algorithms.
Stochastic gradient descent, L2 regularization, and momentum concepts from Days 31-33.
All update equations and mathematical proofs are self-contained in the curriculum lessons.
Compared Candidate Sources & Selection Analysis (2)
Original Adam algorithm implementing L2 regularization via gradient addition, leading to sub-optimal generalization compared to decoupled weight decay.
Standard library reference; Loshchilov & Hutter provides theoretical mathematical derivations and empirical justification.
nanoGPT: train.py pretraining loop
Relevant Section / Range: train.py: optimizer = model.configure_optimizers(weight_decay, learning_rate, (beta1, beta2), device_type)
Verified Version & Date: Git master commit 71c3905 (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Python, PyTorch, and Day 46-48 Transformer model architecture.
Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.
Compared Candidate Sources & Selection Analysis (2)
Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.
High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.