AdamW 解耦权重衰减数学推导
Agent 辅助工程分析
先修要求
第 31-33 日梯度下降与动量法；L2 正则化概念。
学习目标
推导 AdamW 参数更新公式 w_{t+1} = w_t (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps)，阐释传统 Adam 中 L2 正则化的失真原因。
实验指南
在标准 Adam 中若使用传统 L2 正则化，其方式是将 lambda * w 直接加到损失梯度上。经过自适应二阶动量除法后，历史梯度大的参数受到的实际衰减被极度压缩，而梯度小的参数却被过度衰减。Loshchilov & Hutter (2019) 提出了 AdamW，将权重衰减与梯度动量解耦。计算 w_0 = 1.0, lambda = 0.1, eta = 0.01, g_1 = 0.5, beta_1 = 0.9, beta_2 = 0.999, eps = 1e-8 下的第 1 步更新。
实战与预测记录
1. 分别写出含 L2 正则化的传统 Adam 与 AdamW 的权重更新公式。
2. 手算第 1 步更新下的 m_hat_1, v_hat_1 及更新后的参数 w_1。
3. 为何解耦权重衰减能显著改善 Transformer 模型的泛化表现？
m_1 = (1 - 0.9) * 0.5 = 0.05 => m_hat_1 = 0.5。v_1 = (1 - 0.999) * 0.25 = 0.00025 => v_hat_1 = 0.25。sqrt(v_hat_1) = 0.5。
1. 传统 Adam 将梯度修改为 g + lambda * w；AdamW 则直接执行独立衰减：w_{t+1} = w_t * (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps)。
2. 手算结果：
- m_hat_1 = 0.5, v_hat_1 = 0.25, sqrt(v_hat_1) = 0.5
- 衰减部分：1.0 * (1 - 0.01 * 0.1) = 0.9990
- 梯度步长：0.01 * (0.5 / 0.5) = 0.0100
- 更新后参数：w_1 = 0.9990 - 0.0100 = 0.9890。
3. AdamW 保证了所有权重受到与梯度历史幅值无关的恒定比例收缩，防止频繁更新的词元通道权重无节制膨胀。
学习者能够推导两种更新公式，并手算验证第 1 步数值结果。
权威来源与学术版权说明
Decoupled Weight Decay Regularization
对应章节/时间戳: Section 2 Decoupling the Weight Decay from the Gradient-Based Update (Algorithm 2, AdamW)
核验版本与日期: ICLR 2019 / arXiv:1711.05101v3 (Verified: 2026-10-30)
Foundational primary literature defining AdamW, demonstrating that L2 regularization does not equal decoupled weight decay in adaptive gradient algorithms.
Stochastic gradient descent, L2 regularization, and momentum concepts from Days 31-33.
All update equations and mathematical proofs are self-contained in the curriculum lessons.
候选来源对比与遴选分析 (2)
Original Adam algorithm implementing L2 regularization via gradient addition, leading to sub-optimal generalization compared to decoupled weight decay.
Standard library reference; Loshchilov & Hutter provides theoretical mathematical derivations and empirical justification.
nanoGPT: train.py pretraining loop
对应章节/时间戳: train.py: optimizer = model.configure_optimizers(weight_decay, learning_rate, (beta1, beta2), device_type)
核验版本与日期: Git master commit 71c3905 (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Python, PyTorch, and Day 46-48 Transformer model architecture.
Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.
候选来源对比与遴选分析 (2)
Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.
High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.