AdamW 解耦權重衰減數學推導
Agent 輔助工程分析
先修要求
第 31-33 日梯度下降與動量法；L2 正則化概念。
學習目標
推導 AdamW 參數更新公式 w_{t+1} = w_t (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps)，闡釋傳統 Adam 中 L2 正則化的失真原因。
實驗指南
在標準 Adam 中若使用傳統 L2 正則化，其方式是將 lambda * w 直接加到損失梯度上。經過自適應二階動量除法後，歷史梯度大的參數受到的實際衰減被極度壓縮，而梯度小的參數卻被過度衰減。Loshchilov & Hutter (2019) 提出了 AdamW，將權重衰減與梯度動量解耦。計算 w_0 = 1.0, lambda = 0.1, eta = 0.01, g_1 = 0.5, beta_1 = 0.9, beta_2 = 0.999, eps = 1e-8 下的第 1 步更新。
實戰與預測記錄
1. 分別寫出含 L2 正則化的傳統 Adam 與 AdamW 的權重更新公式。
2. 手算第 1 步更新下的 m_hat_1, v_hat_1 及更新後的參數 w_1。
3. 為何解耦權重衰減能顯著改善 Transformer 模型的泛化表現？
m_1 = (1 - 0.9) * 0.5 = 0.05 => m_hat_1 = 0.5。v_1 = (1 - 0.999) * 0.25 = 0.00025 => v_hat_1 = 0.25。sqrt(v_hat_1) = 0.5。
1. 傳統 Adam 將梯度修改為 g + lambda * w；AdamW 則直接執行獨立衰減：w_{t+1} = w_t * (1 - eta * lambda) - eta * m_hat_t / (sqrt(v_hat_t) + eps)。
2. 手算結果：
- m_hat_1 = 0.5, v_hat_1 = 0.25, sqrt(v_hat_1) = 0.5
- 衰減部分：1.0 * (1 - 0.01 * 0.1) = 0.9990
- 梯度步長：0.01 * (0.5 / 0.5) = 0.0100
- 更新後參數：w_1 = 0.9990 - 0.0100 = 0.9890。
3. AdamW 保證了所有權重受到與梯度歷史幅值無關的恆定比例收縮，防止頻繁更新的詞元通道權重無節制膨脹。
學習者能夠推導兩種更新公式，並手算驗證第 1 步數值結果。
權威來源與學術版權說明
Decoupled Weight Decay Regularization
對應章節/時間戳: Section 2 Decoupling the Weight Decay from the Gradient-Based Update (Algorithm 2, AdamW)
核驗版本與日期: ICLR 2019 / arXiv:1711.05101v3 (Verified: 2026-10-30)
Foundational primary literature defining AdamW, demonstrating that L2 regularization does not equal decoupled weight decay in adaptive gradient algorithms.
Stochastic gradient descent, L2 regularization, and momentum concepts from Days 31-33.
All update equations and mathematical proofs are self-contained in the curriculum lessons.
候選來源對比與遴選分析 (2)
Original Adam algorithm implementing L2 regularization via gradient addition, leading to sub-optimal generalization compared to decoupled weight decay.
Standard library reference; Loshchilov & Hutter provides theoretical mathematical derivations and empirical justification.
nanoGPT: train.py pretraining loop
對應章節/時間戳: train.py: optimizer = model.configure_optimizers(weight_decay, learning_rate, (beta1, beta2), device_type)
核驗版本與日期: Git master commit 71c3905 (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Python, PyTorch, and Day 46-48 Transformer model architecture.
Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.
候選來源對比與遴選分析 (2)
Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.
High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.