第 51 日：运行预训练闭环：损失追踪、检查点保存与故障诊断

01 120 分钟

预训练核心步进：前向、损失、反向传播与梯度裁剪

Agent 辅助工程分析

先修要求

第 49 日数据加载器与第 50 日 AdamW 优化器。

学习目标

执行经典的 5 步训练迭代，理解 set_to_none=True 的显存释放机制，施加全局 L2 范数梯度裁剪。

实验指南

学习来源：在本活动现有的阅读与练习时段内，参考 CS336 Assignment 1: Basics (Spring 2025 tagged release) 的 tests/adapters.py: run_gradient_clipping; tests/test_nn_utils.py: test_gradient_clipping，用自己的话解释一处与本地任务的联系。保留现有练习与验收检查；若外部资料无法访问，使用课程自写的解释。在每个预训练步进中：(1) optimizer.zero_grad(set_to_none=True) 释放梯度张量；(2) logits, loss = model(x, y) 计算前向传播与交叉熵损失；(3) loss.backward() 累加梯度；(4) torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) 将全局梯度 L2 模长约束至不超过 1.0；(5) optimizer.step() 更新参数。分析梯度裁剪防止权重暴走的关键机理。

实战与预测记录

1. 给出执行一次完整训练迭代的 Python 代码。
2. 若未裁剪的全局梯度模长 ||g||_2 = 5.0 且 max_norm = 1.0，各个参数的梯度会被乘以何种缩放比例？
3. 为何 optimizer.zero_grad(set_to_none=True) 在工程性能上显著优于默认的 zero_grad()？

标记为完成

查看参考解答与分析:

1. 训练步进代码：

optimizer.zero_grad(set_to_none=True)
logits, loss = model(x, y)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

2. 缩放系数：1.0 / 5.0 = 0.2。所有参数的梯度均被乘以 0.2，将全局 L2 范数硬性限制在 1.0 以内。
3. set_to_none=True 直接释放了梯度张量的显存指针，无需在已有缓冲区中逐个填充零，节省了显存读写带宽，并加速随后的 autograd 内存分配。

独立掌握标准

学习者能够编写规范的 5 步训练逻辑，准确计算梯度裁剪系数，并阐明 set_to_none=True 的显存优势。

权威来源与学术版权说明

linking MIT

CS336 Assignment 1: Basics (Spring 2025 tagged release)

作者/提供方: Stanford University CS336 Staff | 原始来源链接 ↗

对应章节/时间戳: tests/adapters.py: run_gradient_clipping; tests/test_nn_utils.py: test_gradient_clipping

核验版本与日期: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01

入选理由:

Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.

先修背景:

Fluent Python, PyTorch, and linear algebra

安全降级（离线韧性）:

Local self-contained pretraining scripts run on standard local CPUs or budget GPUs without Stanford cluster dependency.

候选来源对比与遴选分析 (2)

Karpathy nanoGPT ↗ compared_alternative [MIT]

Simpler code layout, useful complementary reference for minimal PyTorch Transformer.

Hugging Face Transformers Course ↗ compared_alternative [Apache-2.0]

High-level API focus rather than building from raw tensor fundamentals.

linking MIT

nanoGPT: train.py pretraining loop

作者/提供方: Andrej Karpathy | 原始来源链接 ↗

对应章节/时间戳: train.py: training loop (while True): forward under ctx, scaler.scale(loss).backward(), clip_grad_norm_(grad_clip), scaler.step(optimizer)

核验版本与日期: Git master commit 71c3905 (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

先修背景:

Python, PyTorch, and Day 46-48 Transformer model architecture.

安全降级（离线韧性）:

Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.

候选来源对比与遴选分析 (2)

Hugging Face Accelerate / Trainer ↗ compared_alternative [Apache-2.0]

Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.

Megatron-LM (NVIDIA) ↗ compared_alternative [BSD-3-Clause]

High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.

02 120 分钟

训练集与验证集损失估算及过拟合追踪

独立完成核心推导

先修要求

活动 51-01 与 PyTorch 自动求导机制。

学习目标

在 torch.no_grad() 与 model.eval() 环境下实现 estimate_loss 函数，监控训练集与验证集损失差异以检测过拟合。

实验指南

单批次训练损失具有较强随机噪声。为了准确评估模型泛化能力，需定期（如每 250 步）分别在 train 和 val 上采样 200 个批次并取均值。切换至 model.eval() 关闭 Dropout 随机性，并在 torch.no_grad() 上下文中运行以避免在显存中保留庞大的反向传播计算图。

实战与预测记录

1. 给出 estimate_loss(model, eval_iters) 的 Python 实现。
2. 若在 estimate_loss 中遗漏了 torch.no_grad()，会引发何种严重的显存故障？
3. 如何通过训练集与验证集损失曲线的分离趋势诊断过拟合？

标记为完成

查看参考解答与分析:

1. estimate_loss 实现：

@torch.no_grad()
def estimate_loss(model, eval_iters=200):
    out = {}
    model.eval()
    for split in ['train', 'val']:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            logits, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean().item()
    model.train()
    return out

2. 若无 torch.no_grad()，PyTorch 会为 200 个评估批次全部构建并保留动态计算图，导致显存瞬间被耗尽并抛出致命的 CUDA Out-Of-Memory (OOM) 崩溃。
3. 当训练集损失单调持续下降，而验证集损失逐渐停滞甚至拐头上升时，两者的泛化差距（generalization gap）扩大，表明模型已发生过拟合，开始死记硬背训练语料的局部特征。

独立掌握标准

独立实现规范的评估函数，正确配置 eval 模式与 no_grad，并准确认定过拟合特征。

权威来源与学术版权说明

linking MIT

nanoGPT: train.py pretraining loop

作者/提供方: Andrej Karpathy | 原始来源链接 ↗

对应章节/时间戳: train.py: estimate_loss() averages eval_iters batches per split; run every eval_interval

核验版本与日期: Git master commit 71c3905 (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

先修背景:

Python, PyTorch, and Day 46-48 Transformer model architecture.

安全降级（离线韧性）:

Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.

候选来源对比与遴选分析 (2)

Hugging Face Accelerate / Trainer ↗ compared_alternative [Apache-2.0]

Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.

Megatron-LM (NVIDIA) ↗ compared_alternative [BSD-3-Clause]

High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.

linking MIT

CS336 Assignment 1: Basics (Spring 2025 tagged release)

作者/提供方: Stanford University CS336 Staff | 原始来源链接 ↗

对应章节/时间戳: README setup; tests/adapters.py; tests/test_optimizer.py; tests/test_nn_utils.py; tests/test_serialization.py

核验版本与日期: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01

入选理由:

Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.

先修背景:

Fluent Python, PyTorch, and linear algebra

安全降级（离线韧性）:

Local self-contained pretraining scripts run on standard local CPUs or budget GPUs without Stanford cluster dependency.

候选来源对比与遴选分析 (2)

Karpathy nanoGPT ↗ compared_alternative [MIT]

Simpler code layout, useful complementary reference for minimal PyTorch Transformer.

Hugging Face Transformers Course ↗ compared_alternative [Apache-2.0]

High-level API focus rather than building from raw tensor fundamentals.

03 120 分钟

检查点序列化、原子保存与训练恢复

Agent 辅助工程分析

先修要求

活动 51-01 与 51-02。

学习目标

将模型权重、AdamW 优化器动量状态、迭代步数与验证集损失序列化为检查点，实现精确无缝续训。

实验指南

学习来源：在本活动现有的阅读与练习时段内，参考 CS336 Assignment 1: Basics (Spring 2025 tagged release) 的 tests/adapters.py: run_save_checkpoint, run_load_checkpoint; tests/test_serialization.py: test_checkpointing，用自己的话解释一处与本地任务的联系。保留现有练习与验收检查；若外部资料无法访问，使用课程自写的解释。预训练通常持续数天，极易因硬件故障或算力抢占而中断。检查点必须持久化保存：model.state_dict()、optimizer.state_dict()、iter_num、best_val_loss 与 config。为防止保存过程中断产生损坏的半写文件，应采用原子写入模式（先写入临时文件，再用 os.replace 重命名）。

实战与预测记录

1. 给出包含原子写入机制的检查点保存代码片段。
2. 给出从检查点恢复模型与优化器状态的代码。
3. 若在恢复训练时仅加载了模型权重而漏掉了 optimizer.state_dict()，会引发何种优化异常？

标记为完成

查看参考解答与分析:

1. 保存检查点：

checkpoint = {
    'model': model.state_dict(),
    'optimizer': optimizer.state_dict(),
    'iter_num': iter_num,
    'best_val_loss': best_val_loss,
    'config': config,
}
temp_path = os.path.join(out_dir, 'ckpt_tmp.pt')
final_path = os.path.join(out_dir, 'ckpt.pt')
torch.save(checkpoint, temp_path)
os.replace(temp_path, final_path)

2. 恢复检查点：

ckpt = torch.load(final_path, map_location=device)
model.load_state_dict(ckpt['model'])
optimizer.load_state_dict(ckpt['optimizer'])
iter_num = ckpt['iter_num']
best_val_loss = ckpt['best_val_loss']

3. 优化器动量被清零后，会在当前学习率下以"冷启动"方式直接执行未校准的大步长更新，往往导致损失在重启后的前几步出现剧烈尖峰（loss spike），甚至造成模型发散。

独立掌握标准

学习者能够编写鲁棒的原子检查点存取代码，并准确解释优化器状态丢失引发的冷启动异常。

权威来源与学术版权说明

linking MIT

CS336 Assignment 1: Basics (Spring 2025 tagged release)

作者/提供方: Stanford University CS336 Staff | 原始来源链接 ↗

对应章节/时间戳: tests/adapters.py: run_save_checkpoint, run_load_checkpoint; tests/test_serialization.py: test_checkpointing

核验版本与日期: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01

入选理由:

Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.

先修背景:

Fluent Python, PyTorch, and linear algebra

安全降级（离线韧性）:

Local self-contained pretraining scripts run on standard local CPUs or budget GPUs without Stanford cluster dependency.

候选来源对比与遴选分析 (2)

Karpathy nanoGPT ↗ compared_alternative [MIT]

Simpler code layout, useful complementary reference for minimal PyTorch Transformer.

Hugging Face Transformers Course ↗ compared_alternative [Apache-2.0]

High-level API focus rather than building from raw tensor fundamentals.

linking MIT

nanoGPT: train.py pretraining loop

作者/提供方: Andrej Karpathy | 原始来源链接 ↗

对应章节/时间戳: train.py: checkpoint dict {'model', 'optimizer', 'model_args', 'iter_num', 'best_val_loss', 'config'} written with torch.save to out_dir/ckpt.pt; the init_from == 'resume' branch reloads it

核验版本与日期: Git master commit 71c3905 (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

先修背景:

Python, PyTorch, and Day 46-48 Transformer model architecture.

安全降级（离线韧性）:

Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.

候选来源对比与遴选分析 (2)

Hugging Face Accelerate / Trainer ↗ compared_alternative [Apache-2.0]

Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.

Megatron-LM (NVIDIA) ↗ compared_alternative [BSD-3-Clause]

High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.

04 120 分钟

预训练闭环实操与异常诊断考核

完全独立闭卷考核

先修要求

完成活动 51-01 至 51-03。

学习目标

独立运行完整的预训练脚本，诚实记录观测指标，诊断预训练异常（损失突变为 NaN、陷入平台等）。

实验指南

在闭卷环境下诊断大模型预训练中最常见的运行异常：(1) 损失突变发散为 NaN；(2) 损失长时间停滞在 ln(V) ≈ 10.8。在小型数据集上执行 50 步参考训练，并如实汇报损失下降指标，严禁作出无依据的收敛承诺。

实战与预测记录

1. 列举导致预训练损失变为 NaN 的三大主因及相应修复方案。
2. 若损失在训练上千步后依然死锁在 10.8 左右，表明哪个系统环节发生了故障？
3. 汇报在小型词元数据集上运行 50 步训练所观测到的实际损失变化特征。

标记为完成

独立掌握标准

学习者能够准确排查常见预训练异常，并在不作无根据收敛保证的前提下客观描述训练行为。

权威来源与学术版权说明

linking MIT

nanoGPT: train.py pretraining loop

作者/提供方: Andrej Karpathy | 原始来源链接 ↗

对应章节/时间戳: train.py: config defaults, get_lr(it) and the training loop (while True)

核验版本与日期: Git master commit 71c3905 (Verified: 2026-10-30)

入选理由:

Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.

先修背景:

Python, PyTorch, and Day 46-48 Transformer model architecture.

安全降级（离线韧性）:

Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.

候选来源对比与遴选分析 (2)

Hugging Face Accelerate / Trainer ↗ compared_alternative [Apache-2.0]

Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.

Megatron-LM (NVIDIA) ↗ compared_alternative [BSD-3-Clause]

High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.

linking arXiv non-exclusive license / Open Access

Decoupled Weight Decay Regularization

作者/提供方: Ilya Loshchilov, Frank Hutter | 原始来源链接 ↗

对应章节/时间戳: Section 2 Decoupling the Weight Decay from the Gradient-Based Update (Algorithm 2, AdamW)

核验版本与日期: ICLR 2019 / arXiv:1711.05101v3 (Verified: 2026-10-30)

入选理由:

Foundational primary literature defining AdamW, demonstrating that L2 regularization does not equal decoupled weight decay in adaptive gradient algorithms.

先修背景:

Stochastic gradient descent, L2 regularization, and momentum concepts from Days 31-33.

安全降级（离线韧性）:

All update equations and mathematical proofs are self-contained in the curriculum lessons.

候选来源对比与遴选分析 (2)

Kingma & Ba: Adam: A Method for Stochastic Optimization (2014) ↗ compared_alternative [Open Access]

Original Adam algorithm implementing L2 regularization via gradient addition, leading to sub-optimal generalization compared to decoupled weight decay.

PyTorch torch.optim.AdamW documentation ↗ compared_alternative [Modified BSD]

Standard library reference; Loshchilov & Hutter provides theoretical mathematical derivations and empirical justification.