損壞、截斷與不相容檢查點的精準診斷與自動回退
Agent 輔助工程檢查
前置要求
第 55 天檢查點基礎與 Python 異常處理機制。
學習目標
診斷非完整寫入造成的截斷文件（EOFError、UnpicklingError 反序列化崩潰），集成 SHA256 校驗和旁路驗證，並實現自動向後遍歷健康歷史快照的自癒回退管線。
操作指引
1. 探究進程非正常退出導致的半寫故障特徵：文件位元組數不為零，但末尾結構殘缺。
2. 直接調用 torch.load 會拋出 _pickle.UnpicklingError: pickle data was truncated 或 EOFError。
3. 實現旁路雜湊校驗機制：落盤時同步輸出 'ckpt.pt.sha256'，加載前執行雜湊比對，快速剔除受損文件。
4. 構建多級自動回退機制：當最新快照（如 step 10000）校驗失敗或反序列化崩潰時，無縫降級嘗試 step 9000 快照並記錄告警日誌。
自主實作與記錄
編寫容錯加載函數 safe_load_checkpoint(checkpoint_dir)，逆序掃描目錄下的所有快照，安全捕獲損壞異常，成功定位並加載最新可用的完整快照。
按倒序（sorted(files, reverse=True)）遍歷快照。若存在 .sha256 檔案，先校驗雜湊；再用 try/except (EOFError, RuntimeError, Exception) 包裹 torch.load，跳過任何載入失敗的檔案。
import os, torch, hashlib
def safe_load_checkpoint(ckpt_dir):
files = sorted([f for f in os.listdir(ckpt_dir) if f.startswith('ckpt_') and f.endswith('.pt')], reverse=True)
for fname in files:
path = os.path.join(ckpt_dir, fname)
try:
if os.path.exists(path + '.sha256'):
with open(path, 'rb') as f:
digest = hashlib.sha256(f.read()).hexdigest()
with open(path + '.sha256') as f:
if f.read().split()[0] != digest:
raise ValueError('sha256 mismatch')
ckpt = torch.load(path, map_location='cpu')
print(f'Successfully recovered from {fname}')
return ckpt
except Exception as e:
print(f'Skipping corrupted checkpoint {fname}: {e}')
raise RuntimeError('No valid checkpoint found in directory')
在最新快照被截斷損壞的嚴苛測試下，系統能自動降級並成功從次新快照無縫恢復。
權威來源資源與權利證明
PyTorch Tutorial: Saving and Loading Models
對應章節 / 時間戳: Saving & Loading a General Checkpoint for Inference and/or Resuming Training
驗證版本與日期: PyTorch 2.x Documentation (Verified: 2026-10-30)
Official documentation detailing state_dict serialization, optimizer state saving, and safe loading patterns.
Python, PyTorch tensors and nn.Module basics.
All checkpoint serialization semantics, dictionary schema keys, and safe recovery steps are self-contained in lesson code.
對比候選資源與選型分析 (2)
TensorFlow trackable checkpoint architecture; PyTorch's explicit dictionary is our target implementation standard.
Safetensors provides safe zero-copy tensor storage without pickle, compared directly when discussing pickle execution risks.
nanoGPT: train.py pretraining loop
對應章節 / 時間戳: train.py: checkpoint dict {'model', 'optimizer', 'model_args', 'iter_num', 'best_val_loss', 'config'} written with torch.save to out_dir/ckpt.pt; the init_from == 'resume' branch reloads it
驗證版本與日期: Git master commit 71c3905 (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Transformer architecture from Days 46-48 and training loop from Days 49-51.
Complete checkpoint saving patterns and resuming loops are directly implemented in lesson exercises.
對比候選資源與選型分析 (2)
High-complexity multi-GPU distributed model-parallel checkpointing; nanoGPT provides accessible single-GPU clarity.
High-level framework callback abstraction; nanoGPT keeps explicit control over state dict dictionaries.