Evaluation Protocols: Train vs Validation Loss Gap
Agent-Assisted Engineering Check
Prerequisites
Day 51 Pretraining Loop and Day 42 Cross-Entropy.
Learning Objective
Measure the divergence between train loss and validation loss across training steps and define the exact onset of overfitting.
Instructions
Define estimate_loss() evaluating the model without gradient computation (torch.no_grad()) across eval_iters batches. Plot or tabulate: Step 0: train 4.10, val 4.12; Step 500: train 2.20, val 2.25; Step 1500: train 1.10, val 1.95. Mark step 1000 as the generalization plateau.
Practice & Prediction Log
1. Why must the evaluation loop execute inside torch.no_grad() and model.eval()?
2. What does a growing gap between training loss and validation loss indicate?
3. State the rule for choosing the best checkpoint: train loss or validation loss?
model.eval() disables dropout; torch.no_grad() prevents graph creation and reduces memory use.
torch.no_grad() eliminates gradient graph retention, reducing VRAM by >50%; model.eval() deactivates stochastic dropout. A widening gap indicates overfitting (memorization of training tokens rather than generalized language modeling). Best checkpoint is strictly chosen by lowest validation loss.
Defines evaluation hygiene and explains the validation-loss selection criterion.
Authoritative Source Resources & Provenance
nanoGPT: model.py architecture implementation
Relevant Section / Range: model.py: class GPT forward(idx, targets): wte + wpe, Blocks, ln_f, lm_head (returns cross-entropy loss when targets are given)
Verified Version & Date: Git master (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Python, PyTorch, and Day 43-45 attention fundamentals.
Local models and classes in the curriculum mirror nanoGPT semantics directly.
Compared Candidate Sources & Selection Analysis (2)
Original legacy TensorFlow 1.x code; nanoGPT provides clean modern PyTorch equivalent.
Includes advanced production features (RoPE, GQA), but nanoGPT's simplicity is ideal for the core Days 46-48 assembly.
Google Machine Learning Crash Course
Relevant Section / Range: Modules: Training and Test Sets; Overfitting; Classification accuracy
Verified Version & Date: Current online edition (Verified: 2026-10-30)
Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.
Basic algebra; short self-contained modules.
Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.
Compared Candidate Sources & Selection Analysis (2)
Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.
Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.