Final Checkpoint Validation, Perplexity and Coherence Evaluation
Agent-Assisted Engineering Check
Prerequisites
Day 59 Cloud Pretraining execution and Day 53 Validation metrics.
Learning Objective
Evaluate the final trained checkpoint on held-out validation tokens, compute validation loss and perplexity, and run autoregressive sampling to audit text coherence.
Instructions
1. Load the final checkpoint with map_location='cpu' (or your GPU), load its 'model' state into a freshly built model, and call model.eval().
2. Build a fixed list of held-out validation batches, so that two checkpoints are always compared on identical tokens.
3. Compute L_val as the mean cross-entropy over those batches inside torch.no_grad(), then PPL = exp(L_val).
4. Compare with the uniform baseline ln(V): for character-level Shakespeare, V = 65 and ln(65) = 4.1744, which is why L_val below 4.0 means the model beats guessing.
5. Generate five samples with temperature=0.8 and top_k=40 from the same prompt, and test each for degenerate loops (a long run of one token, or too few distinct tokens).
6. Read the samples yourself and record the checkpoint step, L_val, PPL, the samples, the decoding settings and the seed.
Time plan (120 min): reading 20 + exercise 70 + check 30. Reading: the estimate_loss and sampling code in nanoGPT and the CS336 notes on evaluation. Exercise: steps 1-6 on your final checkpoint. Check: explain the L_val and PPL thresholds from ln(V) without notes.
Practice & Prediction Log
Run an evaluation script on the final checkpoint. Assert that validation loss is lower than initial random initialization (L_val < 4.0 for Shakespeare, or PPL < 50) and that generated completions contain no repetitive single-token loops.
Check perplexity = math.exp(val_loss). Verify text contains diverse tokens.
import math
import torch
@torch.no_grad()
def estimate_loss(model, val_batches):
# Mean cross-entropy over a FIXED list of held-out (x, y) batches, so two checkpoints are
# compared on identical tokens. model(x, y) returns (logits, loss) as in the Day 48 model.
model.eval()
losses = [model(x, y)[1].item() for x, y in val_batches]
return sum(losses) / len(losses)
@torch.no_grad()
def generate_sample(model, idx, max_new_tokens=100, temperature=0.8, top_k=40, block_size=256):
model.eval()
for _ in range(max_new_tokens):
logits, _ = model(idx[:, -block_size:])
logits = logits[:, -1, :] / temperature
kth = torch.topk(logits, top_k).values[:, [-1]]
logits[logits < kth] = float('-inf') # keep only the top_k candidates
idx = torch.cat([idx, torch.multinomial(torch.softmax(logits, dim=-1), 1)], dim=1)
return idx
def has_token_loop(tokens, max_run=8, min_distinct_ratio=0.3):
# Degenerate output: one token repeated more than max_run times in a row,
# or too few distinct tokens overall (e.g. 'the the the ...').
longest = run = 1
for a, b in zip(tokens, tokens[1:]):
run = run + 1 if a == b else 1
longest = max(longest, run)
return longest > max_run or len(set(tokens)) / len(tokens) < min_distinct_ratio
ckpt = torch.load('ckpt.pt', map_location='cpu')
model.load_state_dict(ckpt['model'])
model.eval()
val_loss = estimate_loss(model, val_batches)
ppl = math.exp(val_loss)
print(f'L_val={val_loss:.4f} PPL={ppl:.2f} uniform baseline ln(V)={math.log(vocab_size):.4f}')
prompt = torch.tensor([encode('Once upon a time')])
samples = [generate_sample(model, prompt)[0, prompt.size(1):].tolist() for _ in range(5)]
# Why these thresholds: an untrained model guesses uniformly, so its loss is ln(V).
# For character-level Shakespeare V = 65 and ln(65) = 4.1744, so L_val < 4.0 means the model beats
# guessing; PPL < 50 is the same check for a larger vocabulary (ln(50) = 3.9120).
assert val_loss < math.log(vocab_size)
assert ppl < 50.0
assert not any(has_token_loop(s) for s in samples)
for s in samples:
print(repr(decode(s))) # read them: grammatical fragments, names, line breaks; not one token
# Record: checkpoint step, L_val, PPL, the five samples, temperature=0.8, top_k=40 and the seed.
Validate final checkpoint performance with mathematical perplexity and coherence verification.
Authoritative Source Resources & Provenance
nanoGPT: train.py pretraining loop
Relevant Section / Range: train.py: estimate_loss() averages eval_iters batches per split; run every eval_interval; best_val_loss decides when a checkpoint is saved
Verified Version & Date: Git master commit 71c3905 (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
Python, PyTorch, and Day 46-48 Transformer model architecture.
Runnable pretraining scripts and step-by-step tensor mechanics are fully self-contained in the curriculum lessons.
Compared Candidate Sources & Selection Analysis (2)
Heavy abstraction layer wrapping distributed training; nanoGPT exposes raw PyTorch loops without framework overhead.
High-performance enterprise distributed framework; unsuitable for local educational first-principles comprehension.
CS336 Assignment 1: Basics (Spring 2025 tagged release)
Relevant Section / Range: README setup; tests/adapters.py; tests/test_optimizer.py; tests/test_nn_utils.py; tests/test_serialization.py
Verified Version & Date: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01
Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.
Fluent Python, PyTorch, and linear algebra
Local self-contained pretraining scripts run on standard local CPUs or budget GPUs without Stanford cluster dependency.
Compared Candidate Sources & Selection Analysis (2)
Simpler code layout, useful complementary reference for minimal PyTorch Transformer.
High-level API focus rather than building from raw tensor fundamentals.