Train, validation, and test: fixing the comparison
Assistance allowed only after prediction and observed evidence are recorded.
Prerequisites
Day 28 statistics; the prediction-review routine.
Learning Objective
Explain the role of train/validation/test splits in evaluation and why the test set is read once; fix the evaluation protocol before any tuning begins.
Instructions
Evaluation answers 'does it generalize?' using data the model never trained on. The training split fits parameters; the validation split guides choices (which model, which settings); the test split is the final, untouched read. With 1,000 examples a 60/20/20 split gives 600 train, 200 validation, 200 test. Comparisons are only valid when every candidate is scored on the same split with the same metric — fix that protocol before tuning, not after seeing results.
Practice & Prediction Log
A learner has 1,000 labeled examples. (a) Allocate train/validation/test at 60/20/20. (b) Model M1 was selected after 40 trials scored on the test set — name the protocol violation and its consequence. (c) State what must be identical before M1 and M2 scores can be compared.
Repeated test reads turn 'test' into validation — the reported number becomes optimistically biased. Before comparing: same held-out data, same metric, same preprocessing.
(a) 600 train, 200 validation, 200 test. (b) The test set was reused for model selection, so the reported score is biased upward — the violation is selection on the test set; fix by selecting on validation and reading test once. (c) Same test split, same metric, same preprocessing and evaluation code; otherwise the difference is uninterpretable.
Learner states each split's role in evaluation, names the test-reuse violation, and writes the fixed-protocol checklist before any tuning.
Authoritative Source Resources & Provenance
Dive into Deep Learning (d2l.ai)
Relevant Section / Range: Section: Model Selection, Underfitting, and Overfitting
Verified Version & Date: Current online edition (Verified: 2026-10-30)
Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.
Linear algebra, calculus, and probability at the Days 19-28 level.
All required formulas and worked numbers are reproduced as original explanations; the book is further reading.
Compared Candidate Sources & Selection Analysis (2)
Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.
Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.
Google Machine Learning Crash Course
Relevant Section / Range: Modules: Training and Test Sets; Validation
Verified Version & Date: Current online edition (Verified: 2026-10-30)
Industrial, beginner-legible treatment of training/test splitting and overfitting; complements d2l.ai's mathematical framing.
Basic algebra; short self-contained modules.
Split discipline, leakage rules, and metric definitions are fully explained inside the lesson; the course is further reading.
Compared Candidate Sources & Selection Analysis (2)
Co-primary: deeper mathematical coverage of model selection, generalization, and optimization for the mathematics-track capstone.
Normative reference once learners call sklearn; kept as an alternative because these days teach evaluation protocol before library usage.