Gradient descent mechanics on a convex bowl
Assistance allowed only after prediction and observed evidence are recorded.
Prerequisites
Day 29 evaluation; single-variable derivative (power rule) from the mathematics block.
Learning Objective
Derive the gradient-descent update x ← x − η·2x = x(1 − 2η) and predict the whole optimization trajectory from the factor (1 − 2η).
Instructions
Optimization here means iteratively lowering a loss. On f(x) = x² the gradient is 2x, so each step multiplies x by (1 − 2η). Starting at x₀ = 4 with η = 0.1 the factor is 0.8: x₁ = 3.2, x₂ = 2.56, x₃ = 2.048, and in general x_k = 4·0.8^k. Trace by hand first, then confirm with a short Python loop. Note η = 0.5 gives factor 0 — one-step convergence on this pure quadratic.
Practice & Prediction Log
For f(x) = x² with x₀ = 4: (a) write x_k in closed form; (b) compute x₁, x₂, x₃ and x₁₀ for η = 0.1; (c) for f(x) = (a/2)x² the factor is (1 − ηa) — derive the stability condition |1 − ηa| < 1 and the learning-rate bound it implies.
x_k = 4·(1−2η)^k; 0.8^10 ≈ 0.1074 so x₁₀ ≈ 0.4295. The bound: −1 < 1 − ηa < 1 → 0 < η < 2/a; for x², a = 2 so η < 1.
(a) x_k = 4·(1−2η)^k. (b) η = 0.1: x₁ = 3.2, x₂ = 2.56, x₃ = 2.048, x₁₀ ≈ 0.4295. (c) |1 − ηa| < 1 ⇔ 0 < η < 2/a; for f = x² (a = 2) the bound is η < 1 — and η = 0.5 lands exactly on the minimum in one step.
Closed form derived, all four iterates correct, and the η < 2/a condition stated with its a = 2 specialization.
Authoritative Source Resources & Provenance
Dive into Deep Learning (d2l.ai)
Relevant Section / Range: Chapter Optimization Algorithms: Gradient Descent
Verified Version & Date: Current online edition (Verified: 2026-10-30)
Primary open reference tying evaluation discipline (model selection, underfitting, overfitting) and optimization algorithms (gradient descent, minibatch) to the rest of the curriculum.
Linear algebra, calculus, and probability at the Days 19-28 level.
All required formulas and worked numbers are reproduced as original explanations; the book is further reading.
Compared Candidate Sources & Selection Analysis (2)
Detailed treatment of optimization difficulties (chapter 8) including saddle points and plateaus; all-rights-reserved text kept as a linking-only alternative.
Strong practitioner framing, but organized around the fastai library and video lessons rather than self-contained mathematics.
Gradient Descent, How Neural Networks Learn
Relevant Section / Range: Neural networks chapter 2: the gradient and the descent update
Verified Version & Date: Neural networks series chapter 2, 2017 (Verified: 2026-10-30)
Visual intuition connecting the gradient vector to steepest descent; complements d2l.ai's worked equations for the optimization day.
High-school functions and slope intuition.
The lesson contains complete derivations and numerical traces; the video is optional visual intuition.
Compared Candidate Sources & Selection Analysis (2)
A detailed optimization text, kept as an advanced alternative; its scope exceeds a single study day.
Rigorous discussion of saddle points and plateaus in high-dimensional optimization; kept as a linking-only alternative.