Day 26: Random Variables, Expectation, Sampling, and Maximum Likelihood

01 120 min

Random variables and probability distributions

Assistance allowed only after prediction and observed evidence are recorded.

Prerequisites

Complete Day 25 (probability foundations); use the listed recovery link (Day 21) if blocked.

Learning Objective

Define a discrete random variable as a function from outcomes to numbers, validate a probability distribution table, and connect it to categorical distributions over tokens.

Instructions

A random variable $X$ maps each outcome of the sample space to a number; its distribution lists $P(X=x)$ with $\sum_x P(X=x)=1$. For a fair die, $X$ is uniform on $\{1,\dots,6\}$. Required work: compute $E[X]$ for the die, then validate the token-type distribution $p=[0.5,0.25,0.125,0.125]$ over values $\{0,1,2,3\}$.

Practice & Prediction Log

(a) Compute $E[X]$ for a fair die and explain why a non-integer expectation does not contradict the outcomes. (b) For $p=[0.5,0.25,0.125,0.125]$ over $\{0,1,2,3\}$, verify $\sum p = 1$ and compute $P(X\ge 2)$. (c) Interpret $p$ as a categorical distribution over four token types: which value is the most likely token?

Mark as Completed

Independent Mastery Criterion

You validate the normalization $\sum p = 1$, compute $P(X\ge2)=0.25$, and explain expectation as an average distinct from any single outcome.

Authoritative Source Resources & Provenance

linking CC BY-NC-SA 4.0 noncommercial share-alike

Introductory Statistics 2e

Author: OpenStax, Rice University (Barbara Illowsky, Susan Dean) | Original Source Link ↗

Relevant Section / Range: Chapter 4: Discrete Random Variables (probability distribution functions)

Verified Version & Date: 2e web edition (Verified: 2026-10-30)

Selection Rationale:

Open textbook treatment of discrete random variables and distribution tables.

Source Prerequisites:

High-school algebra; Day 25 probability foundations

Safe Fallback (Offline Resilience):

All required definitions, tables, and worked answers are reproduced locally in this lesson.

Compared Candidate Sources & Selection Analysis (3)

MIT OCW 6.041 Probabilistic Systems Analysis ↗ compared_deferred [CC BY-NC-SA 4.0]

Rigorous but paced for a full graduate semester; exceeds the depth needed for Days 25-27 training-objective foundations.

Khan Academy Statistics and Probability ↗ compared_alternative [CC BY-NC-SA 3.0]

Accessible video-first coverage; noncommercial/share-alike terms and video pacing make it a supplementary rather than primary link.

Harvard Stat 110 Introduction to Probability (Blitzstein & Hwang) ↗ compared_alternative [CRC Press copyright (book); free lecture videos]

Strong intuition-building course; the printed textbook is paywalled, so it complements rather than replaces the openly licensed OpenStax text.

linking Stanford copyright; public course notes No text extraction or redistribution permitted

CS229 Review of Probability Theory

Author: Stanford CS229 course staff (Arian Maleki, Tom Do) | Original Source Link ↗

Relevant Section / Range: Section 2.4-2.5: Random variables, distributions, expectation

Verified Version & Date: 2020 public course notes (Verified: 2026-10-30)

Selection Rationale:

ML-oriented definition of random variables bridging to categorical token distributions.

Source Prerequisites:

Multivariate calculus notation exposure

Safe Fallback (Offline Resilience):

Self-contained definitions and derivations in activity instructions and offline guide.

Compared Candidate Sources & Selection Analysis (2)

OpenStax Introductory Statistics 2e ↗ compared_alternative [CC BY-NC-SA 4.0]

Co-selected: provides the gentler beginner narrative; CS229 provides the ML-oriented notation bridge.

MIT OCW 18.05 Introduction to Probability and Statistics ↗ compared_alternative [CC BY-NC-SA 4.0]

Strong full-course treatment with noncommercial/share-alike terms; heavier than the compact CS229 reference for this three-day block.

02 120 min

Expectation, variance, and worked distribution tables

Independent Mastery Check (no AI assistance).

Prerequisites

Complete the previous activity; use the listed recovery link if blocked.

Learning Objective

Compute $E[X]$, $E[X^2]$, $\mathrm{Var}(X)=E[X^2]-E[X]^2$, and the standard deviation for discrete distributions without assistance.

Instructions

Work each row of the distribution table before combining them. Required work: the four-type distribution below, a fair die, and a Bernoulli(0.25) variable. Record predicted values before computing and keep exact fractions where possible.

Practice & Prediction Log

For $p=[0.5,0.25,0.125,0.125]$ over $\{0,1,2,3\}$ compute $E[X]$, $E[X^2]$, $\mathrm{Var}(X)$, $\sigma_X$. Then compute $\mathrm{Var}$ for a fair die ($E[X^2]=91/6$) and for $\mathrm{Bernoulli}(0.25)$.

Mark as Completed

Independent Mastery Criterion

All values match the reference fractions/decimals computed unaided; variance is reported in squared units and $\sigma$ in the original units.

Authoritative Source Resources & Provenance

linking CC BY-NC-SA 4.0 noncommercial share-alike

Introductory Statistics 2e

Author: OpenStax, Rice University (Barbara Illowsky, Susan Dean) | Original Source Link ↗

Relevant Section / Range: Chapter 4: Discrete Random Variables (mean, expected value, standard deviation)

Verified Version & Date: 2e web edition (Verified: 2026-10-30)

Selection Rationale:

Step-by-step open examples of expected value and variance from distribution tables.

Source Prerequisites:

High-school algebra; Day 25 probability foundations

Safe Fallback (Offline Resilience):

All required definitions, tables, and worked answers are reproduced locally in this lesson.

Compared Candidate Sources & Selection Analysis (3)

MIT OCW 6.041 Probabilistic Systems Analysis ↗ compared_deferred [CC BY-NC-SA 4.0]

Rigorous but paced for a full graduate semester; exceeds the depth needed for Days 25-27 training-objective foundations.

Khan Academy Statistics and Probability ↗ compared_alternative [CC BY-NC-SA 3.0]

Accessible video-first coverage; noncommercial/share-alike terms and video pacing make it a supplementary rather than primary link.

Harvard Stat 110 Introduction to Probability (Blitzstein & Hwang) ↗ compared_alternative [CRC Press copyright (book); free lecture videos]

Strong intuition-building course; the printed textbook is paywalled, so it complements rather than replaces the openly licensed OpenStax text.

linking Stanford copyright; public course notes No text extraction or redistribution permitted

CS229 Review of Probability Theory

Author: Stanford CS229 course staff (Arian Maleki, Tom Do) | Original Source Link ↗

Relevant Section / Range: Section 2.5-2.6: Expectation, variance, common discrete distributions

Verified Version & Date: 2020 public course notes (Verified: 2026-10-30)

Selection Rationale:

Concise reference for Var(X)=E[X^2]-E[X]^2 and Bernoulli variance used in ML derivations.

Source Prerequisites:

Multivariate calculus notation exposure

Safe Fallback (Offline Resilience):

Self-contained definitions and derivations in activity instructions and offline guide.

Compared Candidate Sources & Selection Analysis (2)

OpenStax Introductory Statistics 2e ↗ compared_alternative [CC BY-NC-SA 4.0]

Co-selected: provides the gentler beginner narrative; CS229 provides the ML-oriented notation bridge.

MIT OCW 18.05 Introduction to Probability and Statistics ↗ compared_alternative [CC BY-NC-SA 4.0]

Strong full-course treatment with noncommercial/share-alike terms; heavier than the compact CS229 reference for this three-day block.

03 120 min

Sampling, empirical distributions, and maximum likelihood

Assistance allowed only after prediction and observed evidence are recorded.

Prerequisites

Complete the previous activity; use the listed recovery link if blocked.

Learning Objective

Build an empirical distribution with seeded sampling, compare it with the true distribution using the standard error $\sigma/\sqrt n$, and derive maximum-likelihood estimates for Bernoulli and unigram models, including the zero-count failure.

Instructions

Run: python3 -c "import random, collections; random.seed(26); s=[random.randint(1,6) for _ in range(100)]; print(round(sum(s)/100,4), collections.Counter(s))". The sample mean of a die has standard error $\sigma/\sqrt n \approx 1.7078/10 \approx 0.1708$. For a coin with 36 heads in 60 flips the likelihood is $p^{36}(1-p)^{24}$, maximized at $\hat p = 36/60$. For unigrams, MLE sets $\hat p(w) = \mathrm{count}(w)/N$.

Practice & Prediction Log

(a) Report the seeded sample mean and one empirical frequency that deviates from $1/6$. (b) Derive $\hat p$ for 36 heads in 60 flips by setting the log-likelihood derivative to zero. (c) In the corpus "the cat sat on the mat the cat ran" ($N=9$, six observed types), give the unigram MLE for 'the' and for 'dog'. What breaks if a model assigns 0 to a held-out token?

Mark as Completed

Independent Mastery Criterion

You report the seeded numbers honestly (mean $3.47$, the face-3 shortfall), derive $\hat p = 0.6$ from the log-likelihood, and explain the zero-count failure plus the add-1 fix.

Authoritative Source Resources & Provenance

linking CC BY-NC-SA 4.0 noncommercial share-alike

Introductory Statistics 2e

Author: OpenStax, Rice University (Barbara Illowsky, Susan Dean) | Original Source Link ↗

Relevant Section / Range: Chapter 1: Sampling and Data (sampling, variation); Chapter 4 (estimating parameters)

Verified Version & Date: 2e web edition (Verified: 2026-10-30)

Selection Rationale:

Open coverage of sampling variability and estimation that motivates the MLE and standard-error work here.

Source Prerequisites:

High-school algebra; Day 25 probability foundations

Safe Fallback (Offline Resilience):

All required definitions, simulations, and worked answers are reproduced locally in this lesson.

Compared Candidate Sources & Selection Analysis (3)

MIT OCW 6.041 Probabilistic Systems Analysis ↗ compared_deferred [CC BY-NC-SA 4.0]

Rigorous but paced for a full graduate semester; exceeds the depth needed for Days 25-27 training-objective foundations.

Khan Academy Statistics and Probability ↗ compared_alternative [CC BY-NC-SA 3.0]

Accessible video-first coverage; noncommercial/share-alike terms and video pacing make it a supplementary rather than primary link.

Harvard Stat 110 Introduction to Probability (Blitzstein & Hwang) ↗ compared_alternative [CRC Press copyright (book); free lecture videos]

Strong intuition-building course; the printed textbook is paywalled, so it complements rather than replaces the openly licensed OpenStax text.

linking PSF License Version 2

Python 3 random: Generate pseudo-random numbers

Author: Python Software Foundation | Original Source Link ↗

Relevant Section / Range: random.seed, random.randint, random.sample, random.choices

Verified Version & Date: Python 3.12 documentation (Verified: 2026-10-30)

Selection Rationale:

Normative reference for the seeded draws used to build empirical distributions.

Source Prerequisites:

Basic Python syntax

Safe Fallback (Offline Resilience):

Use the installed interpreter's help(random) and the included deterministic commands.

Compared Candidate Sources & Selection Analysis (2)

NumPy random Generator documentation ↗ compared_deferred [BSD 3-Clause]

Faster vectorized sampling but adds a third-party dependency before the tensor track; deferred to the models-and-training days.

Python secrets module documentation ↗ compared_rejected_for_days_25_27 [PSF License Version 2]

Cryptographically secure randomness is unnecessary for reproducible statistical pedagogy and obscures the seed-based determinism the exercises require.

04 120 min

Independent mastery check: estimation and sampling

Independent Mastery Check (no AI assistance).

Prerequisites

Complete the previous activity; use the listed recovery link if blocked.

Learning Objective

Solve unseen estimation problems unaided and state the statistical assumptions behind each result.

Instructions

No AI or references during this check; arithmetic only. Record predictions first, solve all four items, then review and pick a recovery link for misses. Preserve the first unaided attempt unchanged; afterwards, error review and any assisted recovery are a distinct phase, and a missed idea counts as remediated only after a fresh equivalent unaided check whose answer is not shown in advance. Second check (only after remediation; unaided, no hints): (a) For $p=[0.2,0.3,0.4,0.1]$ over $\{0,1,2,3\}$ compute $E[X]$ and $\mathrm{Var}(X)$. (b) Compute $P(\text{at least one 6 in 3 rolls})$. (c) 9 heads in 15 flips: give $\hat p$ and its approximate standard error. (d) In one sentence, explain why a finite-sample proportion need not equal the true $p$. This card gives no answer for the second check: grade it by the same standard as the mastery criterion (correct values, i.i.d. and finite-sample assumptions stated), and verify only after finishing, by a second method ($\mathrm{Var}=E[(X-E[X])^2]$ directly, or enumerating all $6^3$ outcomes). Remediate a missed (a) with day-26-act-01 (valid distribution tables) and day-26-act-02 (expectation, variance), (b) with day-25-act-02 (independence) and day-25-act-01 (counting), and (c)-(d) with day-26-act-03 (sampling, standard error, maximum likelihood).

Practice & Prediction Log

(a) For $p=[0.1,0.2,0.4,0.3]$ over $\{0,1,2,3\}$ compute $E[X]$ and $\mathrm{Var}(X)$. (b) Compute $P(\text{at least one 6 in 4 rolls})$. (c) 14 heads in 20 flips: give $\hat p$ and one reason the true $p$ could still differ. (d) Explain in one sentence why a finite sample mean need not equal $E[X]$.

Mark as Completed

Independent Mastery Criterion

Unaided values match $1.9$, $0.89$, $0.5177$, $0.7$; the i.i.d. and finite-sample assumptions are stated; misses route to a recovery link, and each missed idea is remediated only by a fresh equivalent unaided check.

Authoritative Source Resources & Provenance

linking CC BY-NC-SA 4.0 noncommercial share-alike

Introductory Statistics 2e

Author: OpenStax, Rice University (Barbara Illowsky, Susan Dean) | Original Source Link ↗

Relevant Section / Range: Chapters 1 and 4 review: sampling, distributions, expectation, variance

Verified Version & Date: 2e web edition (Verified: 2026-10-30)

Selection Rationale:

Open review coverage aligned with this day's assessment scope.

Source Prerequisites:

High-school algebra; Day 25 probability foundations

Safe Fallback (Offline Resilience):

The assessment is self-contained with original items and worked reference answers.

Compared Candidate Sources & Selection Analysis (3)

MIT OCW 6.041 Probabilistic Systems Analysis ↗ compared_deferred [CC BY-NC-SA 4.0]

Rigorous but paced for a full graduate semester; exceeds the depth needed for Days 25-27 training-objective foundations.

Khan Academy Statistics and Probability ↗ compared_alternative [CC BY-NC-SA 3.0]

Accessible video-first coverage; noncommercial/share-alike terms and video pacing make it a supplementary rather than primary link.

Harvard Stat 110 Introduction to Probability (Blitzstein & Hwang) ↗ compared_alternative [CRC Press copyright (book); free lecture videos]

Strong intuition-building course; the printed textbook is paywalled, so it complements rather than replaces the openly licensed OpenStax text.

linking Stanford copyright; public course notes No text extraction or redistribution permitted

CS229 Review of Probability Theory

Author: Stanford CS229 course staff (Arian Maleki, Tom Do) | Original Source Link ↗

Relevant Section / Range: Sections 2.4-3: random variables, expectation, variance, maximum likelihood

Verified Version & Date: 2020 public course notes (Verified: 2026-10-30)

Selection Rationale:

Reference for estimation and MLE concepts this assessment checks without assistance.

Source Prerequisites:

Multivariate calculus notation exposure

Safe Fallback (Offline Resilience):

The assessment is self-contained with original items and worked reference answers.

Compared Candidate Sources & Selection Analysis (2)

OpenStax Introductory Statistics 2e ↗ compared_alternative [CC BY-NC-SA 4.0]

Co-selected: provides the gentler beginner narrative; CS229 provides the ML-oriented notation bridge.

MIT OCW 18.05 Introduction to Probability and Statistics ↗ compared_alternative [CC BY-NC-SA 4.0]

Strong full-course treatment with noncommercial/share-alike terms; heavier than the compact CS229 reference for this three-day block.