Counting bigrams: from token stream to count table
Agent-Assisted Engineering Check
Prerequisites
Complete Day 37. If blocked, use the Day 35 recovery link for probability review.
Learning Objective
Turn a token stream into a bigram count table C(w_{i-1}, w_i), separate context totals from pair counts, and verify every count against a manual trace.
Instructions
A bigram is two consecutive tokens. Required work: slide a length-2 window over the stream 'the cat sat the cat ran the dog sat' (the three train lines joined into one 9-token stream, as the text_tools train file behaves) and tally each adjacent pair with Counter(zip(words, words[1:])). An empty string '' yields an empty bigram dictionary; Unicode strings like 'é猫' follow the identical pairing logic. Optional work: repeat on the val split and compare sparsity.
Practice & Prediction Log
Write the 8 adjacent pairs and the full count table before running any code. Then compute character-level bigrams of 'banana' the same way and record both tables.
Stream pairs: (the,cat), (cat,sat), (sat,the), (the,cat), (cat,ran), (ran,the), (the,dog), (dog,sat). For 'banana': (b,a), (a,n), (n,a), (a,n), (n,a).
Stream counts — 8 instances, 7 types: (the,cat)=2, (cat,sat)=1, (sat,the)=1, (cat,ran)=1, (ran,the)=1, (the,dog)=1, (dog,sat)=1. Context totals (bigrams starting with each token): the=3, cat=2, sat=1, ran=1, dog=1. 'banana' chars: (a,n)=2, (n,a)=2, (b,a)=1. Empty string '' evaluates to empty count {}.
Your hand-written table matches Counter(zip(words, words[1:])) output exactly, and you can explain why context totals — not unigram counts — are the correct denominators for the next step.
Authoritative Source Resources & Provenance
makemore: Character-level language models on names.txt
Relevant Section / Range: makemore.py part 1 bigrams: counting consecutive character pairs into a frequency table
Verified Version & Date: Verified 2026-10-30
Standard minimal implementation of a count-based bigram language model on the same names.txt corpus family.
Basic Python and Day 37 token ids
The lesson builds the full count table locally in pure Python with verified counts.
Compared Candidate Sources & Selection Analysis (2)
More rigorous and comprehensive but assumes PyTorch fluency; retained as the systems-rigor companion source.
Strong Transformer training reference but beyond the count-based baseline scope; deferred to later pretraining days.
Speech and Language Processing, 3rd ed. draft, Chapter 3: N-gram Language Models
Relevant Section / Range: Sections 3.1-3.2: n-gram counts, Markov assumption, and maximum-likelihood estimation
Verified Version & Date: Draft of 2026-08-19 (Verified: 2026-10-30)
Standard reference text defining n-gram models, count normalization, and the chain-rule decomposition used in this lesson.
Basic probability
All definitions and worked count tables are included in the lesson itself.
Compared Candidate Sources & Selection Analysis (2)
Detailed smoothing survey but far beyond the add-one scope of a first n-gram day; optional further reading.
Solid academic lecture treatment but less current and less integrated with executable count tables than the SLP3 draft.