From characters to bytes: code points and UTF-8
Agent-Assisted Engineering Check
Prerequisites
Complete Day 36. If blocked, use the Day 35 recovery link for probability and Python review.
Learning Objective
Distinguish a visible character, its Unicode code point, and its UTF-8 byte sequence; predict byte counts before measuring; record a prediction-versus-observation comparison for empty, ASCII, accented, CJK, and emoji text.
Instructions
A character is what a reader sees; a code point is its Unicode number (written U+XXXX); an encoding turns code points into bytes. Required work: predict then run python3 -c "s='猫aé😀'; print(len(s), [hex(ord(c)) for c in s], list(s.encode('utf-8')))" and compare each prediction. Optional work: encode the same string as UTF-16 and compare byte counts.
Practice & Prediction Log
Predict len(s), the code points, and the UTF-8 byte list for s='猫aé😀' and for the empty string ''. Then verify in Python and explain why the byte count differs from the code-point count.
UTF-8 uses 1 byte for ASCII, 2 bytes for most accented Latin, 3 bytes for CJK, and 4 bytes for most emoji. ord() returns the code point; .encode('utf-8') returns the bytes.
s='猫aé😀' has 4 code points — U+732B (猫), U+0061 (a), U+00E9 (é), U+1F600 (😀) — but 10 UTF-8 bytes: 猫→[231,140,171], a→[97], é→[195,169], 😀→[240,159,152,128]. The empty string has 0 code points and 0 bytes. UTF-8 length depends on the code-point range, so character count and byte count generally differ.
Your written predictions match the observed ord() and encode() outputs for all probe strings (including empty), and you can state the rule mapping code-point ranges to UTF-8 byte counts.
Authoritative Source Resources & Provenance
Python Unicode HOWTO
Relevant Section / Range: Sections: Definitions of Character, Code Point, and Encodings (UTF-8)
Verified Version & Date: Python 3.12 documentation (Verified: 2026-10-30)
Normative reference for Python 3 code points, encodings, and variable-length UTF-8 byte behavior used in every tokenization step.
Basic Python strings and byte literals
The lesson defines every required Unicode concept locally and all commands run offline.
Compared Candidate Sources & Selection Analysis (2)
Exhaustive normative standard but far too deep for a first contact; the Python HOWTO teaches the same code point/encoding distinction in executable form.
Strong motivation and history but pre-Python-3 and not runnable; retained as optional background reading only.
minbpe: Minimal Byte-Pair Encoding tokenizer (Let's build the GPT Tokenizer)
Relevant Section / Range: README exercise flow and minbpe/base.py; lecture segment on Unicode code points vs raw bytes
Verified Version & Date: commit 1acefe9 (Verified: 2026-10-30)
Standard teaching reference connecting Unicode code points, UTF-8 bytes, and subword token ids for GPT-style tokenizers.
Basic Python and strings
The lesson re-implements the same algorithms locally in pure Python with verified outputs.
Compared Candidate Sources & Selection Analysis (3)
Production BPE with Rust core and regex pre-tokenization; selected as the production contrast source rather than the primary teaching implementation.
High-quality BPE coverage but frames tokenization through library APIs; minbpe exposes the raw merge loop for hand tracing.
Industrial tokenizer trainer supporting BPE and unigram models; C++ core hides the teaching mechanics.