从字符到字节：码位与 UTF-8
Agent 辅助工程检查
前置要求
完成第 36 天。如受阻，使用第 35 天的回退链接复习概率与 Python。
学习目标
区分可见字符、其 Unicode 码位与其 UTF-8 字节序列；先预测字节数再测量；针对空串、ASCII、带重音字符、中日韩字符与 emoji 记录预测与观测的对照。
操作指引
字符是读者看到的东西；码位是它的 Unicode 编号（记作 U+XXXX）；编码把码位变成字节序列。必做工作：先预测，再运行 python3 -c "s='猫aé😀'; print(len(s), [hex(ord(c)) for c in s], list(s.encode('utf-8')))"，逐项对照预测。选做工作：把同一字符串用 UTF-16 编码并比较字节数。
自主演练与记录
预测 s='猫aé😀' 与空字符串 '' 的 len(s)、码位序列与 UTF-8 字节列表；随后在 Python 中验证，并解释为什么字节数与码位数不同。
UTF-8 中 ASCII 占 1 字节、多数带重音拉丁字符占 2 字节、中日韩字符占 3 字节、多数 emoji 占 4 字节。ord() 返回码位，.encode('utf-8') 返回字节。
s='猫aé😀' 有 4 个码位——U+732B（猫）、U+0061（a）、U+00E9（é）、U+1F600（😀）——但 UTF-8 字节有 10 个：猫→[231,140,171]，a→[97]，é→[195,169]，😀→[240,159,152,128]。空字符串有 0 个码位、0 个字节。UTF-8 长度取决于码位所在范围，因此字符数与字节数一般不相等。
对包括空串在内的全部探测字符串，你的书面预测与 ord()、encode() 的观测输出一致，并能说出码位范围到 UTF-8 字节数的对应规则。
权威源资源与权利证明
Python Unicode HOWTO
对应章节 / 范围: Sections: Definitions of Character, Code Point, and Encodings (UTF-8)
验证版本与日期: Python 3.12 documentation (Verified: 2026-10-30)
Normative reference for Python 3 code points, encodings, and variable-length UTF-8 byte behavior used in every tokenization step.
Basic Python strings and byte literals
The lesson defines every required Unicode concept locally and all commands run offline.
对比候选资源与选型分析 (2)
Exhaustive normative standard but far too deep for a first contact; the Python HOWTO teaches the same code point/encoding distinction in executable form.
Strong motivation and history but pre-Python-3 and not runnable; retained as optional background reading only.
minbpe: Minimal Byte-Pair Encoding tokenizer (Let's build the GPT Tokenizer)
对应章节 / 范围: README exercise flow and minbpe/base.py; lecture segment on Unicode code points vs raw bytes
验证版本与日期: commit 1acefe9 (Verified: 2026-10-30)
Standard teaching reference connecting Unicode code points, UTF-8 bytes, and subword token ids for GPT-style tokenizers.
Basic Python and strings
The lesson re-implements the same algorithms locally in pure Python with verified outputs.
对比候选资源与选型分析 (3)
Production BPE with Rust core and regex pre-tokenization; selected as the production contrast source rather than the primary teaching implementation.
High-quality BPE coverage but frames tokenization through library APIs; minbpe exposes the raw merge loop for hand tracing.
Industrial tokenizer trainer supporting BPE and unigram models; C++ core hides the teaching mechanics.