第 37 天：從文字到詞元 — Unicode、位元組編碼與詞彙表

01 120 分鐘

從字元到位元組：碼位與 UTF-8

Agent 輔助工程檢查

前置要求

完成第 36 天。如受阻，使用第 35 天的回退連結複習機率與 Python。

學習目標

區分可見字元、其 Unicode 碼位與其 UTF-8 位元組序列；先預測位元組數再測量；針對空字串、ASCII、帶重音字元、中日韓字元與 emoji 記錄預測與觀測的對照。

操作指引

字元是讀者看到的形體；碼位是其 Unicode 編號（記作 U+XXXX）；編碼把碼位轉換成位元組序列。必做工作：先預測，再執行 python3 -c "s='貓aé😀'; print(len(s), [hex(ord(c)) for c in s], list(s.encode('utf-8')))"，逐項對照預測。選做工作：把同一字串用 UTF-16 編碼並比較位元組數。

自主實作與記錄

預測 s='貓aé😀' 與空字串 '' 的 len(s)、碼位序列與 UTF-8 位元組列表；隨後在 Python 中驗證，並解釋為什麼位元組數與碼位數不同。

標記為已完成

獨立掌握標準

對包括空字串在內的全體探測字串，你的書面預測與 ord()、encode() 的觀測輸出一致，並能說明碼位範圍到位元組數的對應規則。

權威來源資源與權利證明

linking PSF License Version 2

Python Unicode HOWTO

作者 / 機構: Python Software Foundation | 原始出處連結 ↗

對應章節 / 時間戳: Sections: Definitions of Character, Code Point, and Encodings (UTF-8)

驗證版本與日期: Python 3.12 documentation (Verified: 2026-10-30)

入選理由:

Normative reference for Python 3 code points, encodings, and variable-length UTF-8 byte behavior used in every tokenization step.

來源前置條件:

Basic Python strings and byte literals

安全回退方案（離線彈性）:

The lesson defines every required Unicode concept locally and all commands run offline.

對比候選資源與選型分析 (2)

Unicode Standard (unicode.org) official documentation ↗ compared_alternative [Unicode License Agreement / Terms of Use]

Exhaustive normative standard but far too deep for a first contact; the Python HOWTO teaches the same code point/encoding distinction in executable form.

Strong motivation and history but pre-Python-3 and not runnable; retained as optional background reading only.

linking MIT License (repository) / Standard YouTube License (video, subtitle rights unknown) Video subtitles have unknown distribution rights and are not imported

minbpe: Minimal Byte-Pair Encoding tokenizer (Let's build the GPT Tokenizer)

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: README exercise flow and minbpe/base.py; lecture segment on Unicode code points vs raw bytes

驗證版本與日期: commit 1acefe9 (Verified: 2026-10-30)

入選理由:

Standard teaching reference connecting Unicode code points, UTF-8 bytes, and subword token ids for GPT-style tokenizers.

來源前置條件:

Basic Python and strings

安全回退方案（離線彈性）:

The lesson re-implements the same algorithms locally in pure Python with verified outputs.

對比候選資源與選型分析 (3)

OpenAI tiktoken ↗ compared_alternative [MIT]

Production BPE with Rust core and regex pre-tokenization; selected as the production contrast source rather than the primary teaching implementation.

Hugging Face tokenizers + NLP course chapter 6 ↗ compared_alternative [Apache-2.0 (library) / CC BY-NC-SA? (course text varies)]

High-quality BPE coverage but frames tokenization through library APIs; minbpe exposes the raw merge loop for hand tracing.

Google SentencePiece ↗ compared_alternative [Apache-2.0]

Industrial tokenizer trainer supporting BPE and unigram models; C++ core hides the teaching mechanics.

02 120 分鐘

詞彙表與詞元 ID 映射

獨立掌握檢查（無 AI 輔助）

前置要求

完成上一活動；如受阻，使用列出的回退連結。

學習目標

從字元清單構建排序後的詞彙表；為未知輸入制定明確的策略（<unk> vs 位元組級別全覆蓋）；手算編碼與解碼以驗證可逆無損。

操作指引

詞元化器需要兩張表：stoi（符號→ID）與 itos（ID→符號）。必做工作：按排序順序為訓練文本中不同的符號分配 ID 0..V-1，把 len(vocab) 保留為的 ID，然後對探測字串編碼並解碼。選做工作：與不需要的 256 項位元組級詞彙表做對比。

自主實作與記錄

對訓練文本 'banana'：建立排序後的字元詞彙表，編碼 'banana' 和 'cab'，解碼第一個結果，並用編碼 'banana!'。然後把 'é貓' 按 UTF-8 位元組編碼，說明為什麼位元組級輸入永遠不會落入。

標記為已完成

獨立掌握標準

給定任意短訓練文本，你能寫出詞彙表、手工正確地編碼/解碼探測字串，並說明何時需要 <unk>、何時位元組級覆蓋使其不再必要。

權威來源資源與權利證明

linking MIT License (repository) / Standard YouTube License (video, subtitle rights unknown) Video subtitles have unknown distribution rights and are not imported

minbpe: Minimal Byte-Pair Encoding tokenizer (Let's build the GPT Tokenizer)

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: minbpe/base.py: vocab dict, ids<->bytes mapping, encode/decode contract

驗證版本與日期: commit 1acefe9 (Verified: 2026-10-30)

入選理由:

Shows the exact vocab, merges, and encode/decode invariants a teaching tokenizer must satisfy.

來源前置條件:

Basic Python dicts and lists

安全回退方案（離線彈性）:

All mapping exercises are re-implemented locally with verified expected ids.

對比候選資源與選型分析 (3)

OpenAI tiktoken ↗ compared_alternative [MIT]

Production BPE with Rust core and regex pre-tokenization; selected as the production contrast source rather than the primary teaching implementation.

Hugging Face tokenizers + NLP course chapter 6 ↗ compared_alternative [Apache-2.0 (library) / CC BY-NC-SA? (course text varies)]

High-quality BPE coverage but frames tokenization through library APIs; minbpe exposes the raw merge loop for hand tracing.

Google SentencePiece ↗ compared_alternative [Apache-2.0]

Industrial tokenizer trainer supporting BPE and unigram models; C++ core hides the teaching mechanics.

linking MIT License

tiktoken: Fast byte-pair encoding tokenizer used by OpenAI models

作者 / 機構: OpenAI | 原始出處連結 ↗

對應章節 / 時間戳: README: encoding.get_encoding("cl100k_base"), encode/decode API, special token policy

驗證版本與日期: Verified 2026-10-30

入選理由:

Production reference that contrasts a teaching tokenizer's explicit vocab dict with industrial byte-level BPE ranks and special tokens.

來源前置條件:

Basic Python

安全回退方案（離線彈性）:

The lesson's vocabulary/id exercises run entirely on a local Python dict with fixed expected values.

對比候選資源與選型分析 (2)

Hugging Face tokenizers library ↗ compared_alternative [Apache-2.0]

Broad production tokenizer framework with training APIs; tiktoken is a cleaner contrast for GPT-style compatibility claims.

Google SentencePiece ↗ compared_alternative [Apache-2.0]

Widely used trainer/tokenizer for open models; treated as further production context rather than the contrast reference.

03 120 分鐘

確定性的 train/val/test 劃分與出處記錄

Agent 輔助工程檢查

前置要求

完成上一活動；如受阻，使用列出的回退連結。

學習目標

重現 text_tools 的確定性劃分產物並閱讀清單（manifest），使下游所有模型只在 train 上擬合計數、在 val 上做選擇、完全不接觸 test。

操作指引

必做工作：執行 python3 projects/text_tools/cli.py --input data/names.txt --split --output-dir /tmp/lm-data（預設值：seed 42，比例 0.8/0.1/0.1），然後打開 /tmp/lm-data/manifest.json，找到 split_counts、split_sha256 與 leakage_check。換第二個輸出目錄重跑一次並確認雜湊一致。選做工作：用 --seed 99 重複，觀察計數不變而成員改變。

自主實作與記錄

先預測 32,033 行語料的 train/val/test 行數再執行驗證。對一個 10 行檔案預測同樣的劃分。指出清單中哪些欄位證明確定性、以及沒有任何索引同時出現在兩個劃分裡。

標記為已完成

獨立掌握標準

你能重新產生產物，引用精確的劃分計數與證明確定性、互不相交的清單欄位，並說明為什麼計數、詞彙表與合併表只能在 train 上擬合。

權威來源資源與權利證明

linking MIT License (repository) / Standard YouTube License (video, subtitle rights unknown) Video subtitles have unknown distribution rights and are not imported

makemore: Character-level language models on names.txt

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: README and notebook part 1: names.txt corpus, 90% train / 5% dev / 5% test split rationale

驗證版本與日期: Verified 2026-10-30

入選理由:

Uses the same data/names.txt corpus family and demonstrates why model fitting must see only the training split while dev/test remain held out.

來源前置條件:

Basic Python

安全回退方案（離線彈性）:

Local projects/text_tools/cli.py produces the deterministic splits and manifest offline.

對比候選資源與選型分析 (2)

Stanford CS336: Language Modeling from Scratch ↗ compared_alternative [No license declared (the repository has no LICENSE file, so all rights are reserved by default)]

More rigorous and comprehensive but assumes PyTorch fluency; retained as the systems-rigor companion source.

Karpathy nanoGPT ↗ compared_deferred [MIT]

Strong Transformer training reference but beyond the count-based baseline scope; deferred to later pretraining days.

linking MIT

CS336 Assignment 1: Basics (Spring 2025 tagged release)

作者 / 機構: Stanford University CS336 Staff | 原始出處連結 ↗

對應章節 / 時間戳: README setup; tests/adapters.py; tests/test_optimizer.py; tests/test_nn_utils.py; tests/test_serialization.py

驗證版本與日期: spring2025 tag; README, tests and MIT LICENSE checked 2026-11-01

入選理由:

Lead assignment-driven source for pretraining component practice and tests. nanoGPT remains a compact historical code-reading companion, not the current recommended training curriculum.

來源前置條件:

Python systems programming

安全回退方案（離線彈性）:

Local projects/text_tools deterministic artifacts provide the complete pipeline offline.

對比候選資源與選型分析 (2)

Karpathy nanoGPT ↗ compared_alternative [MIT]

Simpler code layout, useful complementary reference for minimal PyTorch Transformer.

Hugging Face Transformers Course ↗ compared_alternative [Apache-2.0]

High-level API focus rather than building from raw tensor fundamentals.

04 120 分鐘

獨立掌握檢查：位元組、ID 與劃分證據

獨立掌握檢查（無 AI 輔助）

前置要求

完成前面所有活動；如受阻，使用列出的回退連結。

學習目標

在無輔助的情況下重現碼位/位元組計算、詞彙表編碼/解碼、劃分計數與畸形輸入策略，然後為每個失誤記錄誠實的回退計劃。

操作指引

本檢查期間不要查看答案或使用 AI。必做工作：先在草稿日誌中寫下全部四個預測答案，再用 Python 與 CLI 逐一驗證。選做工作：舉一個對嘈雜語料合理使用 --errors ignore 的例子。

自主實作與記錄

閉卷作答：(a) 'é貓' 有多少個碼位、多少個 UTF-8 位元組？(b) 詞彙表 {a:0,b:1,c:2} 下 encode('cab') 是多少？(c) 10 行按 0.8/0.1/0.1 如何劃分？(d) 含非法位元組 0xFF 的檔案在 --errors strict 下的結束代碼是多少？replace 下會出現哪個字元？

標記為已完成

獨立掌握標準

四個預測全部在無輔助情況下與觀測輸出一致；每個失誤都連同用於修復的回退連結（第 35 天複習或對應的第 37 天活動）一併記錄。

權威來源資源與權利證明

linking PSF License Version 2

Python Unicode HOWTO

作者 / 機構: Python Software Foundation | 原始出處連結 ↗

對應章節 / 時間戳: Sections: Encodings, UTF-8 rules, and error handling strategies

驗證版本與日期: Python 3.12 documentation (Verified: 2026-10-30)

入選理由:

Normative reference for the strict/replace decoding behavior verified in the mastery check.

來源前置條件:

Basic Python strings and byte literals

安全回退方案（離線彈性）:

All mastery-check answers are computable with a local Python interpreter.

對比候選資源與選型分析 (2)

Unicode Standard (unicode.org) official documentation ↗ compared_alternative [Unicode License Agreement / Terms of Use]

Exhaustive normative standard but far too deep for a first contact; the Python HOWTO teaches the same code point/encoding distinction in executable form.

Strong motivation and history but pre-Python-3 and not runnable; retained as optional background reading only.

linking MIT License (repository) / Standard YouTube License (video, subtitle rights unknown) Video subtitles have unknown distribution rights and are not imported

makemore: Character-level language models on names.txt

作者 / 機構: Andrej Karpathy | 原始出處連結 ↗

對應章節 / 時間戳: Corpus-to-tensor pipeline: stoi/itos lookup tables built from the training split only

驗證版本與日期: Verified 2026-10-30

入選理由:

Reinforces that vocabulary tables and counts are fit on training data only, the rule assessed in this check.

來源前置條件:

Basic Python

安全回退方案（離線彈性）:

The local deterministic artifacts in projects/text_tools provide all required evidence offline.

對比候選資源與選型分析 (2)

Stanford CS336: Language Modeling from Scratch ↗ compared_alternative [No license declared (the repository has no LICENSE file, so all rights are reserved by default)]

More rigorous and comprehensive but assumes PyTorch fluency; retained as the systems-rigor companion source.

Karpathy nanoGPT ↗ compared_deferred [MIT]

Strong Transformer training reference but beyond the count-based baseline scope; deferred to later pretraining days.