13 / 90

Day 13: Text and data Learning Project

Turn the existing text_tools project into a reproducible corpus-to-artifact pipeline.

Self-Directed Learning & Python 480 active minutes (breaks and unattended runtime excluded)
01 120 min

Build the CLI core

Agent assistance is allowed only after prediction; inspect and test every generated diff.

Prerequisites

Days 01–12: strings, dictionaries, files, exceptions, tests, and terminal use.

Learning Objective

Implement or inspect projects/text_tools/core.py and cli.py for statistics, deterministic splits, a provenance manifest, and explicit encoding errors.

Instructions

Required work: create a private workspace for this exercise (WORKDIR=$(mktemp -d)), write a small corpus to $WORKDIR/corpus.txt, record a prediction, then run the complete invocation python3 projects/text_tools/cli.py --input "$WORKDIR/corpus.txt" --stats --json --split --output-dir "$WORKDIR/artifacts"; inspect the diff, and exercise --input, --stats, --split, --output-dir, --seed, --encoding, and --errors. Optional work: compare the Python documentation links.

Practice and prediction

Use a small corpus containing repeated tokens, CJK, accents, and an emoji. Predict character/token counts, then verify stats.json and manifest.json -- including that source_sha256 equals the SHA256 of the original input bytes (e.g. sha256sum "$WORKDIR/corpus.txt") and that input_decoding records the encoding, error handling, and newline policy.

Learning Project Submission & Evidence

0 / 4096 bytes
Declared Assistance Condition

Current Submitted Evidence Details

No submission recorded yet.
Independent mastery criterion

The learner can explain and run the real text_tools CLI, with prediction/evidence recorded and no claim that an agent-written diff proves independent understanding.

Source resources and provenance

linking PSF License Version 2

Python Standard Library: Built-in Types

Author: Python Software Foundation | Original Source Link ↗

Relevant Section / Range: str methods and Unicode semantics used by analyze_text (split, join, isalpha, isdigit)

Verified Version & Date: Python 3.12 documentation; verified 2026-10-30

Selection Rationale:

Primary reference for the string methods and Unicode semantics behind corpus statistics.

Source Prerequisites:

Study Days 01–12 Python fundamentals

Safe Fallback (Offline Resilience):

The lesson explains all required behavior locally and the reference implementation ships in the repository; pydoc mirrors the same documentation offline.

Compared Candidate Sources & Selection Analysis (2)
GNU coreutils wc/split ↗ compared_rejected_for_day13 [GPL-3.0-or-later]

Powerful system tools, but the learning target is implementing and testing the behavior in Python, not delegating to platform-specific utilities.

Python HOWTO: Unicode ↗ compared_alternative [PSF License Version 2]

Deeper Unicode background for the encoding edge cases; str method reference remains the primary doc.

linking PSF License Version 2

argparse — Parser for command-line options, arguments and subcommands

Author: Python Software Foundation | Original Source Link ↗

Relevant Section / Range: add_argument flags, choices, and type conversion used by cli.py

Verified Version & Date: Python 3.12 documentation; verified 2026-10-30

Selection Rationale:

Primary reference for the CLI flags the learner exercises on the real project.

Source Prerequisites:

Study Days 01–12 Python fundamentals

Safe Fallback (Offline Resilience):

argparse is part of the standard library; python3 -c "import argparse; help(argparse)" works fully offline.

Compared Candidate Sources & Selection Analysis (2)
Click package documentation ↗ compared_rejected_for_day13 [BSD-3-Clause]

Strong third-party CLI framework, but the project intentionally uses only the standard library so the exercise needs no extra dependency.

docopt ↗ compared_rejected_for_day13 [MIT]

Usage-string DSL for CLIs, but an external dependency is unjustified when argparse covers the project.

02 120 min

Independent core implementation

Independent Mastery Check: no AI, copied answer, or agent-generated code.

Prerequisites

Complete Day 13 Activity 1 and close all reference tabs.

Learning Objective

Without AI assistance, implement a bounded statistics or split function against the actual project interface.

Instructions

Required work: independently implement exactly one assessed interface -- either analyze_text(text) -> dict (statistics) or compute_splits(lines, train_ratio, val_ratio, test_ratio, seed) -> dict (partitioning) -- stating which you chose, then test it on a DISTINCT fresh unaided input whose result you predict first. Optional work: compare with the reference only after submission. Canonical worked outputs are public reference, never hidden evaluation.

Practice and prediction

Choose ONE assessed interface and declare it. Route A analyze_text(text) -> dict: predict the complete result for the input 'a a\n猫🙂' -- keys are total_chars, total_tokens, vocab_size, char_vocab_size, token_frequencies, char_frequencies, min_token_length, max_token_length, avg_token_length, and char_distribution. Route B compute_splits(lines, train_ratio, val_ratio, test_ratio, seed) -> dict: for ['a','b','c','d','e'] with seed 7 and ratios .6/.2/.2, predict counts and prove the index sets are pairwise disjoint. For your chosen route, also explain the distinctness rule it depends on (route B: equal text values still need index-level checks; route A: equal text values do not inflate vocabulary).

Independent mastery criterion

Independent code on a distinct fresh unaided input passes held-out behavior checks and the written explanation distinguishes index leakage, duplicate records, token/character counting, and evaluation limitations.

Source resources and provenance

original_explanation Apache-2.0 AND CC-BY-SA-4.0 share-alike

Text and Data Tools Learning Project

Author: Curriculum team (delivered in Ticket 16) | Original Source Link ↗

Relevant Section / Range: projects/text_tools/core.py: analyze_text and compute_splits interfaces

Verified Version & Date: Verified against the checked-in implementation on 2026-10-30

Selection Rationale:

The actual project whose interfaces the learner re-implements independently, keeping the exercise tied to real artifacts.

Source Prerequisites:

Study Days 01–12 Python fundamentals

Safe Fallback (Offline Resilience):

The implementation and its project manifest are committed in the repository and work offline.

Compared Candidate Sources & Selection Analysis (2)
External toy CLI tutorials ↗ compared_rejected_for_day13 [PSF License Version 2]

Generic tutorials duplicate earlier days and disconnect the exercise from the deterministic data artifacts later experiments reuse.

CPython Lib/string.py reference implementation ↗ compared_alternative [PSF License Version 2]

Production string-tooling internals are instructive but heavier than the project's focused interfaces.

03 120 min

Held-out behavior checks and debugging

Agent-assisted engineering check: direct the agent, inspect the diff, then run the held-out checks yourself.

Prerequisites

Have an independently implemented function and a clean temporary output directory.

Learning Objective

Diagnose real failures without changing the curriculum repository: missing input, malformed UTF-8, empty input, invalid ratios, and nondeterministic ordering.

Instructions

Required work: run each debugging case in a process-isolated temporary directory, capture exit status/stderr, and regenerate artifacts from held-out data. Optional work: ask an agent to suggest tests, then reject unsafe or tautological tests.

Practice and prediction

A held-out behavior check uses an empty file, a corrupt byte, multilingual lines, and the same seed twice. Predict which calls succeed, which fail, and which artifact bytes must match.

Learning Project Submission & Evidence

0 / 4096 bytes
Declared Assistance Condition

Current Submitted Evidence Details

No submission recorded yet.
Independent mastery criterion

Evidence includes commands, exit codes, observed output, and a diagnosis/remediation for at least three cases; it does not confuse a passing subprocess with mastery.

Source resources and provenance

original_explanation Apache-2.0 AND CC-BY-SA-4.0 share-alike

Text and Data Tools Learning Project

Author: Curriculum team (delivered in Ticket 16) | Original Source Link ↗

Relevant Section / Range: cli.py error paths: missing input, strict decoding, --output-dir requirement, ratio validation

Verified Version & Date: Verified against the checked-in implementation on 2026-10-30

Selection Rationale:

Held-out debugging must target the real CLI seams and exit codes, not a simplified re-implementation.

Source Prerequisites:

Study Days 01–12 Python fundamentals

Safe Fallback (Offline Resilience):

All debugging cases run on a temporary corpus in a disposable directory with no external services.

Compared Candidate Sources & Selection Analysis (2)
External debugging courses ↗ compared_alternative [PSF License Version 2]

pdb documentation is a useful secondary reference, but the held-out cases exercise CLI behavior end to end.

pytest documentation ↗ compared_alternative [MIT]

Natural next step to formalize the held-out cases; the activity first exercises the CLI seams directly.

04 120 min

Project report: assumptions and limits

Independent Mastery Check: the first unaided report is sealed and submitted before any assistance; later AI polishing is a separately labeled, unassessed artifact that never replaces it.

Prerequisites

Complete the three preceding activities and retain their evidence.

Learning Objective

Write a short report connecting deterministic preparation to later tokenizer and model experiments while stating assumptions and limits.

Instructions

Required work: write and seal (submit, date) your unaided report first -- covering source attribution, encoding, token definition, split ratios/seed, leakage policy, observed measurements, reproducibility, and limits. Any later assisted polish is a separately labeled artifact that does not replace the sealed version; if assistance exposes a substantive gap, remediate with a fresh unaided check before counting mastery. Optional work: propose a later tokenizer experiment without changing this project.

Practice and prediction

Explain why fixed train/validation/test artifacts can be reused by Makemore, minbpe, and NanoGPT, and why a fixed split does not prove model generalization.

Independent mastery criterion

Report is complete, cites the actual project artifacts, lists at least three assumptions and three limitations, and distinguishes the sealed unaided draft from any separately labeled assisted polish.

Source resources and provenance

original_explanation Apache-2.0 AND CC-BY-SA-4.0 share-alike

Text and Data Tools Learning Project

Author: Curriculum team (delivered in Ticket 16) | Original Source Link ↗

Relevant Section / Range: manifest.json fields: source_sha256, split ratios/counts/hashes, leakage_check, environment, created_at

Verified Version & Date: Verified against the checked-in implementation on 2026-10-30

Selection Rationale:

The report must cite the real manifest and artifacts the learner generated, preserving attribution to the project.

Source Prerequisites:

Study Days 01–12 Python fundamentals

Safe Fallback (Offline Resilience):

Report evidence consists of local artifacts and command output captured in earlier activities.

Compared Candidate Sources & Selection Analysis (2)
Google Technical Writing One ↗ compared_alternative [CC BY 4.0 (course content)]

Solid general technical-writing training, but the report rubric here is specific to the project's manifest fields and provenance.

Write the Docs: Beginner's guide to documentation ↗ compared_alternative [CC BY-NC-SA 4.0]

General documentation guidance; the report rubric remains bound to this project's manifest fields.