85 / 90

Day 85: Capstone Charter - Problem, Scope & Model Selection

Bind the previously verified projects into one bounded capstone: define the problem and user, freeze an acceptance contract, and justify the model choice from real evidence.

AI Agents & Systems 480 active minutes (8 hours)
01 120 min

Capstone Charter: Problem, User, and Bounded Scope

Agent-Assisted Engineering Check

Prerequisites

Days 82-84 scientific agent evaluation; the bounded agent Learning Project at projects/agent_learning and its four-language guides.

Learning Objective

Author a one-page capstone charter that binds a concrete problem, a named user, an explicit scope boundary, and non-goals, reusing the previously verified bounded agent rather than hiding a new unbounded system inside the capstone.

Instructions

1. Re-read projects/agent_learning/README.md and agent.py; list the registered bounded tools (list_notes, read_note, write_report) and the fixture task summarize-notes.
2. Write the problem statement: the capstone extends the verified notes-to-report agent with exactly one new bounded tool (for example count_words or list_deadlines) and one additional fixed task, still scored by the existing independent scorer.
3. Name the user (a learner assembling study notes into a report) and the usage context (a local, offline-first disposable fixture workspace).
4. Draw the scope line. IN: the deterministic adapter, registered bounded tools, fixed fixtures, step/retry budgets. OUT: open-web browsing, arbitrary shell execution, unbounded retries, production autonomy.
5. Declare non-goals: no live-model claims until the ticket-23 authorization gate is approved; no new agent framework; no hidden network calls.
6. Save the draft as capstone_charter.md.

Practice & Prediction Log

Draft capstone_charter.md with sections Problem, User, In-Scope, Out-of-Scope, and Non-Goals. Then classify three feature requests as in-scope or out-of-scope: (a) 'read a third note from the fixture', (b) 'let the agent pip-install any package it wants', (c) 'add a bounded word-count tool'.

Independent Mastery Criterion

Produce a charter whose every in-scope item maps to a stated bound and whose non-goals explicitly exclude at least three capabilities (e.g., web browsing, arbitrary code execution, unbounded retries).

Authoritative Source Resources & Provenance

linking Open Access / CC BY 4.0

ReAct: Synergizing Reasoning and Acting in Language Models

Author: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao | Original Source Link ↗

Relevant Section / Range: Sections 2-4: interleaved reasoning/action protocol; Figure 1 trace format

Verified Version & Date: ICLR 2023 / arXiv:2210.03629v3 (Verified: 2026-10-30)

Selection Rationale:

Primary literature for interleaving explicit reasoning steps with bounded tool actions; anchors the capstone's requirement that every step produces an inspectable observation inside a budgeted loop.

Source Prerequisites:

Language-model prompting, tool-call loops, and basic action/observation traces.

Safe Fallback (Offline Resilience):

The curriculum restates the action/observation loop and reimplements a deterministic three-tool agent locally; no paper text is imported.

Compared Candidate Sources & Selection Analysis (2)

Pure reasoning traces without tool actions; ReAct adds the observable action/observation protocol that the capstone harness enforces.

Embodied closed-loop planning for robotics; ReAct's text-domain tool protocol maps directly onto the capstone's bounded tools.

linking arXiv non-exclusive license / Open Access

Model Cards for Model Reporting

Author: Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru | Original Source Link ↗

Relevant Section / Range: Section 4 Model Card Sections (4.1-4.9)

Verified Version & Date: FAT* 2019 / arXiv:1810.03993v3 (Verified: 2026-10-30)

Selection Rationale:

Standard framework for disclosing a model's intended use, evaluation conditions, and limitations; supplies the capstone's model-choice and provenance documentation pattern.

Source Prerequisites:

Basic model evaluation vocabulary: dataset, metric, intended use.

Safe Fallback (Offline Resilience):

The curriculum provides a local model-decision table template covering intended use, evaluation data, and caveats.

Compared Candidate Sources & Selection Analysis (2)
Gebru et al.: Datasheets for Datasets ↗ compared_alternative [Open Access]

Dataset-side documentation rather than model-side; used alongside Model Cards in the provenance table rather than as a replacement.

OpenAI GPT-4 System Card ↗ compared_alternative [All Rights Reserved (web publication)]

Vendor-specific safety disclosure; Model Cards provide the general reusable schema.

02 120 min

Acceptance Contract: Fixed Tasks, Baseline, and Ceilings

Independent Mastery Check (no AI assistance).

Prerequisites

Day 85 Activity 1 charter draft; the ticket-23 contract schema in projects/agent_learning/evidence/live_eval_contract.json.

Learning Objective

Convert the charter into a measurable acceptance contract: a fixed task set, the deterministic baseline and its report hash, metric definitions, resource ceilings, failure cases, and the exact submission evidence a reviewer will inspect.

Instructions

1. Copy the schema style of projects/agent_learning/evidence/live_eval_contract.json and create capstone_acceptance.json.
2. Bind the task set: keep the six live-eval task categories (success, two malformed tool requests, missing input, loop, budget exhaustion) and add the one new fixed task from the charter; freeze it before any tuning.
3. Record the baseline: adapter deterministic-v1, fixture summarize-notes, expected_report_sha256 9e8315034af750009a4b2cfb261a077428afe441bc45c602cb7ead32ac0fd787, plus the baseline command python -m projects.agent_learning.cli evaluate --fixture summarize-notes --output .
4. Define metrics per task: status, steps, retries, termination reason, and report-hash equality, mirroring the contract's scorer dimensions (outcome, tool behavior, budget, failure handling).
5. Set resource ceilings: max 6 steps per task, max 2 retries per fault, a 30-minute runtime ceiling, a USD 5.00 cost ceiling for any authorized live run, and stop conditions copied from the contract.
6. List submission evidence: charter, this contract, the baseline ledger, per-task results, the failure log, and the provenance table.

Practice & Prediction Log

Write capstone_acceptance.json containing all nine required elements - problem, user, scope, model_choice, baseline, evaluation_set, failure_cases, resource_limits, submission_evidence - with live-run fields explicitly null and labeled not-run-awaiting-authorization.

Independent Mastery Criterion

Produce a contract under which a reviewer can predict pass/fail for each task before any run; live-run fields stay null and carry the not-run-awaiting-authorization label.

Authoritative Source Resources & Provenance

linking Open Access / CC BY 4.0

AgentBench: Evaluating LLMs as Agents

Author: Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. | Original Source Link ↗

Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis

Verified Version & Date: ICLR 2024 / arXiv:2308.03688v4 (Verified: 2026-10-30)

Selection Rationale:

Systematic multi-environment agent benchmark; supplies the capstone's fixed-task-set discipline and per-task scoring decomposition used by the acceptance contract.

Source Prerequisites:

Agent loops, tool use, and benchmark design fundamentals.

Safe Fallback (Offline Resilience):

The capstone task set is fixed locally in the acceptance contract and does not require downloading the benchmark.

Compared Candidate Sources & Selection Analysis (2)

Deep single-domain web tasks with hosted infrastructure; AgentBench's multi-environment view better fits a general agent evaluation.

Assistant-level real-world questions; also cited by the capstone for outcome-level task design.

linking arXiv non-exclusive license / Open Access

GAIA: a benchmark for General AI Assistants

Author: Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom | Original Source Link ↗

Relevant Section / Range: Sections 3-4: level design, tooling requirements, human vs agent gap

Verified Version & Date: arXiv:2311.12983v3 (Verified: 2026-10-30)

Selection Rationale:

Defines graded assistant tasks with unambiguous factual scoring; motivates the capstone's distinction between success tasks, malformed-request handling, and failure cases.

Source Prerequisites:

Agent capabilities, tool use, and evaluation metrics.

Safe Fallback (Offline Resilience):

The capstone evaluation tasks are defined in the local acceptance contract and need no external download.

Compared Candidate Sources & Selection Analysis (2)
Liu et al.: AgentBench: Evaluating LLMs as Agents ↗ compared_alternative [Open Access / CC BY 4.0]

Multi-environment execution benchmark; GAIA adds graded real-world assistance tasks, and both inform the fixed task taxonomy.

Zhou et al.: WebArena ↗ compared_alternative [Open Access]

Web-only task environment; GAIA covers general assistant capability beyond browsing.

03 120 min

Model Selection: Trained Adapters vs Existing Models

Agent-Assisted Engineering Check

Prerequisites

Day 85 Activities 1-2; the ticket-19 evidence package and the ticket-21 LoRA comparison contract.

Learning Objective

Decide which model powers the capstone agent by inspecting the cloud pretraining evidence package and the LoRA comparison contract, while keeping an appropriate existing model admissible when it performs better for the final agent.

Instructions

1. Open curriculum/projects/docs/pretrain-cloud-evidence/evidence-package.json: note the 4-layer/4-head/64-embedding GPT recipe, the tinyshakespeare data provenance and SHA-256, and run_status=blocked - the pretraining work teaches the data-to-checkpoint loop but yields no production-capable model.
2. Open projects/lora/evidence/comparison_contract.json: note base_model openai-community/gpt2, LoRA r=8 alpha=16 on q_proj/v_proj, seed 42 splits 0.8/0.1/0.1, status=blocked-awaiting-authorization, and success criteria that predeclare direction only.
3. Read projects/agent_learning/evidence/model_selection_plan.md: one open-weight local candidate recommended plus three provider API alternatives, each with version, access, data-usage, and cost conditions.
4. Build the decision table. Candidates: (a) self-trained GPT from the pretraining recipe, (b) the LoRA-adapted gpt2, (c) the recommended existing open model. Score each on observed evidence, not aspiration: deterministic-baseline parity, license and redistribution rights, pinned revisions, and cost ceilings.
5. Record the decision honestly: the deterministic-v1 adapter remains the offline evaluation baseline; for any authorized live run the recommended open-weight model is admissible because self-trained artifacts are unproven - pretraining and LoRA results inform the design without obliging their use.
6. Save the analysis as model_decision.md with one license row per candidate.

Practice & Prediction Log

Produce model_decision.md: a three-row candidate table (evidence file, license, verdict) plus a one-paragraph rationale naming the chosen live-run candidate and explaining why the self-trained options inform but do not drive the final agent.

Independent Mastery Criterion

Justify the model choice with citations to all three evidence artifacts, a license check per candidate, and zero invented measurements.

Authoritative Source Resources & Provenance

linking arXiv non-exclusive license / Open Access

Datasheets for Datasets

Author: Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford | Original Source Link ↗

Relevant Section / Range: Section 3 Questions and Workflow (3.1 Motivation through 3.7 Maintenance)

Verified Version & Date: Communications of the ACM 64(12) / arXiv:1803.09010v8 (Verified: 2026-10-30)

Selection Rationale:

Standard questionnaire for documenting dataset origin, composition, rights, and maintenance; structures the capstone evidence package's data-provenance table.

Source Prerequisites:

Dataset provenance, splits, and licensing vocabulary.

Safe Fallback (Offline Resilience):

The provenance table template in the capstone instructions restates the required disclosure fields locally.

Compared Candidate Sources & Selection Analysis (2)
Mitchell et al.: Model Cards for Model Reporting ↗ compared_alternative [Open Access]

Model-side reporting rather than dataset-side; the capstone provenance table cites both.

Bender & Friedman: Data Statements for Natural Language Processing ↗ compared_alternative [CC BY 4.0 (ACL Anthology, 2016 or later)]

NLP-specific demographic disclosure; Datasheets cover the general lifecycle needed for arbitrary capstone data.

linking arXiv non-exclusive license / Open Access

Model Cards for Model Reporting

Author: Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru | Original Source Link ↗

Relevant Section / Range: Section 4 Model Card Sections (4.1-4.9)

Verified Version & Date: FAT* 2019 / arXiv:1810.03993v3 (Verified: 2026-10-30)

Selection Rationale:

Standard framework for disclosing a model's intended use, evaluation conditions, and limitations; supplies the capstone's model-choice and provenance documentation pattern.

Source Prerequisites:

Basic model evaluation vocabulary: dataset, metric, intended use.

Safe Fallback (Offline Resilience):

The curriculum provides a local model-decision table template covering intended use, evaluation data, and caveats.

Compared Candidate Sources & Selection Analysis (2)
Gebru et al.: Datasheets for Datasets ↗ compared_alternative [Open Access]

Dataset-side documentation rather than model-side; used alongside Model Cards in the provenance table rather than as a replacement.

OpenAI GPT-4 System Card ↗ compared_alternative [All Rights Reserved (web publication)]

Vendor-specific safety disclosure; Model Cards provide the general reusable schema.

04 120 min

Assessment: Charter Review Gate and Scope Freeze

Independent Mastery Check (no AI assistance).

Prerequisites

Day 85 Activities 1-3.

Learning Objective

Run a structured go/no-go review of the charter and acceptance contract against a fixed nine-element checklist, freeze scope, and write the recovery plan for any failed element.

Instructions

1. Assemble the nine-element checklist: concrete problem, named user, scope boundary, model choice, baseline, evaluation set, failure cases, resource limits, submission evidence.
2. Score the charter and contract element by element; each element is PRESENT or MISSING with a one-line pointer to where it is defined.
3. If every element is present, freeze scope: record scope_frozen=true and the freeze date; later feature requests must route through re-review.
4. If any element is missing, write a NO-GO verdict naming the missing elements and the specific activity (act-01, act-02, or act-03) that repairs it.
5. Dry-run one scope-creep request ('add a chat UI') and show it fails the scope line.
6. Save the signed review as charter_review.json.

Practice & Prediction Log

Produce charter_review.json: nine checklist verdicts with evidence pointers, a final GO or NO-GO, and - regardless of verdict - the scope-creep ruling for 'add a chat UI'.

Independent Mastery Criterion

Complete a review gate where every verdict cites stored evidence and the scope-freeze or NO-GO decision is explicit and reproducible.

Authoritative Source Resources & Provenance

linking Open Access / CC BY 4.0

ReAct: Synergizing Reasoning and Acting in Language Models

Author: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao | Original Source Link ↗

Relevant Section / Range: Sections 2-4: interleaved reasoning/action protocol; Figure 1 trace format

Verified Version & Date: ICLR 2023 / arXiv:2210.03629v3 (Verified: 2026-10-30)

Selection Rationale:

Primary literature for interleaving explicit reasoning steps with bounded tool actions; anchors the capstone's requirement that every step produces an inspectable observation inside a budgeted loop.

Source Prerequisites:

Language-model prompting, tool-call loops, and basic action/observation traces.

Safe Fallback (Offline Resilience):

The curriculum restates the action/observation loop and reimplements a deterministic three-tool agent locally; no paper text is imported.

Compared Candidate Sources & Selection Analysis (2)

Pure reasoning traces without tool actions; ReAct adds the observable action/observation protocol that the capstone harness enforces.

Embodied closed-loop planning for robotics; ReAct's text-domain tool protocol maps directly onto the capstone's bounded tools.

linking Open Access / CC BY 4.0

AgentBench: Evaluating LLMs as Agents

Author: Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. | Original Source Link ↗

Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis

Verified Version & Date: ICLR 2024 / arXiv:2308.03688v4 (Verified: 2026-10-30)

Selection Rationale:

Systematic multi-environment agent benchmark; supplies the capstone's fixed-task-set discipline and per-task scoring decomposition used by the acceptance contract.

Source Prerequisites:

Agent loops, tool use, and benchmark design fundamentals.

Safe Fallback (Offline Resilience):

The capstone task set is fixed locally in the acceptance contract and does not require downloading the benchmark.

Compared Candidate Sources & Selection Analysis (2)

Deep single-domain web tasks with hosted infrastructure; AgentBench's multi-environment view better fits a general agent evaluation.

Assistant-level real-world questions; also cited by the capstone for outcome-level task design.