86 / 90

Day 86: Capstone Integration - Harness, Baseline & Controlled Delivery

Wire the bounded agent harness to the chosen model adapter, re-establish the deterministic baseline ledger, and rehearse CI delivery and fault recovery on disposable fixtures.

AI Agents & Systems 480 active minutes (8 hours)
01 120 min

Integration Architecture: The Adapter Boundary

Agent-Assisted Engineering Check

Prerequisites

Day 85 frozen charter and acceptance contract; projects/agent_learning source layout.

Learning Objective

Design the integration seam that composes the verified harness (tools, budgets, scorer) with the selected model adapter - adding exactly one bounded tool - without weakening any existing limit.

Instructions

1. Map the components in projects/agent_learning: cli.py entry point, agent.py loop with step/retry budgets and path checks, fixtures.py disposable workspace, the deterministic-v1 adapter, and the independent scorer.
2. Draw the adapter boundary: the adapter proposes actions; the harness validates them, executes bounded tools, and returns structured observations. The model never bypasses validation.
3. Specify the one new bounded tool (for example count_words(note_name) -> int): registered name, typed arguments, fixture-confined path handling, and its own failure contract.
4. Specify the live-adapter slot: the same action schema as deterministic-v1, gated by the ticket-23 authorization contract; the harness cannot tell which adapter produced an action - that opacity is the seam.
5. Write integration_design.md: a text component diagram, the tool contract, and a 'limits preserved' checklist (step budget, retry budget, payload limit, path check, termination conditions).

Practice & Prediction Log

Write the count_words tool contract (name, arguments, return type, error modes) and the limits-preserved checklist proving that swapping adapters cannot raise a budget.

Independent Mastery Criterion

Produce a design where every control lives harness-side and the adapter is replaceable without touching budgets, paths, or termination.

Authoritative Source Resources & Provenance

linking arXiv.org perpetual non-exclusive license / Open Access

Toolformer: Language Models Can Teach Themselves to Use Tools

Author: Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom | Original Source Link ↗

Relevant Section / Range: Sections 2-3: API-call annotation, sampling, filtering, tool integration

Verified Version & Date: NeurIPS 2023 / arXiv:2302.04761v4 (Verified: 2026-10-30)

Selection Rationale:

Primary reference for model-tool integration with explicit API contracts; informs the capstone's adapter boundary where the harness owns tool validation and the model only proposes calls.

Source Prerequisites:

Language-model finetuning, API calls, and tool schemas.

Safe Fallback (Offline Resilience):

The capstone's tool contract template (name, typed arguments, error modes) is fully specified in the lesson instructions.

Compared Candidate Sources & Selection Analysis (2)
Yao et al.: ReAct ↗ compared_alternative [Open Access / CC BY 4.0]

Prompt-time interleaving rather than finetuned tool calls; both define the model-side of the same adapter boundary.

Large API-corpus retrieval; Toolformer's minimal single-tool mechanics match the capstone's bounded tool set.

linking All Rights Reserved (personal research blog) Prose is linked, not copied; the curriculum uses its own bounded-agent implementation

LLM Powered Autonomous Agents

Author: Lilian Weng | Original Source Link ↗

Relevant Section / Range: Sections: planning, memory, tool use, challenges

Verified Version & Date: Blog post June 23, 2023 (Verified: 2026-10-30)

Selection Rationale:

Widely used survey mapping agent architecture components (planning, memory, tools) and their failure modes; supports the capstone integration architecture review.

Source Prerequisites:

Agent loops, tool use, and planning vocabulary.

Safe Fallback (Offline Resilience):

The capstone architecture lesson specifies the adapter boundary, budgets, and tool contracts directly; no blog content is required offline.

Compared Candidate Sources & Selection Analysis (2)
Significant Gravitas: AutoGPT repository ↗ compared_alternative [MIT License (outside autogpt_platform/); Polyform Shield License (autogpt_platform/)]

Open-ended autonomous loop; deliberately excluded as the capstone must remain a bounded, evaluable system.

Yohei Nakajima: BabyAGI repository ↗ compared_alternative [MIT License]

Task-list autonomy demo; the survey article gives vocabulary while the capstone keeps fixed tasks and budgets.

02 120 min

Baseline Re-Execution and Evidence Ledger

Independent Mastery Check (no AI assistance).

Prerequisites

Day 86 Activity 1; the published commands from tickets 19, 21, 22, and 23.

Learning Objective

Re-execute the previously verified public commands unchanged and record a baseline ledger that any reviewer can reproduce - the capstone is compared against this ledger, not against memory.

Instructions

1. Run the deterministic baseline: python -m projects.agent_learning.cli evaluate --fixture summarize-notes --output /tmp/capstone-baseline.json; confirm exit 0 and expected_report_sha256 = 9e8315034af750009a4b2cfb261a077428afe441bc45c602cb7ead32ac0fd787 in the artifact.
2. Run the scorer tests: python -m unittest discover -s tests/projects -p 'test_agent_learning.py' -v; record the pass count.
3. Validate both contracts: python3 projects/lora/evidence/validate_contract.py and python3 projects/agent_learning/evidence/validate_contract.py; record that blocked statuses remain blocked.
4. Record each row in baseline_ledger.md: command, exit code, artifact path, SHA-256, environment (python3 --version), and a one-line observation.
5. Verify the pretraining evidence package still parses: python3 -c "import json; json.load(open('curriculum/projects/docs/pretrain-cloud-evidence/evidence-package.json'))" and note run_status=blocked.
6. Compare: every future capstone run reports identical-or-justified deltas against this ledger.

Practice & Prediction Log

Produce baseline_ledger.md with one row per command (at least five rows), each carrying command, exit code, artifact sha256, and observation; flag any row that did not reproduce instead of editing it to green.

Independent Mastery Criterion

Deliver a ledger reproducing all five commands with artifact-level proof (hashes or counts), including honest failure rows.

Authoritative Source Resources & Provenance

linking MIT

OpenAI Evals: Framework for Evaluating LLMs and Contamination Checks

Author: OpenAI Research | Original Source Link ↗

Relevant Section / Range: README sections 'Running evals' and 'Writing evals'

Verified Version & Date: OpenAI Evals Repository (Verified: 2026-10-30)

Selection Rationale:

Standardized evaluation repository specifying rigorous separation of eval prompts, canary GUID strings, and contamination test protocols.

Source Prerequisites:

Python, evaluation harnesses (MMLU, GSM8K), JSONL datasets.

Safe Fallback (Offline Resilience):

Canary GUID matching, token set intersection, and leak detection scripts are implemented in the curriculum.

Compared Candidate Sources & Selection Analysis (2)
EleutherAI LM Evaluation Harness ↗ compared_alternative [MIT]

Widely adopted evaluation harness; OpenAI Evals provides explicit canary guidance and contamination check specifications.

Hugging Face Lighteval ↗ compared_alternative [MIT]

Modern lightweight eval framework; OpenAI Evals illustrates standard test-set integrity guarantees.

linking Open Access / CC BY 4.0

Reflexion: Language Agents with Verbal Reinforcement Learning

Author: Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao | Original Source Link ↗

Relevant Section / Range: Sections 3-4: self-reflection loop, episodic memory, retry budgets

Verified Version & Date: NeurIPS 2023 / arXiv:2303.11366v4 (Verified: 2026-10-30)

Selection Rationale:

Defines failure-feedback-driven retry with bounded attempts; justifies the capstone's retry budget and its distinction between corrected-in-budget and safe termination.

Source Prerequisites:

Agent loops, tool feedback, and retry semantics.

Safe Fallback (Offline Resilience):

The fault-injection drill specifies verdict classes and budgets locally without needing the paper.

Compared Candidate Sources & Selection Analysis (2)
Yao et al.: ReAct ↗ compared_alternative [Open Access / CC BY 4.0]

Single-episode action/observation protocol; Reflexion adds the cross-attempt self-correction used in the recovery drill.

External-tool critiquing loop; Reflexion's verbal-feedback retry is simpler to bind to the capstone's retry budget.

03 120 min

Controlled Delivery: CI Checks and Rollback Rehearsal

Agent-Assisted Engineering Check

Prerequisites

Day 86 Activity 2 baseline ledger; the ticket-24 practice package at curriculum/engineering/ci_delivery_rollback.

Learning Objective

Rehearse the ticket-24 delivery discipline on the capstone change-set - branch, diff review, automated checks, staged deployment, health gate, and rollback - entirely inside disposable fixtures.

Instructions

1. Run the CI practice pipeline: python3 scripts/ci_practice_runner.py and identify its five stages (clean_install, regression_exercise, conflict_resolution, controlled_deployment, health_failure_and_rollback).
2. Reproduce the stage semantics for the capstone: the 'change' is the new bounded tool plus task; a clean install must still pass the scorer; a deliberately broken tool argument must fail the checks.
3. Practice diff review on the tool diff: read every line; confirm no budget constant changed and no network call appeared.
4. Rehearse rollback: deploy the change in the disposable fixture, fail the health check (for example, the scorer fails on missing_note), restore the known-good fixture state, and record the restore evidence.
5. Write delivery_log.md: stage, command, observable outcome, pass/fail, and the rollback proof - observable outcomes only, never screenshots of commands.

Practice & Prediction Log

Produce delivery_log.md covering all five stages plus the injected-regression diagnosis and the rollback evidence (for example, the restored file hash equals the pre-change hash).

Independent Mastery Criterion

Demonstrate deploy - health failure - verified restore on disposable fixtures with artifact-level evidence at each stage.

Authoritative Source Resources & Provenance

linking All Rights Reserved (martinfowler.com) Article is linked, not copied; pipeline practice uses the repository's own runner

Continuous Integration

Author: Martin Fowler | Original Source Link ↗

Relevant Section / Range: Practices of Continuous Integration: Make the Build Self-Testing; Fix Broken Builds Immediately

Verified Version & Date: Second edition, May 2006 (Verified: 2026-10-30)

Selection Rationale:

Standard essay defining CI as verified mainline integration; grounds the capstone delivery rehearsal's check-before-merge and broken-build-response rules.

Source Prerequisites:

Version control, automated tests, and build scripts.

Safe Fallback (Offline Resilience):

scripts/ci_practice_runner.py demonstrates each stage locally; the article is a reference, not a dependency.

Compared Candidate Sources & Selection Analysis (2)
Humble & Farley: Continuous Delivery (Addison-Wesley) ↗ compared_alternative [All Rights Reserved (book)]

Full deployment-pipeline book; the Fowler essay suffices for the rehearsal vocabulary while the repo runner supplies practice.

Forsgren, Humble, Kim: Accelerate / DORA State of DevOps ↗ compared_alternative [CC BY-NC-ND (reports)]

Industry metrics research; cited as background while the article supplies the operative CI definition.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 8: Release Engineering

Author: Dinah McNutt (ed. Betsy Beyer, Tim Harvey), in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Chapter 8: release pipeline, canarying, rollback criteria

Verified Version & Date: O'Reilly 2016 / sre.google online edition (Verified: 2026-10-30)

Selection Rationale:

Defines staged rollout, health-gated release, and rollback criteria; supplies the controlled-deployment and health-failure stages rehearsed by the capstone.

Source Prerequisites:

Version control, CI basics, and deployment vocabulary.

Safe Fallback (Offline Resilience):

The CI practice runner encodes the same stages in disposable fixtures, so the exercise is self-contained.

Compared Candidate Sources & Selection Analysis (2)
Humble & Farley: Continuous Delivery ↗ compared_alternative [All Rights Reserved (book)]

General delivery pipeline text; the SRE chapter adds explicit rollback/health-gate criteria used by the drill.

Martin Fowler: Continuous Integration ↗ compared_alternative [All Rights Reserved]

Integration-side discipline; the SRE chapter covers the release and rollback side of the same rehearsal.

04 120 min

Assessment: Fault Injection and Bounded Recovery Drill

Independent Mastery Check (no AI assistance).

Prerequisites

Day 86 Activities 1-3.

Learning Objective

Prove the integrated system fails safely by exercising each deterministic fault - malformed arguments, missing input, loop - and verifying that structured observations end in correction or safe termination within budget.

Instructions

1. For each fault in {invalid_arguments, missing_note, loop}, run the fixture adapter with that fault enabled and record the action/observation trace.
2. For invalid_arguments: confirm the harness returns a structured error and the agent corrects or terminates within max_retries=2.
3. For missing_note: confirm a structured missing-note observation rather than an escaped exception.
4. For loop: confirm the step budget (max_steps=6) forces termination instead of an infinite cycle.
5. Classify each outcome as corrected-in-budget, terminated-safe, or UNBOUNDED (automatic fail). Record fault_drill.json with per-fault verdicts.
6. Finish with one full recovery: from a failed run, restore the clean fixture and re-run to green - the capstone's controlled-recovery proof.

Practice & Prediction Log

Produce fault_drill.json: for each of the three faults record the trace pointer, observed termination, retry count, and verdict; plus the recovery re-run result verified by hash.

Independent Mastery Criterion

Every fault produces a structured, budgeted outcome with zero unbounded escapes, and the recovery re-run is verified by artifact hash.

Authoritative Source Resources & Provenance

linking Open Access / CC BY 4.0

Reflexion: Language Agents with Verbal Reinforcement Learning

Author: Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao | Original Source Link ↗

Relevant Section / Range: Sections 3-4: self-reflection loop, episodic memory, retry budgets

Verified Version & Date: NeurIPS 2023 / arXiv:2303.11366v4 (Verified: 2026-10-30)

Selection Rationale:

Defines failure-feedback-driven retry with bounded attempts; justifies the capstone's retry budget and its distinction between corrected-in-budget and safe termination.

Source Prerequisites:

Agent loops, tool feedback, and retry semantics.

Safe Fallback (Offline Resilience):

The fault-injection drill specifies verdict classes and budgets locally without needing the paper.

Compared Candidate Sources & Selection Analysis (2)
Yao et al.: ReAct ↗ compared_alternative [Open Access / CC BY 4.0]

Single-episode action/observation protocol; Reflexion adds the cross-attempt self-correction used in the recovery drill.

External-tool critiquing loop; Reflexion's verbal-feedback retry is simpler to bind to the capstone's retry budget.

linking arXiv.org perpetual non-exclusive license / Open Access

Toolformer: Language Models Can Teach Themselves to Use Tools

Author: Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom | Original Source Link ↗

Relevant Section / Range: Sections 2-3: API-call annotation, sampling, filtering, tool integration

Verified Version & Date: NeurIPS 2023 / arXiv:2302.04761v4 (Verified: 2026-10-30)

Selection Rationale:

Primary reference for model-tool integration with explicit API contracts; informs the capstone's adapter boundary where the harness owns tool validation and the model only proposes calls.

Source Prerequisites:

Language-model finetuning, API calls, and tool schemas.

Safe Fallback (Offline Resilience):

The capstone's tool contract template (name, typed arguments, error modes) is fully specified in the lesson instructions.

Compared Candidate Sources & Selection Analysis (2)
Yao et al.: ReAct ↗ compared_alternative [Open Access / CC BY 4.0]

Prompt-time interleaving rather than finetuned tool calls; both define the model-side of the same adapter boundary.

Large API-corpus retrieval; Toolformer's minimal single-tool mechanics match the capstone's bounded tool set.