Day 87: Capstone Acceptance - Evaluation, Evidence & Review

01 120 min

Fixed Evaluation Protocol and Honest Metrics

Agent-Assisted Engineering Check

Prerequisites

Day 86 integrated harness and baseline ledger.

Learning Objective

Freeze the evaluation before any tuning run: task order, metric definitions, and the evidence kinds (deterministic-fixture vs live-model-run) that keep simulated and real results from being merged.

Instructions

1. Re-open capstone_acceptance.json: the task set, metrics, and ceilings are now frozen - post-hoc edits void the evaluation.
2. Define per-task reporting exactly like live_eval_contract.json: status, steps, retries, termination, report hash; label every live field evidence_kind=live-model-run and every fixture field evidence_kind=deterministic-fixture.
3. Predeclare the comparison rule: deterministic runs must match the baseline ledger; authorized live runs are compared per task against it, and differences are recorded, not hidden.
4. Plan coverage honestly: at least one success task plus every malformed-request and failure task - never only attractive examples.
5. Handle missing evidence explicitly: a task not run is status=not-run, never silently dropped.
6. Write eval_protocol.md: the frozen rules a reviewer can audit against the results table.

Practice & Prediction Log

Write eval_protocol.md with the frozen task list, the metric table, the evidence_kind labeling rule, and the not-run policy; then predict the results-table shape before running anything.

Mark as Completed

Independent Mastery Criterion

Deliver a protocol under which results, regressions, and missing evidence all have predefined, auditable slots.

Authoritative Source Resources & Provenance

linking Open Access / CC BY 4.0

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Author: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R. Narasimhan | Original Source Link ↗

Relevant Section / Range: Sections 3-5: task construction from merged PRs, FAIL_TO_PASS/PASS_TO_PASS metrics

Verified Version & Date: ICLR 2024 / arXiv:2310.06770v3 (Verified: 2026-10-30)

Selection Rationale:

Defines execution-verified agent evaluation with held-out real tasks; motivates the capstone rule that outcomes are measured by test state transitions, not plausible output.

Source Prerequisites:

Software testing, issue resolution tasks, and benchmark design.

Safe Fallback (Offline Resilience):

The capstone's per-task status/steps/termination metrics are defined locally in the evaluation protocol.

Compared Candidate Sources & Selection Analysis (2)

Chen et al.: Evaluating Large Language Models Trained on Code (HumanEval) ↗ compared_alternative [Open Access]

Function-level synthesis tasks; SWE-bench's repository-level issue tasks better match agent evaluation with tool use.

Human-validated subset of SWE-bench; cited as refinement, while the original paper defines the protocol.

linking Open Access / CC BY 4.0

AgentBench: Evaluating LLMs as Agents

Author: Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. | Original Source Link ↗

Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis

Verified Version & Date: ICLR 2024 / arXiv:2308.03688v4 (Verified: 2026-10-30)

Selection Rationale:

Systematic multi-environment agent benchmark; supplies the capstone's fixed-task-set discipline and per-task scoring decomposition used by the acceptance contract.

Source Prerequisites:

Agent loops, tool use, and benchmark design fundamentals.

Safe Fallback (Offline Resilience):

The capstone task set is fixed locally in the acceptance contract and does not require downloading the benchmark.

Compared Candidate Sources & Selection Analysis (2)

Zhou et al.: WebArena: A Realistic Web Environment for Building Autonomous Agents ↗ compared_alternative [Open Access]

Deep single-domain web tasks with hosted infrastructure; AgentBench's multi-environment view better fits a general agent evaluation.

Mialon et al.: GAIA: a benchmark for General AI Assistants ↗ compared_alternative [Open Access]

Assistant-level real-world questions; also cited by the capstone for outcome-level task design.

02 120 min

Capstone Evidence Package Assembly

Independent Mastery Check (no AI assistance).

Prerequisites

Day 87 Activity 1 frozen protocol; all Day 85-86 artifacts.

Learning Objective

Assemble the sanitized evidence package a reviewer can audit end-to-end: charter, contract, baseline ledger, per-task results, failure log, resource accounting, provenance table, and a privacy pass.

Instructions

1. Create capstone_evidence/ with index.md listing every artifact and what it proves.
2. Attach: charter_review.json, capstone_acceptance.json, baseline_ledger.md, the per-task results table, fault_drill.json, and delivery_log.md.
3. Add resource accounting: wall time per run, total steps/retries, token and cost fields left null until an authorized live run; note continuing storage charges per the pretraining evidence plan.
4. Build the provenance table: model id plus license, dataset origin plus hash, the cited source registry ids, and the repository licenses (Apache-2.0 code, CC BY-SA 4.0 content).
5. Run the privacy pass: scan file contents for tokens, keys, emails, and absolute home paths; redact before packaging - provider identifiers stay in private notes, never in the repository.
6. Validate: a reviewer starting at index.md reaches every claim's evidence in one hop.

Practice & Prediction Log

Produce capstone_evidence/index.md plus the provenance table citing at least the five referenced artifacts (agent project, LoRA comparison contract, live-eval contract, pretraining evidence package, CI practice runner), and record the privacy-scan result.

Mark as Completed

Independent Mastery Criterion

Assemble a package where every claim resolves to stored evidence and the provenance and privacy tables are complete.

Authoritative Source Resources & Provenance

linking arXiv non-exclusive license / Open Access

Datasheets for Datasets

Author: Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford | Original Source Link ↗

Relevant Section / Range: Section 3 Questions and Workflow (3.1 Motivation through 3.7 Maintenance)

Verified Version & Date: Communications of the ACM 64(12) / arXiv:1803.09010v8 (Verified: 2026-10-30)

Selection Rationale:

Standard questionnaire for documenting dataset origin, composition, rights, and maintenance; structures the capstone evidence package's data-provenance table.

Source Prerequisites:

Dataset provenance, splits, and licensing vocabulary.

Safe Fallback (Offline Resilience):

The provenance table template in the capstone instructions restates the required disclosure fields locally.

Compared Candidate Sources & Selection Analysis (2)

Mitchell et al.: Model Cards for Model Reporting ↗ compared_alternative [Open Access]

Model-side reporting rather than dataset-side; the capstone provenance table cites both.

Bender & Friedman: Data Statements for Natural Language Processing ↗ compared_alternative [CC BY 4.0 (ACL Anthology, 2016 or later)]

NLP-specific demographic disclosure; Datasheets cover the general lifecycle needed for arbitrary capstone data.

linking arXiv non-exclusive license / Open Access

Model Cards for Model Reporting

Author: Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru | Original Source Link ↗

Relevant Section / Range: Section 4 Model Card Sections (4.1-4.9)

Verified Version & Date: FAT* 2019 / arXiv:1810.03993v3 (Verified: 2026-10-30)

Selection Rationale:

Standard framework for disclosing a model's intended use, evaluation conditions, and limitations; supplies the capstone's model-choice and provenance documentation pattern.

Source Prerequisites:

Basic model evaluation vocabulary: dataset, metric, intended use.

Safe Fallback (Offline Resilience):

The curriculum provides a local model-decision table template covering intended use, evaluation data, and caveats.

Compared Candidate Sources & Selection Analysis (2)

Gebru et al.: Datasheets for Datasets ↗ compared_alternative [Open Access]

Dataset-side documentation rather than model-side; used alongside Model Cards in the provenance table rather than as a replacement.

Vendor-specific safety disclosure; Model Cards provide the general reusable schema.

03 120 min

Post-Mortem, Failure Taxonomy, and Controlled Recovery

Agent-Assisted Engineering Check

Prerequisites

Day 87 Activities 1-2.

Learning Objective

Write the blameless capstone post-mortem: a failure taxonomy, honest treatment of unmet criteria and regressions, live-versus-simulated scoping, and a demonstrated controlled recovery.

Instructions

1. Group every observed failure into the taxonomy: malformed_tool_request, missing_input, loop, budget_exhausted, provider_or_infra, evaluation_defect.
2. For each class record symptom, trace pointer, root cause, and the fix or the accepted limitation. Unmet acceptance criteria stay unmet - label them; do not narrate over them.
3. State live-versus-simulated scope explicitly: deterministic results prove harness behavior; live-model fields remain null and blocked until authorization - no simulation is presented as a real run.
4. Demonstrate one controlled recovery end-to-end: break a fixture, observe the health-check failure, restore the known-good artifact, re-verify by hash; attach the before and after hashes.
5. Note residual risk: billing or storage follow-ups, stale-content revalidation, and what a reviewer could not check.
6. Save postmortem.md and link it from the evidence index.

Practice & Prediction Log

Produce postmortem.md with the six-class taxonomy table, one worked recovery with hashes, an explicit unmet-criteria list (or the literal word 'none'), and the residual-risk paragraph.

Mark as Completed

Independent Mastery Criterion

Deliver a post-mortem where failures are classified with evidence, unmet criteria remain visible, and recovery is proven rather than claimed.

Authoritative Source Resources & Provenance

linking arXiv non-exclusive license / Open Access

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Author: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. | Original Source Link ↗

Relevant Section / Range: Sections 3-4: judge biases (position, verbosity, self-enhancement), rubric design

Verified Version & Date: NeurIPS 2023 / arXiv:2306.05685v3 (Verified: 2026-10-30)

Selection Rationale:

Documents judge bias and rubric pitfalls; informs the capstone's separate independent/assisted rubrics and its rule that verdicts must cite stored evidence rather than impressions.

Source Prerequisites:

Evaluation metrics, pairwise comparison, and grading rubrics.

Safe Fallback (Offline Resilience):

The capstone rubric is a fixed checklist scored against artifacts; it does not depend on the paper's arena data.

Compared Candidate Sources & Selection Analysis (2)

Liang et al.: Holistic Evaluation of Language Models (HELM) ↗ compared_alternative [Open Access / CC BY 4.0]

Broad scenario taxonomy; MT-Bench's judge-bias analysis is more relevant to rubric-based acceptance review.

Dubois et al.: AlpacaEval / AlpacaFarm ↗ compared_alternative [Open Access]

Automatic instruction-following win-rate; MT-Bench better exposes judging bias relevant to human rubric design.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 15: Postmortem Culture: Learning from Failure

Author: John Lunney and Sue Lueder (ed. Gary O'Connor), in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Chapter 15: blameless postmortems, trigger conditions, action items

Verified Version & Date: O'Reilly 2016 / sre.google online edition (Verified: 2026-10-30)

Selection Rationale:

Establishes the blameless, evidence-first post-mortem format; structures the capstone's failure taxonomy, unmet-criteria disclosure, and recovery proof.

Source Prerequisites:

Incident vocabulary, monitoring, and delivery pipelines.

Safe Fallback (Offline Resilience):

The post-mortem template (taxonomy, unmet criteria, recovery evidence, residual risk) is fully specified in the lesson.

Compared Candidate Sources & Selection Analysis (2)

Foundational just-culture essay; the SRE chapter adds the trigger conditions and action-item tracking used by the capstone.

Google SRE Workbook: Postmortem practices ↗ compared_alternative [CC BY-NC-ND 4.0]

Practitioner workbook follow-up; the original chapter is the stable reference for the taxonomy exercise.

04 120 min

Assessment: Dual-Rubric Acceptance and Submission

Independent Mastery Check (no AI assistance).

Prerequisites

Day 87 Activities 1-3.

Learning Objective

Judge the capstone under two separate rubrics - the Independent Mastery Check rubric for unaided work and the Agent-Assisted Engineering Check rubric for directed work - then submit the evidence package through the project submission form.

Instructions

1. Apply the independent rubric to the work done unaided (charter gate, baseline ledger, evidence assembly, this review): each item is scored on correctness plus evidence pointers.
2. Apply the assisted rubric to directed work (integration design, CI rehearsal, post-mortem): scored on direction quality, diff inspection, and correction handling - never on agent output alone.
3. Verify the separation rule: independent mastery must not be inferred from assisted output; record both rubric totals and their scopes separately.
4. Submit below: repository link, report link, and notes summarizing commands re-run and hashes observed - pasted logs are self-reported evidence and the form labels them as such.
5. Distinguish acceptance layers: this learner project acceptance grades your work; the platform Course Readiness Gate (all 90 days, four language editions, platform behavior, ticket 61) is a separate later acceptance - a passing capstone does not declare the course ready.
6. Record the final verdict in capstone_verdict.json.

Practice & Prediction Log

Submit the capstone evidence (repository link, report link, notes) in the form below, then produce capstone_verdict.json with both rubric scores, their scopes, and the explicit statement separating project acceptance from Course Readiness.

Mark as Completed

Learning Project Submission & Evidence

Notice: Pasted execution logs and notes are self-reported evidence and do not constitute independent or automated verification.

Repository Link (GitHub / GitLab)

Report Link (Optional)

Evidence Notes & Diagnostic Log (Max 4096 bytes)

0 / 4096 bytes

Declared Assistance Condition Independent Project (No AI assistance) Agent-Assisted Project (Directed AI assistance)

Note: AI-free work is an honest declared condition, not claimed surveillance.

Agent Tool Track (if applicable)

Current Submitted Evidence Details

No submission recorded yet.

Independent Mastery Criterion

Pass only when both rubrics are scored separately with evidence, the submission is recorded honestly, and the Course Readiness Gate distinction is stated in the verdict.

Authoritative Source Resources & Provenance

linking arXiv non-exclusive license / Open Access

GAIA: a benchmark for General AI Assistants

Author: Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom | Original Source Link ↗

Relevant Section / Range: Sections 3-4: level design, tooling requirements, human vs agent gap

Verified Version & Date: arXiv:2311.12983v3 (Verified: 2026-10-30)

Selection Rationale:

Defines graded assistant tasks with unambiguous factual scoring; motivates the capstone's distinction between success tasks, malformed-request handling, and failure cases.

Source Prerequisites:

Agent capabilities, tool use, and evaluation metrics.

Safe Fallback (Offline Resilience):

The capstone evaluation tasks are defined in the local acceptance contract and need no external download.

Compared Candidate Sources & Selection Analysis (2)

Liu et al.: AgentBench: Evaluating LLMs as Agents ↗ compared_alternative [Open Access / CC BY 4.0]

Multi-environment execution benchmark; GAIA adds graded real-world assistance tasks, and both inform the fixed task taxonomy.

Zhou et al.: WebArena ↗ compared_alternative [Open Access]

Web-only task environment; GAIA covers general assistant capability beyond browsing.

linking arXiv non-exclusive license / Open Access

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Author: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. | Original Source Link ↗

Relevant Section / Range: Sections 3-4: judge biases (position, verbosity, self-enhancement), rubric design

Verified Version & Date: NeurIPS 2023 / arXiv:2306.05685v3 (Verified: 2026-10-30)

Selection Rationale:

Documents judge bias and rubric pitfalls; informs the capstone's separate independent/assisted rubrics and its rule that verdicts must cite stored evidence rather than impressions.

Source Prerequisites:

Evaluation metrics, pairwise comparison, and grading rubrics.

Safe Fallback (Offline Resilience):

The capstone rubric is a fixed checklist scored against artifacts; it does not depend on the paper's arena data.

Compared Candidate Sources & Selection Analysis (2)

Liang et al.: Holistic Evaluation of Language Models (HELM) ↗ compared_alternative [Open Access / CC BY 4.0]

Broad scenario taxonomy; MT-Bench's judge-bias analysis is more relevant to rubric-based acceptance review.

Dubois et al.: AlpacaEval / AlpacaFarm ↗ compared_alternative [Open Access]

Automatic instruction-following win-rate; MT-Bench better exposes judging bias relevant to human rubric design.