87 / 90

第 87 天：综合项目验收 —— 评测、证据与评审

运行固定评测，组装脱敏证据包，撰写事后复盘，并按双重评分细则提交。学习者项目验收不同于平台课程就绪门禁。

AI Agent 与工程实践 480 分钟有效学习时间（8 小时）
01 120 分钟

固定评测规程与诚实指标

Agent 辅助工程检查

前置要求

第 86 天的集成运行壳与基线台账。

学习目标

在任何调优运行之前冻结评测：任务顺序、指标定义，以及把模拟结果与真实结果隔离的证据类别（deterministic-fixture 与 live-model-run）。

操作指引

1. 重开 capstone_acceptance.json：任务集、指标与上限现已冻结——事后修改将使评测作废。
2. 按 live_eval_contract.json 的方式定义逐任务报告：status、steps、retries、终止、报告哈希；每个 live 字段标注 evidence_kind=live-model-run，每个夹具字段标注 evidence_kind=deterministic-fixture。
3. 预先声明比较规则：确定性运行必须与基线台账一致；已授权的真机运行逐任务与台账对照，差异如实记录而非隐藏。
4. 诚实地规划覆盖：至少一个成功任务，加全部畸形请求与失败任务——绝不只挑好看的例子。
5. 显式处理缺失证据：未运行的任务标 status=not-run，绝不悄悄丢弃。
6. 撰写 eval_protocol.md：评审者可用以审计结果表的冻结规则。

自主演练与记录

撰写 eval_protocol.md，包含冻结任务清单、指标表、evidence_kind 标注规则与 not-run 策略；然后在运行任何东西之前预测结果表的形状。

独立掌握标准

交付一份规程，使结果、回退与缺失证据都有预定义且可审计的位置。

权威源资源与权利证明

linking Open Access / CC BY 4.0

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

作者 / 机构: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R. Narasimhan | 原始出处链接 ↗

对应章节 / 时间戳: Sections 3-5: task construction from merged PRs, FAIL_TO_PASS/PASS_TO_PASS metrics

验证版本与日期: ICLR 2024 / arXiv:2310.06770v3 (Verified: 2026-10-30)

入选理由:

Defines execution-verified agent evaluation with held-out real tasks; motivates the capstone rule that outcomes are measured by test state transitions, not plausible output.

来源前置条件:

Software testing, issue resolution tasks, and benchmark design.

安全回退方案（离线弹性）:

The capstone's per-task status/steps/termination metrics are defined locally in the evaluation protocol.

对比候选资源与选型分析 (2)
Chen et al.: Evaluating Large Language Models Trained on Code (HumanEval) ↗ compared_alternative [Open Access]

Function-level synthesis tasks; SWE-bench's repository-level issue tasks better match agent evaluation with tool use.

OpenAI: SWE-bench Verified ↗ compared_alternative [All Rights Reserved (blog); dataset terms separate]

Human-validated subset of SWE-bench; cited as refinement, while the original paper defines the protocol.

linking Open Access / CC BY 4.0

AgentBench: Evaluating LLMs as Agents

作者 / 机构: Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. | 原始出处链接 ↗

对应章节 / 时间戳: Section 4.1 Evaluation Setup and Section 4.3 Analysis

验证版本与日期: ICLR 2024 / arXiv:2308.03688v4 (Verified: 2026-10-30)

入选理由:

Systematic multi-environment agent benchmark; supplies the capstone's fixed-task-set discipline and per-task scoring decomposition used by the acceptance contract.

来源前置条件:

Agent loops, tool use, and benchmark design fundamentals.

安全回退方案（离线弹性）:

The capstone task set is fixed locally in the acceptance contract and does not require downloading the benchmark.

对比候选资源与选型分析 (2)
Zhou et al.: WebArena: A Realistic Web Environment for Building Autonomous Agents ↗ compared_alternative [Open Access]

Deep single-domain web tasks with hosted infrastructure; AgentBench's multi-environment view better fits a general agent evaluation.

Mialon et al.: GAIA: a benchmark for General AI Assistants ↗ compared_alternative [Open Access]

Assistant-level real-world questions; also cited by the capstone for outcome-level task design.

02 120 分钟

综合项目证据包组装

独立掌握检查（无 AI 辅助）

前置要求

第 87 天活动 1 的冻结规程；第 85-86 天全部产物。

学习目标

组装评审者可端到端审计的脱敏证据包：章程、契约、基线台账、逐任务结果、失败日志、资源核算、出处表与隐私检查。

操作指引

1. 创建 capstone_evidence/，在 index.md 中列出每个产物及其证明的内容。
2. 附上：charter_review.json、capstone_acceptance.json、baseline_ledger.md、逐任务结果表、fault_drill.json、delivery_log.md。
3. 加入资源核算：每次运行的墙钟时间、总步数/重试次数；token 与费用字段在授权真机运行前保持 null；按预训练证据计划注明持续的存储费用。
4. 建立出处表：模型 id＋许可证、数据集来源＋哈希、引用的来源注册表 id、仓库许可证（代码 Apache-2.0、内容 CC BY-SA 4.0）。
5. 执行隐私检查：扫描文件内容中的令牌、密钥、邮箱与绝对 home 路径，打包前完成脱敏——提供方标识符只留在私人笔记中，绝不入库。
6. 验证：从 index.md 出发的评审者能一跳到达每条主张的证据。

自主演练与记录

产出 capstone_evidence/index.md 与出处表，至少引用五个参照产物（Agent 项目、LoRA 对比契约、live 评测契约、预训练证据包、CI 练习运行器），并记录隐私扫描结果。

独立掌握标准

组装的包中每条主张都能解析到已存证据，出处表与隐私表完整。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

Datasheets for Datasets

作者 / 机构: Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford | 原始出处链接 ↗

对应章节 / 时间戳: Section 3 Questions and Workflow (3.1 Motivation through 3.7 Maintenance)

验证版本与日期: Communications of the ACM 64(12) / arXiv:1803.09010v8 (Verified: 2026-10-30)

入选理由:

Standard questionnaire for documenting dataset origin, composition, rights, and maintenance; structures the capstone evidence package's data-provenance table.

来源前置条件:

Dataset provenance, splits, and licensing vocabulary.

安全回退方案（离线弹性）:

The provenance table template in the capstone instructions restates the required disclosure fields locally.

对比候选资源与选型分析 (2)
Mitchell et al.: Model Cards for Model Reporting ↗ compared_alternative [Open Access]

Model-side reporting rather than dataset-side; the capstone provenance table cites both.

Bender & Friedman: Data Statements for Natural Language Processing ↗ compared_alternative [CC BY 4.0 (ACL Anthology, 2016 or later)]

NLP-specific demographic disclosure; Datasheets cover the general lifecycle needed for arbitrary capstone data.

linking arXiv non-exclusive license / Open Access

Model Cards for Model Reporting

作者 / 机构: Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru | 原始出处链接 ↗

对应章节 / 时间戳: Section 4 Model Card Sections (4.1-4.9)

验证版本与日期: FAT* 2019 / arXiv:1810.03993v3 (Verified: 2026-10-30)

入选理由:

Standard framework for disclosing a model's intended use, evaluation conditions, and limitations; supplies the capstone's model-choice and provenance documentation pattern.

来源前置条件:

Basic model evaluation vocabulary: dataset, metric, intended use.

安全回退方案（离线弹性）:

The curriculum provides a local model-decision table template covering intended use, evaluation data, and caveats.

对比候选资源与选型分析 (2)
Gebru et al.: Datasheets for Datasets ↗ compared_alternative [Open Access]

Dataset-side documentation rather than model-side; used alongside Model Cards in the provenance table rather than as a replacement.

OpenAI GPT-4 System Card ↗ compared_alternative [All Rights Reserved (web publication)]

Vendor-specific safety disclosure; Model Cards provide the general reusable schema.

03 120 分钟

事后复盘、失败分类与受控恢复

Agent 辅助工程检查

前置要求

第 87 天活动 1-2。

学习目标

撰写不追责的综合项目事后复盘：失败分类法、对未达标准与回退的诚实处理、真机与模拟的范围区分，以及一次演示过的受控恢复。

操作指引

1. 把观测到的全部失败归入分类法：malformed_tool_request、missing_input、loop、budget_exhausted、provider_or_infra、evaluation_defect。
2. 每类记录症状、轨迹指针、根因、修复或已接受的局限。未达成的验收标准保持未达成——标记它，不要用叙述掩盖。
3. 显式声明真机与模拟范围：确定性结果证明的是运行壳行为；live 模型字段在授权前保持 null 且 blocked——不把模拟当真实运行展示。
4. 端到端演示一次受控恢复：破坏一个夹具，观测健康检查失败，恢复已知良好的产物，用哈希再验证；附前后哈希。
5. 记录残余风险：账单/存储的后续跟进、陈旧内容的再验证、评审者无法核查之处。
6. 保存 postmortem.md 并从证据索引链接。

自主演练与记录

产出 postmortem.md：含六类分类表、一次带哈希的实际恢复、显式的未达标准清单（或字面写 none）、以及残余风险段落。

独立掌握标准

交付一份失败均有证据分类、未达标准保持可见、恢复被证明而非声称的事后复盘。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

作者 / 机构: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. | 原始出处链接 ↗

对应章节 / 时间戳: Sections 3-4: judge biases (position, verbosity, self-enhancement), rubric design

验证版本与日期: NeurIPS 2023 / arXiv:2306.05685v3 (Verified: 2026-10-30)

入选理由:

Documents judge bias and rubric pitfalls; informs the capstone's separate independent/assisted rubrics and its rule that verdicts must cite stored evidence rather than impressions.

来源前置条件:

Evaluation metrics, pairwise comparison, and grading rubrics.

安全回退方案（离线弹性）:

The capstone rubric is a fixed checklist scored against artifacts; it does not depend on the paper's arena data.

对比候选资源与选型分析 (2)
Liang et al.: Holistic Evaluation of Language Models (HELM) ↗ compared_alternative [Open Access / CC BY 4.0]

Broad scenario taxonomy; MT-Bench's judge-bias analysis is more relevant to rubric-based acceptance review.

Dubois et al.: AlpacaEval / AlpacaFarm ↗ compared_alternative [Open Access]

Automatic instruction-following win-rate; MT-Bench better exposes judging bias relevant to human rubric design.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 15: Postmortem Culture: Learning from Failure

作者 / 机构: John Lunney and Sue Lueder (ed. Gary O'Connor), in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | 原始出处链接 ↗

对应章节 / 时间戳: Chapter 15: blameless postmortems, trigger conditions, action items

验证版本与日期: O'Reilly 2016 / sre.google online edition (Verified: 2026-10-30)

入选理由:

Establishes the blameless, evidence-first post-mortem format; structures the capstone's failure taxonomy, unmet-criteria disclosure, and recovery proof.

来源前置条件:

Incident vocabulary, monitoring, and delivery pipelines.

安全回退方案（离线弹性）:

The post-mortem template (taxonomy, unmet criteria, recovery evidence, residual risk) is fully specified in the lesson.

对比候选资源与选型分析 (2)
John Allspaw / Etsy: Blameless PostMortems and a Just Culture (Code as Craft) ↗ compared_alternative [All Rights Reserved (blog)]

Foundational just-culture essay; the SRE chapter adds the trigger conditions and action-item tracking used by the capstone.

Google SRE Workbook: Postmortem practices ↗ compared_alternative [CC BY-NC-ND 4.0]

Practitioner workbook follow-up; the original chapter is the stable reference for the taxonomy exercise.

04 120 分钟

考核：双重评分细则验收与提交

独立掌握检查（无 AI 辅助）

前置要求

第 87 天活动 1-3。

学习目标

用两套独立评分细则评定综合项目——独立完成部分适用独立掌握度检查细则，指挥完成部分适用 Agent 辅助工程检查细则——然后通过项目提交表单提交证据包。

操作指引

1. 对独立完成的工作（章程评审门、基线台账、证据组装、本次评审）适用独立细则：按正确性＋证据指针逐项评分。
2. 对指挥完成的工作（集成设计、CI 演练、事后复盘）适用辅助细则：按指挥质量、diff 检查、纠错处理评分——绝不只看 Agent 输出。
3. 核验分离规则：独立掌握度不得由辅助输出推断；两套细则的总分与适用范围分别记录。
4. 在下方表单提交：仓库链接、报告链接，以及概括复跑命令与观测哈希的笔记——粘贴的日志属自我报告证据，表单也如此标注。
5. 区分验收层级：本学习者项目验收评定的是你的工作；平台课程就绪门禁（全部 90 天、四个语言版本、平台行为，工单 61）是另一项更晚的验收——综合项目通过并不等于宣布课程就绪。
6. 把最终判定记录到 capstone_verdict.json。

自主演练与记录

在下方表单提交综合项目证据（仓库链接、报告链接、笔记），然后产出 capstone_verdict.json，包含两套细则得分、各自适用范围，以及区分项目验收与课程就绪的明确声明。

学习项目提交与证据

提示：粘贴的执行日志与文字说明属于自述证据，不自动构成独立检查或自动化测试结果。

0 / 4096 字节
声明的辅助条件

注：无 AI 辅助属于诚实声明条件，而非平台监控。

当前提交的证据详情

暂无提交记录。
独立掌握标准

仅当两套细则分别按证据评分、提交被诚实记录、且判定中写明课程就绪门禁的区别时方可通过。

权威源资源与权利证明

linking arXiv non-exclusive license / Open Access

GAIA: a benchmark for General AI Assistants

作者 / 机构: Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom | 原始出处链接 ↗

对应章节 / 时间戳: Sections 3-4: level design, tooling requirements, human vs agent gap

验证版本与日期: arXiv:2311.12983v3 (Verified: 2026-10-30)

入选理由:

Defines graded assistant tasks with unambiguous factual scoring; motivates the capstone's distinction between success tasks, malformed-request handling, and failure cases.

来源前置条件:

Agent capabilities, tool use, and evaluation metrics.

安全回退方案（离线弹性）:

The capstone evaluation tasks are defined in the local acceptance contract and need no external download.

对比候选资源与选型分析 (2)
Liu et al.: AgentBench: Evaluating LLMs as Agents ↗ compared_alternative [Open Access / CC BY 4.0]

Multi-environment execution benchmark; GAIA adds graded real-world assistance tasks, and both inform the fixed task taxonomy.

Zhou et al.: WebArena ↗ compared_alternative [Open Access]

Web-only task environment; GAIA covers general assistant capability beyond browsing.

linking arXiv non-exclusive license / Open Access

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

作者 / 机构: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. | 原始出处链接 ↗

对应章节 / 时间戳: Sections 3-4: judge biases (position, verbosity, self-enhancement), rubric design

验证版本与日期: NeurIPS 2023 / arXiv:2306.05685v3 (Verified: 2026-10-30)

入选理由:

Documents judge bias and rubric pitfalls; informs the capstone's separate independent/assisted rubrics and its rule that verdicts must cite stored evidence rather than impressions.

来源前置条件:

Evaluation metrics, pairwise comparison, and grading rubrics.

安全回退方案（离线弹性）:

The capstone rubric is a fixed checklist scored against artifacts; it does not depend on the paper's arena data.

对比候选资源与选型分析 (2)
Liang et al.: Holistic Evaluation of Language Models (HELM) ↗ compared_alternative [Open Access / CC BY 4.0]

Broad scenario taxonomy; MT-Bench's judge-bias analysis is more relevant to rubric-based acceptance review.

Dubois et al.: AlpacaEval / AlpacaFarm ↗ compared_alternative [Open Access]

Automatic instruction-following win-rate; MT-Bench better exposes judging bias relevant to human rubric design.