评测预注册：固定任务集、基线与已声明假设
Agent 辅助工程检查
前置要求
Day 81 的 Agent 辅助交付演练及 ticket-23 的证据契约文件。
学习目标
按 live_eval_contract.json 的结构，设计一份预注册评测协议：在任何测量运行之前，固定任务集、基线 Agent、可证伪假设与停止条件。
操作指引
学习来源：在本活动现有的阅读与练习时段内，参考 Demystifying evals for AI agents 的 The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite，用自己的话解释一处与本地任务的联系。保留现有练习与验收检查；若外部资料无法访问，使用课程自写的解释。1. 阅读 projects/agent_learning/evidence/live_eval_contract.json 与 report.en.md，注意其状态为 blocked-awaiting-authorization。
2. 固定任务集：枚举契约中的六个任务（live-summarize-notes-success、live-invalid-arguments-recovery、live-unknown-tool-request、live-missing-note-failure、live-step-budget-exhaustion、live-termination-honesty），并标注各自的 deterministic_analog 夹具任务。
3. 声明基线：deterministic-v1 夹具运行是基线；真实模型是未评测的候选。授权记录出现之前不存在任何在线（live）结论。
4. 预注册可证伪假设（例如“候选以与夹具期望相同的 report SHA-256 完成 live-summarize-notes-success”）和明确的停止条件（授权闸失败 → 不做在线运行）。
5. 在任何执行之前将协议落盘为 eval_protocol.md；此后所有修订都标注日期。
自主演练与记录
撰写 eval_protocol.md，固定六任务集、deterministic-v1 基线、两条可证伪假设与授权停止条件。
从 live_eval_contract.json 逐字复制六个 task_id；每条假设写成可能失败的指标断言。
protocol = {
tasks: ['live-summarize-notes-success','live-invalid-arguments-recovery','live-unknown-tool-request','live-missing-note-failure','live-step-budget-exhaustion','live-termination-honesty'],
baseline: 'deterministic-v1 fixture',
hypotheses: ['outcome_passed == baseline', 'tool_behavior_passed == baseline'],
stop: 'validate_contract.py --require-authorized must pass before any live run'
}
产出一份预注册协议，其指标可独立于 Agent 自身叙述被核验。
权威源资源与权利证明
Demystifying evals for AI agents
对应章节 / 时间戳: The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite
验证版本与日期: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
对比候选资源与选型分析 (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
AgentBench: Evaluating LLMs as Agents
对应章节 / 时间戳: Section 4.1 Evaluation Setup and Section 4.3 Analysis
验证版本与日期: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
对比候选资源与选型分析 (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.
OpenAI Evals: Framework for Evaluating LLMs and Contamination Checks
对应章节 / 时间戳: README sections 'Running evals' and 'Writing evals'
验证版本与日期: OpenAI Evals Repository (Verified: 2026-10-30)
Standardized evaluation repository specifying rigorous separation of eval prompts, canary GUID strings, and contamination test protocols.
Python, evaluation harnesses (MMLU, GSM8K), JSONL datasets.
Canary GUID matching, token set intersection, and leak detection scripts are implemented in the curriculum.
对比候选资源与选型分析 (2)
Widely adopted evaluation harness; OpenAI Evals provides explicit canary guidance and contamination check specifications.
Modern lightweight eval framework; OpenAI Evals illustrates standard test-set integrity guarantees.