評測預註冊：固定任務集、基準與已宣告假設
Agent 輔助工程檢查
前置要求
Day 81 的 Agent 輔助交付實作及 ticket-23 的證據契約檔案。
學習目標
按 live_eval_contract.json 的結構，設計一份預註冊評測協議：在任何測量執行之前，固定任務集、基準 Agent、可否證假設與停止條件。
操作指引
學習來源：在本活動現有的閱讀與練習時段內，參考 Demystifying evals for AI agents 的 The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite，用自己的話解釋一處與本地任務的聯繫。保留現有練習與驗收檢查；若外部資料無法存取，使用課程自寫的解釋。1. 閱讀 projects/agent_learning/evidence/live_eval_contract.json 與 report.en.md，注意其狀態為 blocked-awaiting-authorization。
2. 固定任務集：列舉契約中的六個任務（live-summarize-notes-success、live-invalid-arguments-recovery、live-unknown-tool-request、live-missing-note-failure、live-step-budget-exhaustion、live-termination-honesty），並標註各自的 deterministic_analog 治具任務。
3. 宣告基準：deterministic-v1 治具執行為基準；真實模型是未評測的候選。授權記錄出現之前不存在任何上線（live）結論。
4. 預註冊可否證假設（例如「候選以與治具期望相同的 report SHA-256 完成 live-summarize-notes-success」）與明確停止條件（授權閘失敗 → 不做上線執行）。
5. 在任何執行之前將協議落盤為 eval_protocol.md；此後所有修訂都標註日期。
自主實作與記錄
撰寫 eval_protocol.md，固定六任務集、deterministic-v1 基準、兩條可否證假設與授權停止條件。
從 live_eval_contract.json 逐字複製六個 task_id；每條假設寫成可能失敗的指標斷言。
protocol = {
tasks: ['live-summarize-notes-success','live-invalid-arguments-recovery','live-unknown-tool-request','live-missing-note-failure','live-step-budget-exhaustion','live-termination-honesty'],
baseline: 'deterministic-v1 fixture',
hypotheses: ['outcome_passed == baseline', 'tool_behavior_passed == baseline'],
stop: 'validate_contract.py --require-authorized must pass before any live run'
}
產出一份預註冊協議，其指標可獨立於 Agent 自身敘述被核驗。
權威來源資源與權利證明
Demystifying evals for AI agents
對應章節 / 時間戳: The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite
驗證版本與日期: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
對比候選資源與選型分析 (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
AgentBench: Evaluating LLMs as Agents
對應章節 / 時間戳: Section 4.1 Evaluation Setup and Section 4.3 Analysis
驗證版本與日期: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
對比候選資源與選型分析 (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.
OpenAI Evals: Framework for Evaluating LLMs and Contamination Checks
對應章節 / 時間戳: README sections 'Running evals' and 'Writing evals'
驗證版本與日期: OpenAI Evals Repository (Verified: 2026-10-30)
Standardized evaluation repository specifying rigorous separation of eval prompts, canary GUID strings, and contamination test protocols.
Python, evaluation harnesses (MMLU, GSM8K), JSONL datasets.
Canary GUID matching, token set intersection, and leak detection scripts are implemented in the curriculum.
對比候選資源與選型分析 (2)
Widely adopted evaluation harness; OpenAI Evals provides explicit canary guidance and contamination check specifications.
Modern lightweight eval framework; OpenAI Evals illustrates standard test-set integrity guarantees.