混杂分析：模型、Harness、配置与样本量上限
Agent 辅助工程检查
前置要求
Day 82 的注册协议文档。
学习目标
枚举并控制混杂变量——模型身份与固定 revision、解码配置、Harness 与工具面、提示上下文、样本量——使观察到的差异不被错误归因于单一原因。
操作指引
学习来源：在本活动现有的阅读与练习时段内，参考 Demystifying evals for AI agents 的 Types of graders for agents; capability vs. regression evals; repeated-trial reliability，用自己的话解释一处与本地任务的联系。保留现有练习与验收检查；若外部资料无法访问，使用课程自写的解释。1. 建立混杂变量登记表，列：变量、观测值、受控/自由、估计影响方向。须包含模型 ID+revision、temperature/seed、上下文长度、工具 schema 版本、重试策略、墙钟时间波动。
2. 量化样本量上限：在 6 个任务的套件上，一个结果翻转会使成功率移动约 17 个百分点；计算 6/6 的 Wilson 区间，看看诚实范围有多宽。
3. 写出报告规则：n 小时报告原始计数与区间，绝不做绝对工具排名。
4. 用双轴对比表把 Harness 效应与模型效应分开；无法隔离的格子标“混杂（confounded）”，不要猜。
自主演练与记录
产出 confounder_register.md：至少列出六个变量的受控/自由状态，并附样本量计算说明为何 6 个任务撑不起排名结论。
n=6、无控制对照的差异是带算术的轶事，不是测量。
register = [
{'variable':'model_id+revision','status':'free — not yet pinned'},
{'variable':'temperature/seed','status':'free'},
{'variable':'tool_schema_version','status':'controlled by contract'},
{'variable':'retry_policy','status':'controlled by contract'},
{'variable':'sample_size','status':'n=6 → one flip ≈ ±17pp'}
]
说明一次比较控制了哪些变量、放开了哪些变量，并拒绝样本撑不住的排名结论。
权威源资源与权利证明
Demystifying evals for AI agents
对应章节 / 时间戳: Types of graders for agents; capability vs. regression evals; repeated-trial reliability
验证版本与日期: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
对比候选资源与选型分析 (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
对应章节 / 时间戳: Section 3 τ-bench: A benchmark for Tool-Agent-User Interaction (pass^k definition); Section 4.2 Key Characteristics; Section 5 Experiments (pass^k results)
验证版本与日期: ICLR 2025 / arXiv:2406.12045v2 (Verified: 2026-10-30)
Introduces pass^k consistency metrics and database-state outcome checks for tool-agent-user interaction, separating reliable task completion from single lucky runs.
Tool-calling APIs, conversational agents, and policy-following evaluation.
Consistency metrics and state-based outcome checking are re-implemented on the curriculum deterministic fixture for offline practice.
对比候选资源与选型分析 (2)
Stateful tool-use evaluation with milestone events; tau-bench additionally models a simulated user and reports pass^k reliability.
Live leaderboard for function-call correctness across languages and parallel calls; tau-bench adds policy adherence and multi-trial consistency.
AgentBench: Evaluating LLMs as Agents
对应章节 / 时间戳: Section 4.1 Evaluation Setup and Section 4.3 Analysis
验证版本与日期: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
对比候选资源与选型分析 (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.