混雜分析：模型、Harness、配置與樣本量上限
Agent 輔助工程檢查
前置要求
Day 82 的註冊協議文件。
學習目標
列舉並控制混雜變數——模型身分與固定 revision、解碼配置、Harness 與工具面、提示上下文、樣本量——使觀察到的差異不被錯誤歸因於單一原因。
操作指引
學習來源：在本活動現有的閱讀與練習時段內，參考 Demystifying evals for AI agents 的 Types of graders for agents; capability vs. regression evals; repeated-trial reliability，用自己的話解釋一處與本地任務的聯繫。保留現有練習與驗收檢查；若外部資料無法存取，使用課程自寫的解釋。1. 建立混雜變數登記表，欄位：變數、觀測值、受控/自由、估計影響方向。須包含模型 ID+revision、temperature/seed、上下文長度、工具 schema 版本、重試策略、執行時間波動。
2. 量化樣本量上限：在 6 個任務的套件上，一個結果翻轉會使成功率移動約 17 個百分點；計算 6/6 的 Wilson 區間，看看誠實範圍有多寬。
3. 寫出報告規則：n 小時報告原始計數與區間，絕不做絕對工具排名。
4. 用雙軸對比表把 Harness 效應與模型效應分開；無法隔離的格子標「混雜（confounded）」，不要猜。
自主實作與記錄
產出 confounder_register.md：至少列出六個變數的受控/自由狀態，並附樣本量計算說明為何 6 個任務撐不起排名結論。
n=6、無控制對照的差異是帶算術的軼事，不是測量。
register = [
{'variable':'model_id+revision','status':'free — not yet pinned'},
{'variable':'temperature/seed','status':'free'},
{'variable':'tool_schema_version','status':'controlled by contract'},
{'variable':'retry_policy','status':'controlled by contract'},
{'variable':'sample_size','status':'n=6 → one flip ≈ ±17pp'}
]
說明一次比較控制了哪些變數、放開了哪些變數，並拒絕樣本撐不住的排名結論。
權威來源資源與權利證明
Demystifying evals for AI agents
對應章節 / 時間戳: Types of graders for agents; capability vs. regression evals; repeated-trial reliability
驗證版本與日期: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
對比候選資源與選型分析 (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
對應章節 / 時間戳: Section 3 τ-bench: A benchmark for Tool-Agent-User Interaction (pass^k definition); Section 4.2 Key Characteristics; Section 5 Experiments (pass^k results)
驗證版本與日期: ICLR 2025 / arXiv:2406.12045v2 (Verified: 2026-10-30)
Introduces pass^k consistency metrics and database-state outcome checks for tool-agent-user interaction, separating reliable task completion from single lucky runs.
Tool-calling APIs, conversational agents, and policy-following evaluation.
Consistency metrics and state-based outcome checking are re-implemented on the curriculum deterministic fixture for offline practice.
對比候選資源與選型分析 (2)
Stateful tool-use evaluation with milestone events; tau-bench additionally models a simulated user and reports pass^k reliability.
Live leaderboard for function-call correctness across languages and parallel calls; tau-bench adds policy adherence and multi-trial consistency.
AgentBench: Evaluating LLMs as Agents
對應章節 / 時間戳: Section 4.1 Evaluation Setup and Section 4.3 Analysis
驗證版本與日期: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
對比候選資源與選型分析 (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.