Confounder Analysis: Model, Harness, Configuration, and Sample Limits
Agent-Assisted Engineering Check
Prerequisites
Day 82 registered protocol document.
Learning Objective
Enumerate and control the confounders — model identity and pinned revision, decoding configuration, harness and tool surface, prompt context, and sample size — so observed differences are not misattributed to a single cause.
Instructions
Study source: within this activity's existing reading/practice time, use Demystifying evals for AI agents, Types of graders for agents; capability vs. regression evals; repeated-trial reliability, then explain one connection to the local task in your own words. Keep the local exercise and acceptance checks; use its original explanation if the external source is unavailable. 1. Build a confounder register with columns: variable, observed value, controlled-or-free, and likely effect direction. Include model id + revision, temperature/seed, context length, tool schema versions, retry policy, and wall-clock variance.
2. Quantify sample limits: on a six-task suite one flipped outcome changes the success rate by ~17 percentage points; compute a Wilson interval for 6/6 to see how wide the honest range is.
3. Write the reporting rule: with small n, report raw counts and intervals, never absolute tool rankings.
4. Separate harness effects from model effects in a two-axis comparison table; mark cells you cannot isolate as 'confounded' rather than guessing.
Practice & Prediction Log
Produce confounder_register.md listing at least six variables with controlled/free status, plus the sample-size calculation showing why 6 tasks cannot support a ranking claim.
A difference with n=6 and no controls is an anecdote with arithmetic, not a measurement.
register = [
{'variable':'model_id+revision','status':'free — not yet pinned'},
{'variable':'temperature/seed','status':'free'},
{'variable':'tool_schema_version','status':'controlled by contract'},
{'variable':'retry_policy','status':'controlled by contract'},
{'variable':'sample_size','status':'n=6 → one flip ≈ ±17pp'}
]
Explain which variables a comparison controls and which it leaves free, and refuse ranking claims the sample cannot support.
Authoritative Source Resources & Provenance
Demystifying evals for AI agents
Relevant Section / Range: Types of graders for agents; capability vs. regression evals; repeated-trial reliability
Verified Version & Date: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
Compared Candidate Sources & Selection Analysis (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Relevant Section / Range: Section 3 τ-bench: A benchmark for Tool-Agent-User Interaction (pass^k definition); Section 4.2 Key Characteristics; Section 5 Experiments (pass^k results)
Verified Version & Date: ICLR 2025 / arXiv:2406.12045v2 (Verified: 2026-10-30)
Introduces pass^k consistency metrics and database-state outcome checks for tool-agent-user interaction, separating reliable task completion from single lucky runs.
Tool-calling APIs, conversational agents, and policy-following evaluation.
Consistency metrics and state-based outcome checking are re-implemented on the curriculum deterministic fixture for offline practice.
Compared Candidate Sources & Selection Analysis (2)
Stateful tool-use evaluation with milestone events; tau-bench additionally models a simulated user and reports pass^k reliability.
Live leaderboard for function-call correctness across languages and parallel calls; tau-bench adds policy adherence and multi-trial consistency.
AgentBench: Evaluating LLMs as Agents
Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis
Verified Version & Date: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
Compared Candidate Sources & Selection Analysis (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.