Evaluation Pre-Registration: Fixed Task Suites, Baselines, and Declared Hypotheses
Agent-Assisted Engineering Check
Prerequisites
Day 81 agent-assisted delivery practice and the ticket-23 evidence contract files.
Learning Objective
Design a pre-registered evaluation protocol that fixes the task suite, the baseline agent, falsifiable hypotheses, and stop conditions before any measured run, following the structure of live_eval_contract.json.
Instructions
Study source: within this activity's existing reading/practice time, use Demystifying evals for AI agents, The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite, then explain one connection to the local task in your own words. Keep the local exercise and acceptance checks; use its original explanation if the external source is unavailable. 1. Read projects/agent_learning/evidence/live_eval_contract.json and report.en.md; note status blocked-awaiting-authorization.
2. Fix the task suite: enumerate the six contract tasks (live-summarize-notes-success, live-invalid-arguments-recovery, live-unknown-tool-request, live-missing-note-failure, live-step-budget-exhaustion, live-termination-honesty), each with its deterministic_analog fixture task.
3. Declare the baseline: the deterministic-v1 fixture run is the baseline; the live model is the unevaluated candidate. No live claim exists until authorization is recorded.
4. Pre-register falsifiable hypotheses (e.g., 'candidate completes live-summarize-notes-success with report SHA-256 equal to the fixture expectation') and explicit stop conditions (authorization gate fails → no live run).
5. Commit the protocol to eval_protocol.md before any execution; date every later amendment.
Practice & Prediction Log
Author eval_protocol.md fixing the six-task suite, the deterministic-v1 baseline, two falsifiable hypotheses, and the authorization stop condition.
Copy the six task_ids verbatim from live_eval_contract.json; write each hypothesis as a metric assertion that could fail.
protocol = {
tasks: ['live-summarize-notes-success','live-invalid-arguments-recovery','live-unknown-tool-request','live-missing-note-failure','live-step-budget-exhaustion','live-termination-honesty'],
baseline: 'deterministic-v1 fixture',
hypotheses: ['outcome_passed == baseline', 'tool_behavior_passed == baseline'],
stop: 'validate_contract.py --require-authorized must pass before any live run'
}
Produce a pre-registered protocol whose metrics are checkable independently of the agent's own narrative.
Authoritative Source Resources & Provenance
Demystifying evals for AI agents
Relevant Section / Range: The structure of an evaluation: task, trial, grader, transcript, outcome, harness and suite
Verified Version & Date: Public edition checked 2026-11-01; moving pages may change
Teaching-oriented task/trial/grader/outcome distinctions and a staged eval-building workflow complement the original benchmark definitions. No vendor product is required for the local exercises.
Bounded agent loops, task outcomes, traces and deterministic acceptance tests.
Use the original localized explanations, exercises and checks already present in this activity; no external account or purchase is required for that fallback.
Compared Candidate Sources & Selection Analysis (2)
Retained for issue-to-patch fail-to-pass definitions; not a general teaching workflow.
Retained for the exact pass^k definition; practitioner guidance cannot replace the paper's metric semantics.
AgentBench: Evaluating LLMs as Agents
Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis
Verified Version & Date: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
Compared Candidate Sources & Selection Analysis (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.
OpenAI Evals: Framework for Evaluating LLMs and Contamination Checks
Relevant Section / Range: README sections 'Running evals' and 'Writing evals'
Verified Version & Date: OpenAI Evals Repository (Verified: 2026-10-30)
Standardized evaluation repository specifying rigorous separation of eval prompts, canary GUID strings, and contamination test protocols.
Python, evaluation harnesses (MMLU, GSM8K), JSONL datasets.
Canary GUID matching, token set intersection, and leak detection scripts are implemented in the curriculum.
Compared Candidate Sources & Selection Analysis (2)
Widely adopted evaluation harness; OpenAI Evals provides explicit canary guidance and contamination check specifications.
Modern lightweight eval framework; OpenAI Evals illustrates standard test-set integrity guarantees.