Synthesis Review: Evidence Kinds, Pre-Registration, and Baseline Discipline
Agent-Assisted Engineering Check
Prerequisites
Day 83 interpretation memo.
Learning Objective
Retrieve from memory the block's core distinctions — deterministic fixture evidence versus blocked live evidence, pre-registration versus post-hoc storytelling, and outcome metrics versus tool-behavior metrics — before touching any notes.
Instructions
1. Retrieval practice: from memory, write the four metric families, the six contract task ids, and the difference between evidence_kind deterministic and live.
2. Check your recall against live_eval_contract.json; log every miss for the catch-up plan.
3. Summarize the blocked-awaiting-authorization status in one paragraph: what exists (deterministic runs, contract, validator) and what does not (any live measurement).
4. Explain why baseline discipline — freezing the suite, baseline, and hypotheses first — is the block's central habit.
Practice & Prediction Log
Write recall_log.md listing the four metric families, all six task ids, and the deterministic-versus-live evidence rule — produced from memory, then self-graded against the contract.
If you cannot name a task id or metric family from memory, that gap goes on the spaced-review list.
recall = {'metrics':['outcome_passed','tool_behavior_passed','budget_passed','failure_handling_passed'],'tasks':6,'rule':'deterministic fixture evidence != live model evidence; live_value stays null until authorized'}
Accurately reproduce the block's core concepts from memory and honestly log the gaps.
Authoritative Source Resources & Provenance
AgentBench: Evaluating LLMs as Agents
Relevant Section / Range: Section 4.1 Evaluation Setup and Section 4.3 Analysis
Verified Version & Date: ICLR 2024 / arXiv:2308.03688v2 (Verified: 2026-10-30)
First systematic multi-environment benchmark measuring LLM-as-agent ability across eight interactive settings, establishing fixed task suites, environment-specific scoring, and comparable evaluation conventions.
LLM prompting, tool-use environments, and multi-step task decomposition.
The curriculum fixes its own six-task evaluation suite with deterministic-v1 fixture analogs, so all protocol-design practice runs offline without reproducing AgentBench environments.
Compared Candidate Sources & Selection Analysis (2)
Real computer-control benchmark for open-ended desktop tasks; AgentBench covers a wider environment spectrum for foundational evaluation design.
Generalist web-agent benchmark with 2,350 tasks; AgentBench provides broader multi-domain coverage for studying evaluation protocol design.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Relevant Section / Range: N/A
Verified Version & Date: ICLR 2024 / arXiv:2310.06770v3 (Verified: 2026-10-30)
Defines the standard pattern this block teaches: a bounded issue statement paired with executable fail-to-pass tests as verifiable acceptance criteria for agent-produced changes.
Unit testing, issue descriptions, and repository evaluation vocabulary.
The acceptance-criteria pattern is exercised locally through the Ticket 22 fixture scorer and the Ticket 24 regression exercise without external datasets.
Compared Candidate Sources & Selection Analysis (2)
Function-level synthesis benchmark; SWE-bench models the repository-scoped issue-to-diff workflow that this delivery block rehearses.
Extends issue resolution to multiple programming languages; the original Python-centric SWE-bench provides a cleaner pedagogical anchor.