Day 76: Agent Run States, Checkpoints & Idempotent Effects

01 120 min

Agent Run State Machine and the Action/Observation Trace

Agent-Assisted Engineering Check

Prerequisites

Day 73-75 agent loop and tool-protocol work, plus the Ticket 22 bounded agent project at projects/agent_learning.

Learning Objective

Identify the complete run state the bounded agent carries - the action/observation trace, the step counter, and the retry counter - and enumerate its three legal terminal states: goal_reached, tool_failure, step_budget.

Instructions

1. Open projects/agent_learning/agent.py and read run_task.
- Mutable state per run: trace (an append-only list of {action, observation} pairs), retries (count of observations with ok=false), and the implicit step counter len(trace).
- Terminal transitions: write_report returning ok=true -> status=completed, termination=goal_reached; retries exceeding max_retries (=2) -> status=failed, termination=tool_failure; the loop ending with neither -> status=budget_exhausted, termination=step_budget.
2. Note what is deliberately absent: no chain-of-thought field and no hidden scratchpad - the trace is the entire persisted state (project.json evidence.forbidden_fields lists thought and chain_of_thought).
3. Compare with production checkpointing: LangGraph checkpointers persist graph state at every super-step; Temporal rebuilds state by replaying an event history. Our trace is a minimal event log of the same idea.
4. Run python -m projects.agent_learning.cli evaluate --fixture summarize-notes --output /tmp/agent-run.json and inspect result.status and result.termination.

Practice & Prediction Log

Predict the four-step trace for the nominal run before executing, then run the CLI and mark each step's transition (action proposed -> observation recorded -> counters updated). Confirm the run ends with steps=4, retries=0, termination=goal_reached.

Mark as Completed

Independent Mastery Criterion

Without executing code, name which counter increments on a failed observation (retries), which bound ends a healthy run (max_steps=6), and the only action whose success terminates the run early (write_report).

Authoritative Source Resources & Provenance

linking MIT (LangGraph project) / documentation site terms

LangGraph Persistence: Checkpoints, Threads, and State Snapshots

Author: LangChain / LangGraph Documentation Team | Original Source Link ↗

Relevant Section / Range: Persistence: Checkpointer vs. store (checkpoints keyed by thread_id); the page now redirects to docs.langchain.com

Verified Version & Date: LangGraph documentation (Verified: 2026-10-30)

Selection Rationale:

Primary reference for agent-run checkpointing: LangGraph checkpointers persist graph state at every super-step so interrupted runs resume from the last committed snapshot, the production analogue of this project's action/observation trace.

Source Prerequisites:

Agent loop fundamentals, tool calling, and JSON state serialization.

Safe Fallback (Offline Resilience):

The checkpoint/replay exercise is implemented directly against the projects/agent_learning trace JSON; no framework installation is required.

Compared Candidate Sources & Selection Analysis (2)

Temporal Durable Execution documentation ↗ compared_alternative [MIT / documentation terms]

Temporal persists event histories for replay-based recovery of arbitrary workflows; LangGraph targets agent graphs specifically and stays closer to the learner's tool-loop mental model.

OpenAI Agents SDK session/state documentation ↗ compared_alternative [MIT / documentation terms]

Provider-scoped SDK sessions; LangGraph's explicit checkpoint model exposes the underlying state-transition semantics more transparently.

linking Temporal Technologies copyright (documentation; Temporal server code is MIT) / documentation site terms

Temporal Documentation: Durable Execution and Event History Replay

Author: Temporal Technologies Documentation Team | Original Source Link ↗

Relevant Section / Range: What is Temporal?: Durable Execution

Verified Version & Date: Temporal documentation (Verified: 2026-10-30)

Selection Rationale:

Standard industrial reference for durable execution: workflow state is rebuilt by replaying a persisted event history through deterministic code, exactly the checkpoint/replay mechanism learners reproduce against the agent trace.

Source Prerequisites:

State machines, event logs, and deterministic replay concepts.

Safe Fallback (Offline Resilience):

Replay semantics are exercised offline against the deterministic adapter; Temporal itself is never installed.

Compared Candidate Sources & Selection Analysis (2)

LangGraph Persistence documentation ↗ compared_alternative [MIT / documentation terms]

Agent-specific checkpointing; Temporal supplies the general event-sourced replay model that motivates the exercise.

Cadence workflow documentation ↗ compared_alternative [MIT / documentation terms]

Temporal's predecessor with equivalent replay semantics; Temporal's docs are more actively maintained.

02 120 min

Checkpoint and Replay: Rebuild Run State from the Persisted Trace

Independent Mastery Check (no AI assistance).

Prerequisites

Day 76 Activity 1 run-state model; JSON serialization.

Learning Objective

Persist a run's trace as a JSON checkpoint and prove deterministic replay: feeding each recorded prefix back through adapter.next_action reproduces exactly the recorded action.

Instructions

1. Design a checkpoint document: {'task_id': 'summarize-notes', 'steps': 4, 'retries': 0, 'trace': [...]} written to run_checkpoint.json.
2. Implement replay(trace_prefix, adapter): return adapter.next_action(list(trace_prefix)). Because DeterministicModelAdapter chooses its next action solely from history, replaying prefix k must yield trace[k].action.
3. Verify for every k in 0..steps-1. This is event-sourced recovery in miniature: state = fold(history), the same recovery shape Temporal documents for durable execution.
4. Record the honest limit: replay rebuilds the adapter's decision state but does not roll back external effects - report.txt stays written because the file lives outside the trace.

Practice & Prediction Log

Implement save_checkpoint(result, path) and replay(trace_prefix, adapter). Assert replay(trace[:k]) == trace[k]['action'] for every k in the nominal run, then repeat for the invalid_arguments fault run (5 steps).

Mark as Completed

Independent Mastery Criterion

Demonstrate identical replay of every recorded action from checkpointed prefixes and state one real-world limit: side effects outside the trace are not rewound.

Authoritative Source Resources & Provenance

linking Temporal Technologies copyright (documentation; Temporal server code is MIT) / documentation site terms

Temporal Documentation: Durable Execution and Event History Replay

Author: Temporal Technologies Documentation Team | Original Source Link ↗

Relevant Section / Range: What is Temporal?: Durable Execution

Verified Version & Date: Temporal documentation (Verified: 2026-10-30)

Selection Rationale:

Standard industrial reference for durable execution: workflow state is rebuilt by replaying a persisted event history through deterministic code, exactly the checkpoint/replay mechanism learners reproduce against the agent trace.

Source Prerequisites:

State machines, event logs, and deterministic replay concepts.

Safe Fallback (Offline Resilience):

Replay semantics are exercised offline against the deterministic adapter; Temporal itself is never installed.

Compared Candidate Sources & Selection Analysis (2)

LangGraph Persistence documentation ↗ compared_alternative [MIT / documentation terms]

Agent-specific checkpointing; Temporal supplies the general event-sourced replay model that motivates the exercise.

Cadence workflow documentation ↗ compared_alternative [MIT / documentation terms]

Temporal's predecessor with equivalent replay semantics; Temporal's docs are more actively maintained.

linking MIT (LangGraph project) / documentation site terms

LangGraph Persistence: Checkpoints, Threads, and State Snapshots

Author: LangChain / LangGraph Documentation Team | Original Source Link ↗

Relevant Section / Range: Persistence: Checkpointer vs. store (checkpoints keyed by thread_id); the page now redirects to docs.langchain.com

Verified Version & Date: LangGraph documentation (Verified: 2026-10-30)

Selection Rationale:

Primary reference for agent-run checkpointing: LangGraph checkpointers persist graph state at every super-step so interrupted runs resume from the last committed snapshot, the production analogue of this project's action/observation trace.

Source Prerequisites:

Agent loop fundamentals, tool calling, and JSON state serialization.

Safe Fallback (Offline Resilience):

The checkpoint/replay exercise is implemented directly against the projects/agent_learning trace JSON; no framework installation is required.

Compared Candidate Sources & Selection Analysis (2)

Temporal Durable Execution documentation ↗ compared_alternative [MIT / documentation terms]

Temporal persists event histories for replay-based recovery of arbitrary workflows; LangGraph targets agent graphs specifically and stays closer to the learner's tool-loop mental model.

OpenAI Agents SDK session/state documentation ↗ compared_alternative [MIT / documentation terms]

Provider-scoped SDK sessions; LangGraph's explicit checkpoint model exposes the underlying state-transition semantics more transparently.

03 120 min

Idempotent Tool Effects and the Exactly-Once Illusion

Agent-Assisted Engineering Check

Prerequisites

Day 76 Activities 1-2; the write_report payload contract.

Learning Objective

Explain why re-issuing write_report is safe - its effect is idempotent - and contrast with non-idempotent effects (append, spend) that need dedupe keys or unique constraints.

Instructions

1. Idempotent effect: applying write_report twice with identical content leaves report.txt in the same state - verify by hashing the file after each call.
2. Non-idempotent counter-example: an append_note tool would duplicate lines under retry; a charge_card effect would double-spend - the failure mode Stripe solves with idempotency keys.
3. Budget view: at-least-once delivery (retries) + idempotent effect = effectively-once outcome. Retries still amplify load on degraded systems (SRE cascading-failure model), so the retry budget stays bounded even when retries are safe.
4. Classify each project tool: list_notes (read-only, idempotent), read_note (read-only, idempotent), write_report (overwrite, idempotent by content equality).

Practice & Prediction Log

Verify idempotence live: call _tool_call write_report twice with identical content and hash report.txt after each call. Then sketch an append_note tool and the dedupe key (sha256 of filename+content) that would restore safety.

Mark as Completed

Independent Mastery Criterion

Classify effects as idempotent vs non-idempotent and name the mechanism (dedupe key, unique constraint, or overwrite semantics) that makes each safe under retry.

Authoritative Source Resources & Provenance

linking Stripe site content / linked reference noncommercial

Stripe API Documentation: Idempotent Requests

Author: Stripe, Inc. | Original Source Link ↗

Relevant Section / Range: Idempotent requests (Idempotency-Key header; keys can be removed after at least 24 hours)

Verified Version & Date: Stripe API documentation (Verified: 2026-10-30)

Selection Rationale:

Standard production definition of idempotency keys: Stripe guarantees safe client retries by deduplicating POST requests on a caller-supplied key, the exact mechanism that turns at-least-once retries into effectively-once effects.

Source Prerequisites:

HTTP APIs, retries, and side-effect semantics.

Safe Fallback (Offline Resilience):

Idempotency semantics are demonstrated locally with repeated write_report calls and a sha256 dedupe key, no Stripe account needed.

Compared Candidate Sources & Selection Analysis (2)

IETF draft: The Idempotency-Key HTTP Header Field ↗ compared_alternative [IETF Trust / open access]

Standardization-track header specification; Stripe's live documentation is the more concrete deployed example.

AWS Builder's Library: Making retries safe with idempotent APIs ↗ compared_alternative [Amazon site content]

Deep design treatment of idempotent server design; Stripe provides the tighter client-visible contract cited in the exercise.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 22: Addressing Cascading Failures

Author: Mike Ulrich, in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Preventing Server Overload: Retries

Verified Version & Date: O'Reilly 2016 free online edition (Verified: 2026-10-30)

Selection Rationale:

Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.

Source Prerequisites:

Retries, queueing, and load-shedding basics.

Safe Fallback (Offline Resilience):

The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.

Compared Candidate Sources & Selection Analysis (2)

Release It! (Michael Nygard) - Stability patterns ↗ compared_alternative [Commercial book]

Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.

AWS Builder's Library: Timeouts, retries, and backoff with jitter ↗ compared_alternative [Amazon site content]

Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.

04 120 min

Assessment: Run-State Transition Audit and Ledger Validation

Independent Mastery Check (no AI assistance).

Prerequisites

Day 76 Activities 1-3.

Learning Objective

Build an audit function that re-derives any recorded run's terminal state from its trace and flags illegal transitions, then prove it against all four observed runs.

Instructions

1. audit_run(result) -> {'steps': int, 'retries': int, 'status': str, 'termination': str, 'violations': []}.
2. Rules: steps <= max_steps (6); retries <= max_retries+1 (3), and retries>2 must pair with status='failed'/termination='tool_failure'; status='completed' requires the last action to be write_report with an ok observation; status='budget_exhausted' requires steps==6 with no terminal success.
3. Feed the nominal run plus all three fault traces; assert the expected (status, termination) pairs.
4. Corrupt one trace (for example, clear the write_report observation's ok flag) and show the audit flags the mismatch instead of trusting the declared status field.

Practice & Prediction Log

Implement audit_run and emit one JSON ledger row per recorded run. Confirm the ledger reproduces: completed/goal_reached (4 steps), completed/goal_reached (5 steps, 1 retry), failed/tool_failure (3 steps, 3 retries), budget_exhausted/step_budget (6 steps, 0 retries).

Mark as Completed

Independent Mastery Criterion

Ship a transition audit whose derived verdicts match every recorded run and whose violation flags fire on a deliberately corrupted trace.

Authoritative Source Resources & Provenance

linking MIT (LangGraph project) / documentation site terms

LangGraph Persistence: Checkpoints, Threads, and State Snapshots

Author: LangChain / LangGraph Documentation Team | Original Source Link ↗

Relevant Section / Range: Persistence: Checkpointer vs. store (checkpoints keyed by thread_id); the page now redirects to docs.langchain.com

Verified Version & Date: LangGraph documentation (Verified: 2026-10-30)

Selection Rationale:

Primary reference for agent-run checkpointing: LangGraph checkpointers persist graph state at every super-step so interrupted runs resume from the last committed snapshot, the production analogue of this project's action/observation trace.

Source Prerequisites:

Agent loop fundamentals, tool calling, and JSON state serialization.

Safe Fallback (Offline Resilience):

The checkpoint/replay exercise is implemented directly against the projects/agent_learning trace JSON; no framework installation is required.

Compared Candidate Sources & Selection Analysis (2)

Temporal Durable Execution documentation ↗ compared_alternative [MIT / documentation terms]

Temporal persists event histories for replay-based recovery of arbitrary workflows; LangGraph targets agent graphs specifically and stays closer to the learner's tool-loop mental model.

OpenAI Agents SDK session/state documentation ↗ compared_alternative [MIT / documentation terms]

Provider-scoped SDK sessions; LangGraph's explicit checkpoint model exposes the underlying state-transition semantics more transparently.

linking Stripe site content / linked reference noncommercial

Stripe API Documentation: Idempotent Requests

Author: Stripe, Inc. | Original Source Link ↗

Relevant Section / Range: Idempotent requests (Idempotency-Key header; keys can be removed after at least 24 hours)

Verified Version & Date: Stripe API documentation (Verified: 2026-10-30)

Selection Rationale:

Standard production definition of idempotency keys: Stripe guarantees safe client retries by deduplicating POST requests on a caller-supplied key, the exact mechanism that turns at-least-once retries into effectively-once effects.

Source Prerequisites:

HTTP APIs, retries, and side-effect semantics.

Safe Fallback (Offline Resilience):

Idempotency semantics are demonstrated locally with repeated write_report calls and a sha256 dedupe key, no Stripe account needed.

Compared Candidate Sources & Selection Analysis (2)

IETF draft: The Idempotency-Key HTTP Header Field ↗ compared_alternative [IETF Trust / open access]

Standardization-track header specification; Stripe's live documentation is the more concrete deployed example.

AWS Builder's Library: Making retries safe with idempotent APIs ↗ compared_alternative [Amazon site content]

Deep design treatment of idempotent server design; Stripe provides the tighter client-visible contract cited in the exercise.