Day 77: Consolidation Review - State, Retries & Failure Taxonomy

01 120 min

Retrieval Drill: Run-State and Execution-Limit Vocabulary

Agent-Assisted Engineering Check

Prerequisites

Day 76 state and retry mechanics; Day 73-75 tool protocols.

Learning Objective

Recall closed-book definitions of the week's control vocabulary - step budget, retry budget, trace, termination, checkpoint, idempotent effect, disposable workspace - then verify each against code rather than memory vibes.

Instructions

1. Closed book: write a one-line definition for each of: max_steps, max_retries, trace, termination, checkpoint, idempotent effect, disposable workspace.
2. Self-mark against the authoritative definitions: project.json controls (max_steps=6, max_retries=2, tool_payload_bytes=4096), the agent.py RunResult fields, and the cli.py TemporaryDirectory workspace.
3. Record which terms were fuzzy; schedule those for tomorrow's recall (spaced retrieval beats re-reading).
4. Distinguish the look-alikes: retry budget (failed observations tolerated) vs step budget (total iterations allowed); checkpoint (persisted state enabling resume) vs trace (the record itself); timeout (one blocking wait) vs deadline (end-to-end bound).

Practice & Prediction Log

Score yourself out of 7 on the written definitions before revealing the reference list; log every miss to your retrieval queue.

Mark as Completed

Independent Mastery Criterion

Write all seven definitions unaided and self-score honestly before checking the code; 6/7 minimum.

Authoritative Source Resources & Provenance

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 22: Addressing Cascading Failures

Author: Mike Ulrich, in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Preventing Server Overload: Retries; Latency and Deadlines

Verified Version & Date: O'Reilly 2016 free online edition (Verified: 2026-10-30)

Selection Rationale:

Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.

Source Prerequisites:

Retries, queueing, and load-shedding basics.

Safe Fallback (Offline Resilience):

The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.

Compared Candidate Sources & Selection Analysis (2)

Release It! (Michael Nygard) - Stability patterns ↗ compared_alternative [Commercial book]

Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.

AWS Builder's Library: Timeouts, retries, and backoff with jitter ↗ compared_alternative [Amazon site content]

Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.

linking Amazon Web Services site content / linked reference noncommercial

Exponential Backoff And Jitter

Author: Marc Brooker, AWS Architecture Blog | Original Source Link ↗

Relevant Section / Range: Adding Backoff; Adding Jitter (Full Jitter, Equal Jitter, Decorrelated Jitter)

Verified Version & Date: AWS Architecture Blog 2015 (Verified: 2026-10-30)

Selection Rationale:

Detailed practitioner reference proving that full jitter (sleep = random_between(0, min(cap, base*2**attempt))) prevents synchronized retry storms and reduces total work during contention.

Source Prerequisites:

Retries, timeouts, and basic probability of correlated load.

Safe Fallback (Offline Resilience):

The backoff exercise is implemented in pure Python with a local flaky stub; the derivation of jitter benefits is fully reproduced in lesson instructions.

Compared Candidate Sources & Selection Analysis (2)

Google Cloud retry strategy documentation ↗ compared_alternative [CC BY 4.0 / documentation terms]

Product-scoped truncated exponential backoff recipe; Brooker's analysis explains why jitter matters, not just what to configure.

Polly resilience library retry policies (.NET) ↗ compared_alternative [BSD-3-Clause]

Library implementation reference; the AWS article supplies the comparative evidence (with/without jitter) that anchors the lesson.

02 120 min

Failure Taxonomy Drill: Classify Every Injected Fault

Independent Mastery Check (no AI assistance).

Prerequisites

Day 76 runs of all three fault modes.

Learning Objective

Sort each observed failure into retryable-corrected, retry-budget-exhausted, or step-budget-exhausted classes, and name the counter that enforced each outcome.

Instructions

1. Build the taxonomy table with columns: fault | observation error | counter consumed | status | termination | class.
2. Rows from real runs: invalid_arguments ('../outside.txt' rejected, retries=1, then corrected) -> retryable-corrected; missing_note (not_found three times, retries=3 > 2) -> retry-budget-exhausted tool_failure; loop (six identical list_notes calls, retries=0) -> step-budget exhaustion; oversize write_report (>4096 bytes -> size_limit) -> retryable only after shrinking the payload.
3. Retryable vs permanent: invalid_arguments, not_found, and size_limit are permanent for identical arguments - blind retry is waste; the adapter correcting its next action is what made the retry productive.
4. Map to production: only transient faults (timeouts, 503s) justify blind retry with backoff; permanent faults require correction or dedupe (Stripe's idempotent replay is not failure retry).

Practice & Prediction Log

Complete the 4-row taxonomy table, then predict the class and termination for a hypothetical network_timeout fault and justify it with the counters.

Mark as Completed

Independent Mastery Criterion

Given only a fault description, predict the terminal state and which budget is consumed - then verify against a live run.

Authoritative Source Resources & Provenance

linking Amazon Web Services site content / linked reference noncommercial

Exponential Backoff And Jitter

Author: Marc Brooker, AWS Architecture Blog | Original Source Link ↗

Relevant Section / Range: Adding Backoff; Adding Jitter (Full Jitter, Equal Jitter, Decorrelated Jitter)

Verified Version & Date: AWS Architecture Blog 2015 (Verified: 2026-10-30)

Selection Rationale:

Detailed practitioner reference proving that full jitter (sleep = random_between(0, min(cap, base*2**attempt))) prevents synchronized retry storms and reduces total work during contention.

Source Prerequisites:

Retries, timeouts, and basic probability of correlated load.

Safe Fallback (Offline Resilience):

The backoff exercise is implemented in pure Python with a local flaky stub; the derivation of jitter benefits is fully reproduced in lesson instructions.

Compared Candidate Sources & Selection Analysis (2)

Google Cloud retry strategy documentation ↗ compared_alternative [CC BY 4.0 / documentation terms]

Product-scoped truncated exponential backoff recipe; Brooker's analysis explains why jitter matters, not just what to configure.

Polly resilience library retry policies (.NET) ↗ compared_alternative [BSD-3-Clause]

Library implementation reference; the AWS article supplies the comparative evidence (with/without jitter) that anchors the lesson.

linking Stripe site content / linked reference noncommercial

Stripe API Documentation: Idempotent Requests

Author: Stripe, Inc. | Original Source Link ↗

Relevant Section / Range: Idempotent requests (Idempotency-Key header; keys can be removed after at least 24 hours)

Verified Version & Date: Stripe API documentation (Verified: 2026-10-30)

Selection Rationale:

Standard production definition of idempotency keys: Stripe guarantees safe client retries by deduplicating POST requests on a caller-supplied key, the exact mechanism that turns at-least-once retries into effectively-once effects.

Source Prerequisites:

HTTP APIs, retries, and side-effect semantics.

Safe Fallback (Offline Resilience):

Idempotency semantics are demonstrated locally with repeated write_report calls and a sha256 dedupe key, no Stripe account needed.

Compared Candidate Sources & Selection Analysis (2)

IETF draft: The Idempotency-Key HTTP Header Field ↗ compared_alternative [IETF Trust / open access]

Standardization-track header specification; Stripe's live documentation is the more concrete deployed example.

AWS Builder's Library: Making retries safe with idempotent APIs ↗ compared_alternative [Amazon site content]

Deep design treatment of idempotent server design; Stripe provides the tighter client-visible contract cited in the exercise.

03 120 min

Blameless Post-Mortem: The Stuck-Loop Run

Agent-Assisted Engineering Check

Prerequisites

Day 77 Activity 2 taxonomy; trace inspection.

Learning Objective

Write a blameless post-mortem for the loop-fault run: a timeline reconstructed from the trace, exact budget arithmetic, the contributing cause, and checkable prevention action items.

Instructions

1. Structure it like an SRE postmortem: Summary, Impact, Root cause, Timeline (one line per trace step), What went well, Action items.
2. Facts only: six identical list_notes actions; zero error observations; retries stayed 0; the run ended budget_exhausted/step_budget at step 6; no report.txt was produced.
3. Contributing cause: the adapter fault='loop' ignores history and re-emits list_notes; run_task has no duplicate-action or progress detector.
4. Action items must be checkable: 'detect k consecutive identical actions and abort' or 'score tool diversity in evaluate()' - not 'make the model smarter'.
5. Stay blameless: the system allowed exactly the bounded waste it was designed to allow - six steps, then safe termination.

Practice & Prediction Log

Draft postmortem.md citing trace step indices (e.g., 'steps 1-6: list_notes returned ok with identical files list'). Every claim must cite a trace step or a config constant.

Mark as Completed

Independent Mastery Criterion

Deliver a post-mortem where every claim traces to a recorded step or a declared constant - zero speculation.

Authoritative Source Resources & Provenance

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 15: Postmortem Culture: Learning from Failure

Author: John Lunney and Sue Lueder (ed. Gary O'Connor), in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Chapter 15: blameless postmortems, trigger conditions, action items

Verified Version & Date: O'Reilly 2016 free online edition (Verified: 2026-10-30)

Selection Rationale:

Standard blameless post-mortem structure (summary, impact, root cause, timeline, action items) applied to the bounded agent's stuck-loop run.

Source Prerequisites:

Incident timelines, root-cause analysis, and action items.

Safe Fallback (Offline Resilience):

A complete post-mortem template is embedded in the lesson; learners need only the recorded trace to fill it.

Compared Candidate Sources & Selection Analysis (2)

Etsy Code as Craft: Blameless PostMortems ↗ compared_alternative [Blog content / linked reference]

Facilitation guide for live debriefs; the SRE chapter provides the written artifact structure the exercise reproduces.

Atlassian Incident Postmortem templates ↗ compared_alternative [Vendor site content]

Product-oriented templates; the SRE chapter emphasizes evidence-linked claims, which matches the trace-backed drill.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 22: Addressing Cascading Failures

Author: Mike Ulrich, in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | Original Source Link ↗

Relevant Section / Range: Triggering Conditions for Cascading Failures

Verified Version & Date: O'Reilly 2016 free online edition (Verified: 2026-10-30)

Selection Rationale:

Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.

Source Prerequisites:

Retries, queueing, and load-shedding basics.

Safe Fallback (Offline Resilience):

The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.

Compared Candidate Sources & Selection Analysis (2)

Release It! (Michael Nygard) - Stability patterns ↗ compared_alternative [Commercial book]

Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.

AWS Builder's Library: Timeouts, retries, and backoff with jitter ↗ compared_alternative [Amazon site content]

Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.

04 120 min

Consolidation & Assessment Review: Bounded-Run Evidence Pack

Independent Mastery Check (no AI assistance).

Prerequisites

All Day 76-77 activities.

Learning Objective

Assemble the week's evidence pack - a fresh CLI run artifact, the fault taxonomy table, transition-audit output, and an honest mastery checklist - reproducible by a reviewer from a clean directory.

Instructions

1. Regenerate everything: rerun the CLI evaluate into a fresh temp directory; re-run the three fault adapters; re-execute your transition audit.
2. Assemble evidence_pack.json: {run_report, taxonomy_table, audit_rows, checklist}.
3. Checklist items (mark met/not-met honestly): name the three terminal states; replay a checkpointed prefix; classify each fault; explain the idempotent write; bound the retry policy; state the kill boundary honestly.
4. A claim with no reproducible artifact stays marked not-met - that is honest review, not failure.

Practice & Prediction Log

Produce evidence_pack.json plus a README command list that lets a reviewer regenerate every artifact in under two minutes.

Mark as Completed

Independent Mastery Criterion

A reviewer can re-run your listed commands and reproduce every claim; nothing is asserted from memory.

Authoritative Source Resources & Provenance

linking PSF License Version 2

Python Standard Library: subprocess - Subprocess Management

Author: Python Software Foundation | Original Source Link ↗

Relevant Section / Range: Using the subprocess Module (subprocess.run timeout) and Notes: Timeout Behavior

Verified Version & Date: Python 3.13 documentation (Verified: 2026-10-30)

Selection Rationale:

Normative reference for subprocess.run(timeout=...), TimeoutExpired, DEVNULL stdin, and the documented kill scope: timeout terminates the child process; it does not define process-tree cleanup semantics.

Source Prerequisites:

Processes, arguments, exit codes, and context managers.

Safe Fallback (Offline Resilience):

The kill-boundary experiment runs entirely in the standard library, so the documented semantics are exercised offline.

Compared Candidate Sources & Selection Analysis (2)

POSIX standard: process creation and termination ↗ compared_alternative [IEEE/Open Group standard]

Formal POSIX semantics; the Python docs map directly onto the commands learners type.

psutil process-tree utilities ↗ compared_alternative [BSD-3-Clause]

Third-party tree enumeration; the exercise deliberately demonstrates the stdlib boundary before reaching for helpers.

linking MIT (LangGraph project) / documentation site terms

LangGraph Persistence: Checkpoints, Threads, and State Snapshots

Author: LangChain / LangGraph Documentation Team | Original Source Link ↗

Relevant Section / Range: N/A

Verified Version & Date: LangGraph documentation (Verified: 2026-10-30)

Selection Rationale:

Primary reference for agent-run checkpointing: LangGraph checkpointers persist graph state at every super-step so interrupted runs resume from the last committed snapshot, the production analogue of this project's action/observation trace.

Source Prerequisites:

Agent loop fundamentals, tool calling, and JSON state serialization.

Safe Fallback (Offline Resilience):

The checkpoint/replay exercise is implemented directly against the projects/agent_learning trace JSON; no framework installation is required.

Compared Candidate Sources & Selection Analysis (2)

Temporal Durable Execution documentation ↗ compared_alternative [MIT / documentation terms]

Temporal persists event histories for replay-based recovery of arbitrary workflows; LangGraph targets agent graphs specifically and stays closer to the learner's tool-loop mental model.

OpenAI Agents SDK session/state documentation ↗ compared_alternative [MIT / documentation terms]

Provider-scoped SDK sessions; LangGraph's explicit checkpoint model exposes the underlying state-transition semantics more transparently.