Retry Budgets, Exponential Backoff, and Jitter
Agent-Assisted Engineering Check
Prerequisites
Day 76-77 retry-counter mechanics; basic probability.
Learning Objective
Design a retry policy with a hard attempt budget, exponential backoff, and full jitter; explain how unbounded synchronized retries amplify partial outages into cascading failures.
Instructions
1. The project's policy is deliberately minimal: retries increments on each ok=false observation and the run fails once retries > max_retries=2 - immediate re-issue, no delay.
2. Production policy (AWS Builder's Library): delay = random_between(0, min(cap, base * 2**attempt)); full jitter decorrelates clients and minimizes total work during contention.
3. Cascading-failure model (Google SRE ch.22): retries multiply offered load exactly when the dependency is degraded; a retry budget plus jittered backoff bounds that amplification.
4. Retry only transient faults; permanent faults (invalid_arguments, not_found) need correction - the adapter changes arguments before re-issuing, which is why its retry was productive.
Practice & Prediction Log
Implement retry_with_backoff(fn, attempts=3, base=0.05, cap=0.5) that sleeps a full-jitter interval between tries and re-raises after exhaustion. Drive it with a flaky stub failing twice then succeeding, and a permanent failure that must consume the whole budget.
sleep = random.uniform(0, min(cap, base * 2attempt)) - never a deterministic 2attempt sleep.
import random, time
def retry_with_backoff(fn, attempts=3, base=0.05, cap=0.5):
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(random.uniform(0, min(cap, base * 2**i)))
Explain, with the jitter math, why the project's retry budget (2) and production jittered backoff both bound work - and name the failure each defends (budget exhaustion vs correlated retry storms).
Authoritative Source Resources & Provenance
Exponential Backoff And Jitter
Relevant Section / Range: Adding Backoff; Adding Jitter (Full Jitter, Equal Jitter, Decorrelated Jitter)
Verified Version & Date: AWS Architecture Blog 2015 (Verified: 2026-10-30)
Detailed practitioner reference proving that full jitter (sleep = random_between(0, min(cap, base*2**attempt))) prevents synchronized retry storms and reduces total work during contention.
Retries, timeouts, and basic probability of correlated load.
The backoff exercise is implemented in pure Python with a local flaky stub; the derivation of jitter benefits is fully reproduced in lesson instructions.
Compared Candidate Sources & Selection Analysis (2)
Product-scoped truncated exponential backoff recipe; Brooker's analysis explains why jitter matters, not just what to configure.
Library implementation reference; the AWS article supplies the comparative evidence (with/without jitter) that anchors the lesson.
Site Reliability Engineering, Chapter 22: Addressing Cascading Failures
Relevant Section / Range: Preventing Server Overload: Retries
Verified Version & Date: O'Reilly 2016 free online edition (Verified: 2026-10-30)
Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.
Retries, queueing, and load-shedding basics.
The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.
Compared Candidate Sources & Selection Analysis (2)
Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.
Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.