重试预算、指数退避与抖动
Agent 辅助工程检查
前置要求
第 76-77 天的重试计数器机制;基础概率。
学习目标
设计一个带硬尝试预算、指数退避与全抖动的重试策略;解释无界的同步重试如何把局部故障放大为级联故障。
操作指引
1. 项目的策略刻意保持最小:每次 ok=false 观测使 retries 递增,retries > max_retries=2 时运行失败——立即重发,无延迟。
2. 生产策略(AWS Builder's Library):delay = random_between(0, min(cap, base * 2**attempt));全抖动去相关各客户端,使竞争期的总工作量最小化。
3. 级联故障模型(Google SRE 第 22 章):依赖方降级时重试恰好成倍放大负载;重试预算加抖动退避为这种放大设定了上限。
4. 只对瞬时故障重试;永久故障(invalid_arguments、not_found)需要纠正——适配器先改正参数再重发,这正是其重试有效的原因。
自主演练与记录
实现 retry_with_backoff(fn, attempts=3, base=0.05, cap=0.5):每次尝试间按全抖动间隔休眠,耗尽后重新抛出异常。用一个失败两次后成功的不稳定桩和一个必然耗尽全部预算的永久失败分别驱动它。
sleep = random.uniform(0, min(cap, base * 2attempt))——绝不用确定性的 2attempt 休眠。
import random, time
def retry_with_backoff(fn, attempts=3, base=0.05, cap=0.5):
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(random.uniform(0, min(cap, base * 2**i)))
借助抖动公式说明:项目的重试预算(2)与生产抖动退避都约束了工作量,并指出各自防御的故障(预算耗尽 vs 相关重试风暴)。
权威源资源与权利证明
Exponential Backoff And Jitter
对应章节 / 时间戳: Adding Backoff; Adding Jitter (Full Jitter, Equal Jitter, Decorrelated Jitter)
验证版本与日期: AWS Architecture Blog 2015 (Verified: 2026-10-30)
Detailed practitioner reference proving that full jitter (sleep = random_between(0, min(cap, base*2**attempt))) prevents synchronized retry storms and reduces total work during contention.
Retries, timeouts, and basic probability of correlated load.
The backoff exercise is implemented in pure Python with a local flaky stub; the derivation of jitter benefits is fully reproduced in lesson instructions.
对比候选资源与选型分析 (2)
Product-scoped truncated exponential backoff recipe; Brooker's analysis explains why jitter matters, not just what to configure.
Library implementation reference; the AWS article supplies the comparative evidence (with/without jitter) that anchors the lesson.
Site Reliability Engineering, Chapter 22: Addressing Cascading Failures
对应章节 / 时间戳: Preventing Server Overload: Retries
验证版本与日期: O'Reilly 2016 free online edition (Verified: 2026-10-30)
Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.
Retries, queueing, and load-shedding basics.
The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.
对比候选资源与选型分析 (2)
Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.
Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.