78 / 90

第 78 天:重试预算、超时与诚实的执行边界

用退避与抖动约束重试,强制有限超时,并准确说明项目提供与不提供的隔离能力。

Agent 480 主动学习分钟(8 小时)
01 120 分钟

重试预算、指数退避与抖动

Agent 辅助工程检查

前置要求

第 76-77 天的重试计数器机制;基础概率。

学习目标

设计一个带硬尝试预算、指数退避与全抖动的重试策略;解释无界的同步重试如何把局部故障放大为级联故障。

操作指引

1. 项目的策略刻意保持最小:每次 ok=false 观测使 retries 递增,retries > max_retries=2 时运行失败——立即重发,无延迟。
2. 生产策略(AWS Builder's Library):delay = random_between(0, min(cap, base * 2**attempt));全抖动去相关各客户端,使竞争期的总工作量最小化。
3. 级联故障模型(Google SRE 第 22 章):依赖方降级时重试恰好成倍放大负载;重试预算加抖动退避为这种放大设定了上限。
4. 只对瞬时故障重试;永久故障(invalid_arguments、not_found)需要纠正——适配器先改正参数再重发,这正是其重试有效的原因。

自主演练与记录

实现 retry_with_backoff(fn, attempts=3, base=0.05, cap=0.5):每次尝试间按全抖动间隔休眠,耗尽后重新抛出异常。用一个失败两次后成功的不稳定桩和一个必然耗尽全部预算的永久失败分别驱动它。

独立掌握标准

借助抖动公式说明:项目的重试预算(2)与生产抖动退避都约束了工作量,并指出各自防御的故障(预算耗尽 vs 相关重试风暴)。

权威源资源与权利证明

linking Amazon Web Services site content / linked reference noncommercial

Exponential Backoff And Jitter

作者 / 机构: Marc Brooker, AWS Architecture Blog | 原始出处链接 ↗

对应章节 / 时间戳: Adding Backoff; Adding Jitter (Full Jitter, Equal Jitter, Decorrelated Jitter)

验证版本与日期: AWS Architecture Blog 2015 (Verified: 2026-10-30)

入选理由:

Detailed practitioner reference proving that full jitter (sleep = random_between(0, min(cap, base*2**attempt))) prevents synchronized retry storms and reduces total work during contention.

来源前置条件:

Retries, timeouts, and basic probability of correlated load.

安全回退方案(离线弹性):

The backoff exercise is implemented in pure Python with a local flaky stub; the derivation of jitter benefits is fully reproduced in lesson instructions.

对比候选资源与选型分析 (2)
Google Cloud retry strategy documentation ↗ compared_alternative [CC BY 4.0 / documentation terms]

Product-scoped truncated exponential backoff recipe; Brooker's analysis explains why jitter matters, not just what to configure.

Polly resilience library retry policies (.NET) ↗ compared_alternative [BSD-3-Clause]

Library implementation reference; the AWS article supplies the comparative evidence (with/without jitter) that anchors the lesson.

linking CC BY-NC-ND 4.0 noncommercial no-derivatives

Site Reliability Engineering, Chapter 22: Addressing Cascading Failures

作者 / 机构: Mike Ulrich, in Site Reliability Engineering (eds. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy; Google) | 原始出处链接 ↗

对应章节 / 时间戳: Preventing Server Overload: Retries

验证版本与日期: O'Reilly 2016 free online edition (Verified: 2026-10-30)

入选理由:

Primary analysis of how unbounded retries amplify partial outages into cascading failures, motivating hard retry budgets and deadlines in agent runtimes.

来源前置条件:

Retries, queueing, and load-shedding basics.

安全回退方案(离线弹性):

The retry-amplification scenario and budget arithmetic are fully reproduced in the lesson's worked example.

对比候选资源与选型分析 (2)
Release It! (Michael Nygard) - Stability patterns ↗ compared_alternative [Commercial book]

Broader stability-pattern catalog (circuit breakers, bulkheads); the SRE chapter is free and directly quantifies retry amplification.

Strong companion article; the SRE chapter frames the failure mode (cascading overload) that the retry budget defends against.

02 120 分钟

超时、截止时间与实测终止边界

独立掌握检查(无 AI 辅助)

前置要求

第 77 天的证据包;子进程基础。

学习目标

对子进程强制有限超时,然后实测终止边界:subprocess 超时会终结直接子进程,但无法承诺清理逃逸的子孙进程。

操作指引

1. 本仓库的超时控制:scripts/check_snippets.py 设定 RUN_TIMEOUT=15.0,stdin=DEVNULL,并在临时工作目录中执行;lab/run_all_tests.py 在 subprocess.run(timeout=...) 之前校验超时必须是有限正数(拒绝 bool、0、负数、NaN、inf)。
2. 截止时间 vs 超时:超时约束单次阻塞等待;截止时间约束端到端工作——Agent 循环用步数预算近似截止时间。
3. 终止边界:subprocess.run 在 TimeoutExpired 时杀死子进程;逃逸的子孙(双重 fork 或独立会话)仍然存活——lab 运行器 README 明确写出此缺口:它只终止直接子进程,子孙树清理仍未实现。
4. 进程组是缺失的机制:setpgid/setsid 加 killpg 可让运行器向整组发信号;没有它就没有对逃逸子孙的清理保证。

自主演练与记录

运行一个会启动独立孙进程的子进程(先 subprocess.Popen(['sleep','60']) 再睡眠),设置 timeout=1。捕获 TimeoutExpired 后运行 pgrep -f 'sleep 60' 展示孙进程存活。把存活的 PID 记入证据日志。

独立掌握标准

用实时的进程表演示超时只杀一个进程而非进程树;不声称未经验证的子孙清理。

权威源资源与权利证明

linking PSF License Version 2

Python Standard Library: subprocess - Subprocess Management

作者 / 机构: Python Software Foundation | 原始出处链接 ↗

对应章节 / 时间戳: Using the subprocess Module (subprocess.run timeout) and Notes: Timeout Behavior

验证版本与日期: Python 3.13 documentation (Verified: 2026-10-30)

入选理由:

Normative reference for subprocess.run(timeout=...), TimeoutExpired, DEVNULL stdin, and the documented kill scope: timeout terminates the child process; it does not define process-tree cleanup semantics.

来源前置条件:

Processes, arguments, exit codes, and context managers.

安全回退方案(离线弹性):

The kill-boundary experiment runs entirely in the standard library, so the documented semantics are exercised offline.

对比候选资源与选型分析 (2)
POSIX standard: process creation and termination ↗ compared_alternative [IEEE/Open Group standard]

Formal POSIX semantics; the Python docs map directly onto the commands learners type.

psutil process-tree utilities ↗ compared_alternative [BSD-3-Clause]

Third-party tree enumeration; the exercise deliberately demonstrates the stdlib boundary before reaching for helpers.

linking Linux man-pages license (freely distributable)

Linux man-pages: credentials(7) and setpgid(2) - Process Groups and Sessions

作者 / 机构: Linux man-pages project (Michael Kerrisk et al.) | 原始出处链接 ↗

对应章节 / 时间戳: credentials(7) DESCRIPTION: Process group and session

验证版本与日期: man-pages 6.x online edition (Verified: 2026-10-30)

入选理由:

Normative reference for process-group semantics (PGID, sessions, killpg): the mechanism a runner must use to signal a whole descendant tree, which the lab runner and subprocess timeout path do not employ.

来源前置条件:

POSIX processes, signals, and fork/exec model.

安全回退方案(离线弹性):

Process-group behavior is demonstrated live with os.setsid and process inspection; local man pages cover the same text offline.

对比候选资源与选型分析 (2)
POSIX.1-2017 specification: setpgid ↗ compared_alternative [IEEE/Open Group standard]

Formal standard text; the man page presents the same contract with implementation notes learners can test directly.

Detailed textbook treatment of sessions and job control; the man page is the free, always-current operational reference.

03 120 分钟

隔离分层:校验规则、资源限制与 OS 沙箱

Agent 辅助工程检查

前置要求

第 78 天活动 1-2;项目的完整控制清单。

学习目标

把项目的每项控制放到正确的防御层——AST 门控、参数校验、载荷/步数/重试限制、一次性工作区、命名空间/cgroup、microVM 隔离——并说明各层挡不住的威胁。

操作指引

1. 第 1 层-静态 AST 门控:scripts/check_snippets.py 对每个代码片段执行 ast.parse(拒绝语法错误),classify() 跳过 torch 导入、交互输入与训练代码;AST 规则检查源码结构,而非运行时行为。
2. 第 2 层-参数校验:_tool_call 拒绝未知工具、非映射参数及不满足 Path(filename).name != filename 的文件名(挡住 '../outside.txt' 穿越)。
3. 第 3 层-资源限制:tool_payload_bytes=4096(size_limit)、max_steps=6、max_retries=2——约束消耗量,但不构成隔离。
4. 第 4 层-一次性工作区:每运行的 TemporaryDirectory 属于卫生便利,并非安全边界——其外的绝对路径仍可写。
5. 项目未曾到达的层:内核隔离(命名空间、cgroup、seccomp)与硬件虚拟化边界(Firecracker microVM)——多租户运行时执行不可信代码所必需的水准。

自主演练与记录

为 summarize-notes 运行画出五层阶梯;每层标注一个它挡住的威胁和一个挡不住的威胁(例如 basename 检查挡住 read_note 穿越;没有机制阻止空转的适配器——步数预算只是限制了浪费)。

独立掌握标准

对项目中每项控制,说出一个它挡住的威胁和一个挡不住的威胁——包括明确承认缺少 OS 级封闭。

权威源资源与权利证明

linking Linux man-pages license (freely distributable)

Linux man-pages: namespaces(7) - Overview of Linux Namespaces

作者 / 机构: Linux man-pages project (Michael Kerrisk et al.) | 原始出处链接 ↗

对应章节 / 时间戳: namespaces(7) DESCRIPTION: Namespace types

验证版本与日期: man-pages 6.x online edition (Verified: 2026-10-30)

入选理由:

Normative reference enumerating the eight Linux namespace types; establishes what genuine OS-level isolation requires (mount, PID, network, user namespaces plus seccomp and cgroup limits), all of which the learning project deliberately does not provide.

来源前置条件:

Processes, mounts, network interfaces, and cgroups basics.

安全回退方案(离线弹性):

Namespace concepts are analyzed against project code, not exercised; unshare(1) examples are optional enrichment.

对比候选资源与选型分析 (2)
Docker Engine security documentation ↗ compared_alternative [Apache-2.0 / documentation terms]

Product view assembling namespaces and cgroups into containers; namespaces(7) supplies the primitive inventory the lesson enumerates.

LWN.net series: Namespaces in operation ↗ compared_alternative [LWN site content]

Strong narrative walkthrough; the man page is the maintained reference surface.

linking USENIX open access

Firecracker: Lightweight Virtualization for Serverless Applications

作者 / 机构: Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, Diana-Maria Popa | 原始出处链接 ↗

对应章节 / 时间戳: Section 3.4 Security and Section 3.4.1 Jailer (nsdi20-paper-agache.pdf)

验证版本与日期: NSDI 2020 / USENIX open access (Verified: 2026-10-30)

入选理由:

Primary literature on production multi-tenant code execution: Firecracker wraps workloads in hardware-virtualized microVMs because containers alone are not a sufficient security boundary, the reference standard for what genuine isolation means.

来源前置条件:

Virtualization basics, KVM, process isolation, and threat models.

安全回退方案(离线弹性):

The isolation-layer analysis compares documented properties only; no virtualization is launched in the lesson.

对比候选资源与选型分析 (2)
gVisor architecture documentation ↗ compared_alternative [Apache-2.0]

User-space kernel interception alternative; Firecracker's hardware boundary provides the stronger reference point for the guarantee matrix.

Kata Containers architecture ↗ compared_alternative [Apache-2.0]

VM-backed container runtime; Firecracker's NSDI paper supplies the peer-reviewed threat-model justification.

04 120 分钟

评估:执行限制保证矩阵审计

独立掌握检查(无 AI 辅助)

前置要求

第 78 天活动 1-3。

学习目标

产出一份保证矩阵,将每个记录在案的控制对照“声称防住的威胁”与“不声称防住的威胁”进行审计,每格均由实测证据或明文文档佐证。

操作指引

1. 列:control | 机制与位置 | 防住 | 防不住。
2. 行:工具白名单(ALLOWED_TOOLS frozenset);basename 检查(_tool_call);4096 字节载荷上限;max_steps=6;max_retries=2;TemporaryDirectory 工作区;仅动作/观测轨迹;subprocess 超时(check_snippets.py、lab 运行器);进程组树终止(缺失);OS 沙箱/VM 边界(缺失)。
3. 证据规则:“防住”的论断需要现场演示或单元测试;“防不住”的论断需要现场反例(存活的 sleep PID)或文档声明(README:资源限制是控制手段,不是 OS 沙箱;project.json 的 evidence.resource_limit_note)。
4. 拒绝任何无法绑定证据的论断行——诚实的矩阵交付零条未经验证的承诺。

自主演练与记录

填写 guarantee_matrix.md 的 10 行矩阵;引用行级机制(如 agent.py 中的 'Path(filename).name != filename')以及活动 2 中逃逸子孙的 PID。

独立掌握标准

交付一份保证矩阵:每条否定论断都有证据或文档佐证——零条未经验证的承诺。

权威源资源与权利证明

linking PSF License Version 2

Python Standard Library: ast - Abstract Syntax Trees

作者 / 机构: Python Software Foundation | 原始出处链接 ↗

对应章节 / 时间戳: ast helpers (ast.parse, ast.walk)

验证版本与日期: Python 3.13 documentation (Verified: 2026-10-30)

入选理由:

Normative reference for ast.parse used by scripts/check_snippets.py as the static admission gate: AST rules inspect source structure before any execution and cannot evaluate runtime behavior.

来源前置条件:

Python syntax, parse trees, and compile pipeline basics.

安全回退方案(离线弹性):

Every AST check in the lesson runs in the standard library; no external validator is fetched.

对比候选资源与选型分析 (2)

Production AST linter; the stdlib ast module is the minimal sufficient tool for the admission-gate exercise.

Bandit security linter ↗ compared_alternative [Apache-2.0]

Security-oriented AST rules; deliberately cited as a comparison to show AST gating is necessary but never sufficient for safety.

linking CC BY 4.0

Kubernetes Documentation: Resource Management for Pods and Containers

作者 / 机构: Kubernetes Documentation Contributors | 原始出处链接 ↗

对应章节 / 时间戳: How Kubernetes applies resource requests and limits

验证版本与日期: Kubernetes documentation (Verified: 2026-10-30)

入选理由:

Primary reference for cgroup-enforced resource limits (CPU throttling, memory OOM-kill): demonstrates that production limits are kernel-enforced accounting, contrasted with the project's application-level counters.

来源前置条件:

Containers, CPU/memory accounting, and QoS classes.

安全回退方案(离线弹性):

The lesson contrasts documented cgroup semantics with the project's counters; no cluster is required.

对比候选资源与选型分析 (2)
Linux man-pages: cgroups(7) ↗ compared_alternative [Linux man-pages license]

Kernel-level primitive reference; Kubernetes docs present the enforcement outcome (throttle vs OOM-kill) learners recognize from operations.

Docker resource constraints documentation ↗ compared_alternative [Apache-2.0 / documentation terms]

Single-host flag surface; Kubernetes documentation covers limit semantics plus QoS classification.