Latest Results
fix(distributed): harden idle-worker downscaling against scheduler races
Fixes several issues in the background-reaper downscaling path:
- Two-phase retirement (drain, then release): a worker idle past the
threshold is first hidden from worker_snapshots() so the scheduler
stops assigning work to it, and only released on a later reaper tick
if still idle. This closes a race where the reaper could kill a
worker between the scheduler taking a snapshot and dispatching to it,
failing the whole query. If a draining worker picks up work in the
window, it is put back in service instead.
- Single critical section in reap_idle_workers: the scale-up guard,
min-survivor floor, and candidate selection now operate on one
consistent state snapshot, eliminating TOCTOU races that could
retire workers against in-flight scale-up demand or dip below the
floor. Policy decisions are extracted into a pure, unit-tested
plan_reap() function (scheduling/downscale.rs).
- Clear autoscaling demand on error exits too: the scheduler now clears
outstanding (sticky) Ray autoscaler demand whether the event loop
exits cleanly or with an error, best-effort so cleanup failures do
not fail an otherwise-successful job. Jobs that never requested
scale-up skip the call entirely, so the default path never touches
the cluster-wide request_resources slot.
- Reaper robustness: catch_unwind around each tick, poisoned-lock
recovery, warn/error level logging for tick failures and thread
spawn failure instead of silent debug/drops.
- Config/docs: env vars extracted to constants, documented
DAFT_AUTOSCALING_REAPER_INTERVAL_SECONDS, documented warm-pool
retirement semantics in ray.md and set_runner_ray docstrings.
Tests: 13 new unit tests covering the drain/release/undrain state
machine, floor enforcement, scale-up guard, disable-mid-drain, and
demand-clearing on the scheduler error path.hello-tang-rui:fix/downscale-reaper-hardening Latest Branches
0%
everettVT/trace-video-operations 0%
everettVT/fix-main-failures 0%
everettVT/fix-hf-glob-revision © 2026 CodSpeed Technology