Latest Results
fix(distributed): address review on downscale hardening
Three fixes from review of the idle-worker downscaling change, plus the
formatting the style job was failing on.
1. Draining workers are no longer hidden from the scheduler.
Hiding them made hard-affinity tasks (WorkerAffinity { soft: false })
permanently unschedulable: that path has no fallback worker, so a task
pinned to a draining worker was silently dropped from every scheduling
round. Workers now stay in `worker_snapshots()` carrying a `draining`
flag. Spread placement and soft affinity skip them, hard affinity still
resolves — and placing a task there makes the worker non-idle, so the
next reaper tick puts it back in service. Autoscaling capacity counting
excludes draining workers, so a fully-draining pool correctly asks for
replacements instead of looking fully provisioned.
2. Autoscaling demand is now owned per plan.
`ray.autoscaler.sdk.request_resources` is a single cluster-wide slot
that every call replaces, so the previous code let any one plan clobber
the demand of every other plan running on the same cluster: a finishing
query cleared the slot outright, and the reaper did the same after
retiring a worker, cancelling capacity a running query was still waiting
on. Each scheduler event loop now carries an `AutoscaleDemandId`; the
worker manager keeps per-owner books (bundles, ramp high-water mark and
rate-limit clock) and always publishes the union of every live owner.
Clearing removes only the caller's slice and re-publishes the remainder.
The reaper no longer writes the slot at all — retirement does not change
the union, and it is not the owner of anyone's demand.
3. Failed worker releases no longer leak the worker.
`release()`/`shutdown()` used `.expect()`, so an actor teardown failure
panicked the detached reaper thread; the busy-worker case returned
silently after the worker had already been removed from `ray_workers`
and blacklisted, leaving it alive but invisible to the scheduler while
still holding cluster resources. Both are now fallible: `shutdown`
returns `DaftResult<()>`, `release` returns `Ok(false)` when the worker
picked up work and `Err` when teardown failed, and the reaper puts any
worker it could not release back into the pool (clearing its blacklist
and draining entries). Manager shutdown logs and continues instead of
aborting the remaining teardowns.
Tests: draining is covered for spread / soft affinity / hard affinity /
fully-draining autoscale, and demand ownership is covered by two
concurrent scheduler loops each retracting only their own slice.hello-peter-tang:fix/downscale-reaper-hardening Latest Branches
0%
XuQianJin-Stars:feat/lerobot-v2 -1%
0%
hello-peter-tang:fix/downscale-reaper-hardening © 2026 CodSpeed Technology