Latest Results
fix(distributed): harden idle-worker downscaling against scheduler races
Fixes several issues in the background-reaper downscaling path:
- Two-phase retirement (drain, then release): a worker idle past the
threshold is first hidden from worker_snapshots() so the scheduler
stops assigning work to it, and only released on a later reaper tick
if still idle. This closes a race where the reaper could kill a
worker between the scheduler taking a snapshot and dispatching to it,
failing the whole query. If a draining worker picks up work in the
window, it is put back in service instead.
- Single critical section in reap_idle_workers: the scale-up guard,
min-survivor floor, and candidate selection now operate on one
consistent state snapshot, eliminating TOCTOU races that could
retire workers against in-flight scale-up demand or dip below the
floor. Policy decisions are extracted into a pure, unit-tested
plan_reap() function (scheduling/downscale.rs).
- Clear autoscaling demand on error exits too: the scheduler now clears
outstanding (sticky) Ray autoscaler demand whether the event loop
exits cleanly or with an error, best-effort so cleanup failures do
not fail an otherwise-successful job. Jobs that never requested
scale-up skip the call entirely, so the default path never touches
the cluster-wide request_resources slot.
- Reaper robustness: catch_unwind around each tick, poisoned-lock
recovery, warn/error level logging for tick failures and thread
spawn failure instead of silent debug/drops.
- Config/docs: env vars extracted to constants, documented
DAFT_AUTOSCALING_REAPER_INTERVAL_SECONDS, documented warm-pool
retirement semantics in ray.md and set_runner_ray docstrings.
Tests: 13 new unit tests covering the drain/release/undrain state
machine, floor enforcement, scale-up guard, disable-mid-drain, and
demand-clearing on the scheduler error path.hello-tang-rui:fix/downscale-reaper-hardening fix: truncate default IOConfig fields in explain() output (#7327)
## Changes Made
`df.explain()` and related plan displays were dumping every IO backend's
full default config (S3, Azure, GCS, HTTP, Unity, Gravitino, HF, TOS,
COS, GooseFS, HDFS, …), which made plans hard to read.
This PR makes IOConfig display **sparse / diff-from-default**:
1. Each backend's `multiline_display` only emits fields that differ from
`Default` (extending the existing GooseFS pattern).
2. `IOConfig::multiline_display` skips empty backend sections entirely.
3. `Display for IOConfig` uses the same sparse view (`IOConfig {}` when
fully default).
4. When a scan knows its URI scheme(s) (e.g. `s3://…`), GlobScanOperator
explain filters to only the relevant backend section(s).
## Before / after
Local parquet explain previously printed a multi-line wall of every
backend's defaults. Now:
```
* GlobScanOperator
| Glob paths = [...]
| IO config = <default>
| Use multithreading = true
```
`url_download` with a non-default `max_connections=32` previously
embedded the full multi-backend `IOConfig` dump; now it shows only:
```
IOConfig:
S3 config = { Max connections = 32 }
```
## Testing
- `cargo test -p common-io-config --lib` (43 passed), including new unit
tests for sparse/default/scheme-aware display
- `cargo check -p daft-scan`
- `cargo fmt -p common-io-config -p daft-scan`
- Reproduced local `read_parquet(...).explain()` against a `make build`
wheel
- Updated
`tests/dataframe/test_morsels.py::test_batch_size_from_udf_propagated_through_ops_to_scan`;
passes under `DAFT_RUNNER=native`
## AI usage disclosure
Developed with an AI coding assistant. I reviewed the diff, verified
explain output locally, and ran the tests above.
## Related Issues
Closes #7308
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Cursor <cursoragent@cursor.com> Latest Branches
0%
hello-tang-rui:fix/downscale-reaper-hardening 0%
0%
codex/mcap-pushdown-profiling © 2026 CodSpeed Technology