Latest Results
refactor(shuffle): unify distributed shuffle backend config onto daft-local-plan ShuffleBackend (#7402)
## Summary
Removes the duplicated shuffle backend config types in the distributed
runner and uses `daft_local_plan::ShuffleBackend` as the single config
enum for shuffle backend selection across distributed and local
execution. Follows up on the reviewer suggestion in #7221 to share the
`ShuffleBackend` enum between the two engines.
## Why
`DistributedShuffleBackend` and `FlightShuffleBackendConfig` in
`daft-distributed` were field-for-field duplicates of the plan-level
`ShuffleBackend` enum in `daft-local-plan`, bridged by a hand-written
`local_shuffle_backend()` conversion called at 6 sites.
`daft-distributed` already depends on `daft-local-plan`, so the plan
enum can serve both engines directly with no new crate edges. Suggested
by @srilman in review of #7221; part of the phase 2 shuffle cleanup in
#6472.
## Changes Made
- Deleted `DistributedShuffleBackend` and `FlightShuffleBackendConfig`
from `pipeline_node/shuffles/backends`
- Renamed the per-node wrapper struct `ShuffleBackend` to
`ShuffleContext` so the name `ShuffleBackend` refers to the plan-level
enum everywhere (matching `sort.rs`, which already imports it directly)
- `ShuffleContext::new` still stamps the per-node `shuffle_id` via
`make_shuffle_id`; `select_backend()` now builds
`ShuffleBackend::Flight` with a placeholder `shuffle_id: 0` that each
node overwrites
- Deleted the `local_shuffle_backend()` bridge; call sites pass
`backend().clone()` into the local plan builders
- `flight::register_cleanup` now takes `shuffle_id` and `shuffle_dirs`
directly instead of the deleted config struct
- Added `ShuffleBackend::name()` in `daft-local-plan`, replacing four
hand-rolled display matches in `gather.rs`, `repartition.rs`,
`random_shuffle.rs`, and `into_partitions.rs`
## Behavior
Functionally equivalent for all inputs. The enum swap is config plumbing
only: backend selection, shuffle id stamping, task building, and cleanup
registration are unchanged. Explain output is unchanged (`name()`
produces the same strings as the removed match blocks).
## Test Plan
- `cargo check -p daft-distributed -p daft-local-plan` clean
- `cargo check -p daft-distributed --features python` clean (the CI
configuration)
- `cargo fmt -p daft-distributed -p daft-local-plan` clean
- `cargo clippy -p daft-distributed -p daft-local-plan --lib --no-deps`
clean, no warnings in touched files
- No behavior change; the existing shuffle suite
(`tests/dataframe/test_shuffles.py`) covers both backends
## Related Issues
Part of #6472. Follow-up to #7221 and #7275. docs(sql): document table function options (#7355)
## Changes Made
**`docs/sql/index.md` — new "Table Function Options" section**
The SQL reference had only two bare examples:
```python
daft.sql("SELECT * FROM read_parquet('s3://...')")
daft.sql("SELECT * FROM read_iceberg('s3://.../metadata.json')")
```
Nothing documented the named arguments each reader accepts, so options
like `snapshot_id`, `branch`, `tag`, `hive_partitioning`, and
`ignore_corrupt_files` were only discoverable by reading
`src/daft-sql/src/table_provider/`. Added:
- argument syntax: both `=>` and `:=`, which functions accept a named
`path`, path arrays, struct-literal `schema`
- a per-function option table for `read_parquet`, `read_csv`,
`read_json`, `read_iceberg`, `read_deltalake`
- an Iceberg version-selection example, noting
`snapshot_id`/`branch`/`tag` are mutually exclusive
- links out to the Python reader API docs rather than restating what
each option does
**`docs/connectors/generic-file-source-options.md` — add the SQL form**
The page described `ignore_corrupt_files` as if it were Python-only, but
all three SQL readers support it now (#7130, #7133, #7264). Added the
SQL usage, noted `df.skipped_corrupt_files` works the same for SQL-built
dataframes, and cross-linked the two pages.
## Testing
Every option name and syntax form in the new tables was executed against
this branch rather than read off the source — including the `=>` / `:=`
variants, named `path` (verified
`read_parquet`/`read_csv`/`read_json`/`read_deltalake` accept it and
that `read_iceberg` does not, so it is excluded from that sentence),
path arrays, and struct-literal schemas.
`make docs` builds clean: warning count is identical to `main` (49
pre-existing, none from these pages), the `#table-function-options`
anchor resolves, and the cross-page link renders as
`../../sql/#table-function-options`.
## Related Issues
None — follow-up documentation for #7130, #7133, and #7264. Latest Branches
0%
0%
0%
DogerW666:fix/coalesce-unreachable-validation © 2026 CodSpeed Technology