Latest Results
fix(dashboard): support OUT_DIR on a different filesystem from the source tree (#7246)
## Problem Description
The dashboard build script moves the generated frontend assets into
`OUT_DIR` with `std::fs::rename`, at both call sites (`ci_main` and
`default_main`). `rename(2)` fails with `EXDEV` when source and
destination live on different filesystems, which is routine for cargo:
`CARGO_TARGET_DIR` pointed at a separate mount or tmpfs, a container
with the source bind-mounted and the target on a volume, a CI cache
mount, or a distinct btrfs subvolume.
The build then fails outright:
```
error: failed to run custom build command for `daft-dashboard v0.3.0-dev0`
Error: Os { code: 18, kind: CrossesDevices, message: "Invalid cross-device link" }
```
Reproduction on a stock Linux box, where `/tmp` is tmpfs and therefore a
different device from the checkout:
```console
$ CARGO_TARGET_DIR=/tmp/daft-target cargo build -p daft-dashboard
```
## Changes Made
Fall back to a recursive copy followed by removing the source when
rename reports `ErrorKind::CrossesDevices`. The fallback is not atomic,
unlike `rename(2)`, and it resolves symlinks rather than recreating
them; neither matters for the generated asset tree, which is regular
files and directories only. The rename fast path is unchanged, so
behaviour is identical for anyone whose build already worked. No new
dependencies.
## Verification
A/B on one machine against one target directory, with the patch as the
only variable:
| | result |
|---|---|
| without patch | `exit 101`, `Error: Os { code: 18, kind:
CrossesDevices, ... }` |
| with patch | `exit 0` |
- `make build` end to end: wheel built and installed editable.
- Asset tree in `OUT_DIR` after the copy fallback matches
`frontend/out`: 56 files, 1.5 MB, `index.html` present, nested `_next/`
preserved, and the source directory removed as `rename` would have left
it.
- `cargo clippy -p daft-dashboard` and `cargo fmt -p daft-dashboard --
--check` are clean.
No automated test is included: `build.rs` is not compiled into any test
target, so `move_dir` has no `cargo test` seam without restructuring the
crate, and a meaningful CI test would need two filesystems. Happy to
extract the helper if you would like one.
## Related Issues
None. This is a clearly scoped bugfix, so per `CONTRIBUTING.md` it may
qualify for the `no-issue-needed` label. Glad to open an issue first if
you would rather.
---
This PR was made with the assistance of Opus 4.8. refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions (#7221)
## Summary
Extracts a shared `LocalShuffleBackend` runtime enum and
`FlightShuffleContext` struct in `daft-local-execution` and migrates
`GatherSink` and `IntoPartitionsSink` onto them. This is the first of
two PRs resolving the `// TODO: unify shuffle backends in all local
operations` markers (#7212); the second migrates `RepartitionSink` and
collapses the backend dispatch in `pipeline.rs`.
## Why
`sinks/gather.rs` and `sinks/into_partitions.rs` each defined their own
Ray/Flight backend enum plus a `FlightShared` struct that was duplicated
field-for-field between the two files. Every operator added to the
flight shuffle backend (e.g. Sort or Pre-Shuffle Merge from the #6472
phase 2 checklist) currently has to copy this pattern again. See #7212
for the full plan.
## Changes Made
- Add `sinks/shuffle_backend.rs` with `LocalShuffleBackend { Ray,
Flight(Arc<FlightShuffleContext>) }` and `FlightShuffleContext` holding
the formerly duplicated fields (`shuffle_id`, `shuffle_dirs`,
`compression`, `local_server`, `shuffle_address`,
`target_in_memory_size_per_partition`, `schema`)
- `sinks/gather.rs`: delete `GatherBackend` and `FlightShared`, use the
shared types; `collect_output` becomes a free function over
`LocalShuffleBackend`
- `sinks/into_partitions.rs`: delete `IntoPartitionsBackend` and
`FlightShared`; `schema` moves from the `Ray` enum variant onto
`IntoPartitionsSink` itself
- Net -13 lines (+87/-100)
## Behavior
Functionally equivalent for all inputs. Constructor signatures
(`new_ray` / `try_new_flight`), spill thresholds, state machines, and
node display names (`Gather(Ray)`, `IntoPartitions(Flight)`) are all
unchanged.
## Test Plan
- `cargo check -p daft-local-execution --lib` clean
- `cargo fmt -p daft-local-execution` clean
- `cargo clippy -p daft-local-execution --lib --no-deps` clean for the
touched files (pre-existing warnings elsewhere in the crate unchanged)
- No Rust unit tests exist for these sinks; ran the Python integration
suite locally: `DAFT_RUNNER=ray pytest
tests/dataframe/test_shuffles.py`, 39 passed, covering gather /
into_partitions / repartition under both the Ray and Flight backends
including empty-input edge cases
## Related Issues
First of two PRs for #7212. Part of #6472. Latest Branches
0%
yk/fix-progress-bar-fd-leak 0%
XiaoHongbo-Hope:fix/min-cpu-per-task-wiring -1%
dwallener:codex/add-to-map-function © 2026 CodSpeed Technology