Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

test: progress bar writes happen on a single stable thread
yk/fix-progress-bar-fd-leak
7 minutes ago
fix(dashboard): support OUT_DIR on a different filesystem from the source tree (#7246) ## Problem Description The dashboard build script moves the generated frontend assets into `OUT_DIR` with `std::fs::rename`, at both call sites (`ci_main` and `default_main`). `rename(2)` fails with `EXDEV` when source and destination live on different filesystems, which is routine for cargo: `CARGO_TARGET_DIR` pointed at a separate mount or tmpfs, a container with the source bind-mounted and the target on a volume, a CI cache mount, or a distinct btrfs subvolume. The build then fails outright: ``` error: failed to run custom build command for `daft-dashboard v0.3.0-dev0` Error: Os { code: 18, kind: CrossesDevices, message: "Invalid cross-device link" } ``` Reproduction on a stock Linux box, where `/tmp` is tmpfs and therefore a different device from the checkout: ```console $ CARGO_TARGET_DIR=/tmp/daft-target cargo build -p daft-dashboard ``` ## Changes Made Fall back to a recursive copy followed by removing the source when rename reports `ErrorKind::CrossesDevices`. The fallback is not atomic, unlike `rename(2)`, and it resolves symlinks rather than recreating them; neither matters for the generated asset tree, which is regular files and directories only. The rename fast path is unchanged, so behaviour is identical for anyone whose build already worked. No new dependencies. ## Verification A/B on one machine against one target directory, with the patch as the only variable: | | result | |---|---| | without patch | `exit 101`, `Error: Os { code: 18, kind: CrossesDevices, ... }` | | with patch | `exit 0` | - `make build` end to end: wheel built and installed editable. - Asset tree in `OUT_DIR` after the copy fallback matches `frontend/out`: 56 files, 1.5 MB, `index.html` present, nested `_next/` preserved, and the source directory removed as `rename` would have left it. - `cargo clippy -p daft-dashboard` and `cargo fmt -p daft-dashboard -- --check` are clean. No automated test is included: `build.rs` is not compiled into any test target, so `move_dir` has no `cargo test` seam without restructuring the crate, and a meaningful CI test would need two filesystems. Happy to extract the helper if you would like one. ## Related Issues None. This is a clearly scoped bugfix, so per `CONTRIBUTING.md` it may qualify for the `no-issue-needed` label. Glad to open an issue first if you would rather. --- This PR was made with the assistance of Opus 4.8.
main
3 hours ago
address comments
XiaoHongbo-Hope:fix/min-cpu-per-task-wiring
4 hours ago
refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions (#7221) ## Summary Extracts a shared `LocalShuffleBackend` runtime enum and `FlightShuffleContext` struct in `daft-local-execution` and migrates `GatherSink` and `IntoPartitionsSink` onto them. This is the first of two PRs resolving the `// TODO: unify shuffle backends in all local operations` markers (#7212); the second migrates `RepartitionSink` and collapses the backend dispatch in `pipeline.rs`. ## Why `sinks/gather.rs` and `sinks/into_partitions.rs` each defined their own Ray/Flight backend enum plus a `FlightShared` struct that was duplicated field-for-field between the two files. Every operator added to the flight shuffle backend (e.g. Sort or Pre-Shuffle Merge from the #6472 phase 2 checklist) currently has to copy this pattern again. See #7212 for the full plan. ## Changes Made - Add `sinks/shuffle_backend.rs` with `LocalShuffleBackend { Ray, Flight(Arc<FlightShuffleContext>) }` and `FlightShuffleContext` holding the formerly duplicated fields (`shuffle_id`, `shuffle_dirs`, `compression`, `local_server`, `shuffle_address`, `target_in_memory_size_per_partition`, `schema`) - `sinks/gather.rs`: delete `GatherBackend` and `FlightShared`, use the shared types; `collect_output` becomes a free function over `LocalShuffleBackend` - `sinks/into_partitions.rs`: delete `IntoPartitionsBackend` and `FlightShared`; `schema` moves from the `Ray` enum variant onto `IntoPartitionsSink` itself - Net -13 lines (+87/-100) ## Behavior Functionally equivalent for all inputs. Constructor signatures (`new_ray` / `try_new_flight`), spill thresholds, state machines, and node display names (`Gather(Ray)`, `IntoPartitions(Flight)`) are all unchanged. ## Test Plan - `cargo check -p daft-local-execution --lib` clean - `cargo fmt -p daft-local-execution` clean - `cargo clippy -p daft-local-execution --lib --no-deps` clean for the touched files (pre-existing warnings elsewhere in the crate unchanged) - No Rust unit tests exist for these sinks; ran the Python integration suite locally: `DAFT_RUNNER=ray pytest tests/dataframe/test_shuffles.py`, 39 passed, covering gather / into_partitions / repartition under both the Ray and Flight backends including empty-input edge cases ## Related Issues First of two PRs for #7212. Part of #6472.
main
4 hours ago
fix: satisfy style checks for to_map
dwallener:codex/add-to-map-function
5 hours ago
fix: remove redundant to_map clone
dwallener:codex/add-to-map-function
6 hours ago
fix: handle empty to_map inputs
dwallener:codex/add-to-map-function
6 hours ago

Latest Branches

CodSpeed Performance Gauge
0%
fix: fd and thread leak from tqdm progress bar in Jupyter#7262
21 minutes ago
17aa555
yk/fix-progress-bar-fd-leak
CodSpeed Performance Gauge
0%
4 hours ago
aaf2f6b
XiaoHongbo-Hope:fix/min-cpu-per-task-wiring
CodSpeed Performance Gauge
-1%
5 hours ago
2321faf
dwallener:codex/add-to-map-function
© 2026 CodSpeed Technology
Home Terms Privacy Docs