Eventual-Inc/Daft

Daft

Performance History

Latest Results

fix(daft-distributed): remove unused hash_map_macro feature gate (#6981) ## What Remove the `#![feature(hash_map_macro)]` feature gate from `daft-distributed/src/lib.rs`. ## Why The `hash_map_macro` feature was [removed from Rust after 1.91](https://github.com/rust-lang/rust/issues/87853) and does not exist in stable Rust 1.95.0+. `daft-distributed` declares the feature gate but never uses the `hash_map!` macro anywhere in the crate, so the gate can be safely removed. This eliminates one unnecessary nightly-only feature gate, making it easier to compile daft on stable Rust. ## Testing - Verified that `hash_map!` is not used anywhere in `daft-distributed`: ``` grep -rn "hash_map!" src/daft-distributed/ # no results ``` We've also built 0.7.14 with Rust 1.95 in our own build system. Signed-off-by: Mike DePaulo <mikedep333@redhat.com>

main

28 minutes ago

fix(daft-distributed): remove unused hash_map_macro feature gate The `hash_map_macro` feature was removed from Rust after 1.91 and does not exist in stable Rust 1.95.0+. daft-distributed declares it but never uses the macro, so the gate can be safely removed. This allows daft to compile on stable Rust without RUSTC_BOOTSTRAP for this crate. Signed-off-by: Mike DePaulo <mikedep333@redhat.com>

mikedep333:remove-hash_map_macro

2 hours ago

test(checkpoint): optional real-S3 routing for checkpoint test suites (#6934) ## Changes Made Adds opt-in S3 routing to the iceberg + delta checkpoint test suites. When `CHECKPOINTING_TEST_BUCKET` is set, fixtures route warehouses, parquet inputs, and checkpoint stores through real S3. When absent, behavior is unchanged — local filesystem, the CI default. Shared helpers in `tests/io/_s3_helpers.py`. Side effects on daft's planning config + `AWS_REGION` env var fire at module import time, so only the two suites that import them are affected. Run command: ``` eval "$(aws configure export-credentials --profile <profile> --format env)" CHECKPOINTING_TEST_BUCKET=<bucket> \ AWS_REGION=us-east-2 \ DAFT_RUNNER=ray \ make test EXTRA_ARGS="-v tests/io/iceberg/test_iceberg_writes_checkpoint.py tests/io/delta_lake/test_deltalake_writes_checkpoint.py" ``` ## Related Issues Stacked on top of #6932. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

main

3 hours ago

test(checkpoint): optional real-S3 routing via CHECKPOINTING_TEST_BUCKET Adds opt-in S3 routing to the iceberg + delta checkpoint test suites. When CHECKPOINTING_TEST_BUCKET is set, fixtures route warehouses, parquet inputs, and checkpoint stores through real S3 (plus daft's default planning IO config and AWS_REGION env). When the env var is absent, behavior is unchanged — local filesystem, the CI default. Shared helpers in tests/io/_s3_helpers.py. Side effects on daft's planning config + AWS_REGION env var fire at module import time, so only the two suites that import them are affected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

rohit/test/checkpoint-s3-routable-fixtures

4 hours ago

fix(io): use checked_shl and saturating_mul in retry jitter calculation

ARDA7787:patch-1

7 hours ago

refactor(flotilla): remove unused `cancel_plan` `NativeExecutor::cancel_plan` (and its `PyNativeExecutor` binding) had no callers — orphaned from an earlier design. `cancel_input` is what the flotilla cancel path actually uses; per-plan teardown happens naturally when `try_finish` removes the last `active_input_id`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/fix-distributed-limit-cancel-noise

17 hours ago

fix(flotilla): silence noisy logs from distributed-limit cancellation PR #6942 surfaced three categories of confusing output during limit cancellation. None changed query results; all are removed here. - `ray.cancel(result_handle)` on every non-contributor task marked them FAILED in Ray's task table. Replaced with `actor.cancel_input.remote(...)`, which closes the per-input result channel inside the worker's `NativeExecutor` so `run_plan` drains and returns FINISHED. - `claim` / `start_task` on the torn-down counter actor used to raise `ActorDiedError`, propagate out of `DistributedLimitSink::execute`, and tear down the worker's run loop. They now swallow that error as a residual-race safety net (`LimitNode` drains task cancels first, so this only fires when a sink call is mid-flight). - "Plan execution task has died; cannot enqueue new input" was the downstream symptom of the previous two on concurrent inputs sharing a plan; no longer reproducible. Mechanism: 1. `NativeExecutor` exposes `cancel_input(fingerprint, input_id)`. It sends through a new unbounded channel; `run_execution_loop` drops the per-input result sender, ending the Python iteration cleanly. 2. `TaskResultHandle::cancel_callback` now returns a `Future`. The Ray impl awaits the Python `actor.cancel_input.remote(...)` ack so the scheduler can sequence post-cancel teardown. 3. `LimitNode::limit_execution_loop` drains `running_tasks` after firing `parent_cancel.cancel()` and before `teardown_limit_counter_actor`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/fix-distributed-limit-cancel-noise

19 hours ago

feat: configurable flight shuffle compression Adds `flight_shuffle_compression` to `DaftExecutionConfig`, exposing lz4/zstd Arrow IPC compression for flight shuffle spill files via `daft.context.set_execution_config(...)`. Also enables the lz4 and zstd features on the `arrow-ipc` workspace dep — without them the IPC writer panicked at runtime when compression was set, since the field on `FlightShuffleBackendConfig` had never been reachable from user code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/configurable-flight-shuffle-compression

19 hours ago

Latest Branches

fix(daft-distributed): remove unused hash_map_macro feature gate#6981

3 hours ago

b1c59de

mikedep333:remove-hash_map_macro

test(checkpoint): optional real-S3 routing for checkpoint test suites#6934

8 days ago

856a3f9

rohit/test/checkpoint-s3-routable-fixtures

fix(io): use checked_shl and saturating_mul in retry jitter calculation#6949

7 hours ago

90fef74

ARDA7787:patch-1

Home Terms Privacy Docs