Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

Merge branch 'main' into claude/issue-7374-c0y9le

claude/issue-7374-c0y9le

60 minutes ago

fix(optimizer): harden join-side limit pushdown guard The idempotency guard on the join limit pushdown only recognized an already-limited outer side when the Limit was a direct child. Once the pushed Limit commuted below a Project/Repartition/IntoBatches/ IntoPartitions node, the guard missed it and re-wrapped the outer side on every optimizer pass. The guard now looks through the operators this rule commutes with. Also compute limit + offset in u64 with saturating_add (both in the join arm and the source arm) so a huge offset saturates instead of overflowing usize. Covered by new fixed-point and overflow regression tests.

hello-peter-tang:push-down-limit-outer-join

9 hours ago

test(distributed): relax shuffle regression test to unordered LIMIT semantics The test asserted that left.limit(20) over a multi-partition range returns ids 0..19, but Daft's LIMIT is unordered: which rows survive depends on partition scheduling. CI showed the native runner returning ids 875/880 and the ray runner returning 125/130, failing the exact-set assertion even though the join itself was correct. Assert the properties that hold regardless of which 20 rows the limit keeps: exactly 20 rows, all ids distinct, no null-extended key, and k == id % 100 for every row. This still catches lost, duplicated, mis-routed, or null-extended rows from the shuffle without assuming a deterministic row selection.

hello-peter-tang:fix/shuffle-transpose-empty-outputs

10 hours ago

test(distributed): assert deterministic join keys in shuffle regression test Strengthen the limit-below/above-join regression test to assert the exact set of (id, key) pairs rather than only the row count, so mis-routed, duplicated, or null-extended rows leaking through the shuffle are caught. Sorting keeps the check independent of distributed row ordering. Addresses review feedback.

hello-peter-tang:fix/shuffle-transpose-empty-outputs

11 hours ago

Merge branch 'main' into wl_0603_enum

wuleiwuleiwulei:wl_0603_enum

1 day ago

Merge branch 'main' into perf/sharded-grouped-agg

BABTUNA:perf/sharded-grouped-agg

1 day ago

Merge branch 'main' into codex/agg-sum-literal-rewrite

codex/agg-sum-literal-rewrite

1 day ago

fix(distributed): surface background errors while input is pending (#7396) `JoinableForwardingStream` previously polled background tasks only after its input stream closed. If a scheduler or materializer task failed while the input remained `Pending`, that error stayed queued in the `JoinSet` and the driver could wait indefinitely for another partition. This change surfaces background failures without changing ready-input priority or the stream's existing drop semantics. ## Changes Made - Poll the input first so ready data keeps its existing priority. - While the input is pending, poll completed background tasks until the remaining tasks register the stream waker. - On a task error or panic, stop input forwarding, emit the error immediately, and transition to `AwaitingTasks` with the remaining `JoinSet`. - Continue forwarding input after successful background completions, and stop polling an exhausted `JoinSet`. - Add regression coverage for pending input, delayed failures, wakeup preservation, ready-input priority, remaining-task behavior after failure, and task errors/panics. ## Validation - `cargo test -p daft-distributed test_joinable_forwarding_stream -- --nocapture`: 10 passed. - `cargo test -p daft-distributed`: 75 passed, 1 ignored. - Fork Ray verification: 2 independent repetitions; each ran the focused Ray case 5 times and the CI-equivalent Python 3.10 Ray unit suite. Both repetitions passed in 40m 41s and 44m 13s. - Community PR CI: 35 successful checks and 1 expected skip. - `cargo fmt --all -- --check` and `git diff --check` passed. ## Related Issues Closes #7394 --------- Co-authored-by: zhouwenjia.zwj <zhouwenjia.zwj@alibaba-inc.com>

main

1 day ago

Latest Branches

feat: add daft-device crate with device-resident buffer and allocator foundation#7376

1 hour ago

b4997d2

claude/issue-7374-c0y9le

-1%

feat(optimizer): push limit through row-preserving join sides#7439

8 days ago

654a019

hello-peter-tang:push-down-limit-outer-join

-1%

fix(distributed): tolerate shuffle outputs with no partitions#7442

10 hours ago

6251226

hello-peter-tang:fix/shuffle-transpose-empty-outputs

Home Terms Privacy Docs