Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

fix(parquet): honor row-group constraints in count_rows() count pushdown The count-pushdown shortcut for `count_rows()` returned the whole-file row count from parquet footer metadata and ignored the ScanTask's row-group constraint (`ChunkSpec::Parquet`). This caused two bugs: - Explicit `row_groups=`: `count_rows()` returned the whole-file count instead of the selected row groups' count (any runner). - Scan-task splitting (`enable_scan_task_split_and_merge`, Ray runner): a file split into N subtasks had each subtask return the whole-file count, so the sum was N× the true count — a silent data-correctness error on the officially recommended large-file config. Fix: the count shortcut now sums only the requested row groups. It reads the full footer and treats the `ChunkSpec` indices as positional indices into it — mirroring the normal read path, which also re-reads the full footer and ignores the embedded (pruned) metadata. Row-group validation is shared with the normal path via a new `validate_requested_row_groups` helper (order- and duplicate-preserving, empty → 0, out-of-bounds → error). Defense in depth (guards against custom ScanOperators that bypass the optimizer's invariants; built-in sources never hit these): - `PushDownAggregation::can_pushdown` now also requires `limit.is_none()`. - The parquet count shortcut errors loudly (never silently miscounts or falls back) when it sees a non-`All` count mode, a filter, a limit, or delete rows — none of which the metadata-only path can apply. Tests: Rust unit tests for row-group validation and the execution-layer invariant guard; a native `count_rows()` regression over explicit row_groups (incl. duplicate/non-monotonic/empty/out-of-bounds); and a Ray-runner 10-way split regression asserting no N× inflation. Claude-Session: https://claude.ai/code/session_01WXH9EJ2ySkFmDXJdkPmTBk
FANNG1:fix/count-rows-ignores-row-groups
2 hours ago
fix: predicate prefilter drops rows silently when a column decoder panics The two-phase predicate prefilter spawned per-column decoder tasks but discarded the returned JoinSet. When a decoder panicked, its dropped channel sender was indistinguishable from a normal stream end: the recv loop broke, the unprocessed rows were appended as a skip RowSelector, and the query returned successfully with silently missing rows. Fix ("EOF success barrier + fast cancel"): - On decoder EOF — the only loop exit that reports success without having seen every decoder finish — drop the receivers, then join all decoder tasks so panics surface as JoinError, and require that every selected row was processed before trusting the skip-padding. Receivers must close before joining: decoders block on tx.send (capacity-1 channels), so the order is deadlock-critical. - Limit early-stop and error paths keep abort-on-drop: decoders may still be blocked in column I/O and their remaining output cannot affect the query, so waiting on them would only delay cancellation. - Strengthen recv_one_chunk's lockstep protocol: all-Some is a chunk, all-None is EOF, and a Some/None mix (a decoder exited early while siblings still stream) is now an error instead of a silent EOF. The data-column paths already harvest via combine_stream + join_all; this closes the same gap in the prefilter phase. Tests: unit tests for the harvest helper and the lockstep protocol, plus end-to-end regression tests using a path-keyed, column-targeted test-only panic injection in spawn_col_decoders. The prefilter E2E test was verified to fail against the pre-fix behavior (query returned 0 rows successfully) and pass with the fix (query errors with JoinError). Closes #7285 Claude-Session: https://claude.ai/code/session_01H3HGmK8s37SmtzZDNBdSH7
FANNG1:fix/parquet-prefilter-decoder-panic
8 hours ago
Merge branch 'main' into support_iceberg_rest_catalog_on_gravitino
kyo-tom:support_iceberg_rest_catalog_on_gravitino
2 days ago
docs: add physical AI examples (hand tracking + reward scoring) (#7271) ## Changes Made Adds two physical AI examples from [daft-physical-ai](https://github.com/Eventual-Inc/daft-physical-ai) to the docs, listed first in the nav and examples index: hand tracking, then reward scoring. Each page is the executed `demo.md` copied from the extension repo, with docs-site adaptations: title, intro naming the extension, images moved to `docs/img/`, and (rewards) server-script links made absolute to the extension repo. - **Hand tracking**: verified by running all 10 Python blocks in order in a fresh env; outputs match the committed ones (`detect=100% mean_err=0.116 PCK@.1/.2/.3 = 49/84/96`). - **Reward scoring**: needs a Robometer eval server (local-GPU and Modal launchers linked from the page). Committed outputs come from a real run against a Modal deployment, score-identical across the last three regenerations. - **Mobile header fix**: the `.DocSearch-Button` 11rem width override applied at every viewport, overflowing phones into horizontal scroll - now scoped to >=768px, and Material's dead search toggle removed. Verified at 390/800/1440px. ## Screenshots (local `mkdocs` build) Examples index, light and dark: <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/2d9b6c719/pr-screenshots/examples-index-light.png" width="49%"> <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/2d9b6c719/pr-screenshots/examples-index-dark.png" width="49%"> Hand tracking, light and dark: <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/hand-tracking-top-light.png" width="49%"> <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/hand-tracking-top-dark.png" width="49%"> Reward scoring, light and dark: <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/reward-scoring-top-light.png" width="49%"> <img src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/reward-scoring-top-dark.png" width="49%">
main
2 days ago
ci: retrigger to clear flaky HF 504/429 and network timeouts
BABTUNA:refactor/unify-shuffle-backend-repartition
2 days ago
Merge branch 'main' into issue-2423
Lucas61000:issue-2423
3 days ago
Merge branch 'main' into support_iceberg_rest_catalog_on_gravitino
kyo-tom:support_iceberg_rest_catalog_on_gravitino
3 days ago

Latest Branches

CodSpeed Performance Gauge
0%
fix(parquet): honor row-group constraints in count_rows() count pushdown#7288
2 hours ago
7085702
FANNG1:fix/count-rows-ignores-row-groups
CodSpeed Performance Gauge
0%
8 hours ago
f4681f1
FANNG1:fix/parquet-prefilter-decoder-panic
CodSpeed Performance Gauge
0%
2 days ago
1de3e1c
kyo-tom:support_iceberg_rest_catalog_on_gravitino
© 2026 CodSpeed Technology
Home Terms Privacy Docs