Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

feat(logical-plan): add lineage section to plan JSON Expose plan-level lineage so subscribers can consume it without reaching into internal plan structures. Rather than a custom API, lineage is attached to the LogicalPlan object itself and surfaced through its JSON representation. A new `lineage` module defines typed Lineage/LineageInput/LineageOutput values; `LogicalPlan::lineage()` derives them by walking the plan tree (inputs from source nodes, outputs from sink nodes), and `repr_json` serializes them onto the root node. Inputs cover glob_scan, physical_scan and in_memory; outputs cover file, iceberg, delta_lake, lance and data_sink.
Lucas61000:issue-6496-plan-json-lineage
1 hour ago
fix(io): gate hdfs opendal support
hfutatzhanghb:agent/opendal-hdfs-community
7 hours ago
fix(io): use opendal hdfs service
hfutatzhanghb:agent/opendal-hdfs-community
7 hours ago
chore: remove redundant decode comments
AnayGarodia:fix/5172-video-frames-mkv-webm
7 hours ago
fix(parquet): honor row-group constraints in count_rows() count pushdown (#7288) ## Changes Made `count_rows()`'s count-pushdown shortcut returned the whole-file row count from parquet footer metadata and **ignored the ScanTask's row-group constraint** (`ChunkSpec::Parquet`). This caused: - **Explicit `row_groups=`**: `count_rows()` returned the whole-file count instead of the selected groups' count (any runner). - **Scan-task splitting on Ray** (`enable_scan_task_split_and_merge`, the recommended large-file config): a file split into N subtasks had each subtask return the whole-file count, so the sum was **N× the true count** — a silent data-correctness error. ### Main fix - `stream_parquet_count_pushdown` now takes the requested row groups and sums only those. It reads the full footer and treats the `ChunkSpec` indices as **positional** indices into it — mirroring the normal read path, which also re-reads the full footer and ignores the embedded (pruned) metadata, so original index == positional index in every case. - `scan_task_reader.rs` extracts `ChunkSpec::Parquet` before entering the shortcut and threads it through. - Row-group validation is now shared with the normal read path via a new `validate_requested_row_groups` helper in `helpers.rs` (order- and duplicate-preserving: `[1,1,1]` counts three times; `[]` → 0; out-of-bounds/negative → error), so both paths stay consistent. ### Defense in depth The metadata-only count path cannot apply row-level processing. Built-in sources never reach it with such pushdowns, but to protect against custom `ScanOperator`s that bypass the optimizer's invariants: - `PushDownAggregation::can_pushdown` now also requires `pushdowns.limit.is_none()` (a limit is a global semantic; per-subtask `min` then sum could overcount). - The parquet count shortcut errors loudly — never silently miscounts, never falls back (fallback would yield `SUM(null)` = null after aggregation rewriting) — on a non-`All` count mode, a filter, a limit, or delete rows. These four guards are consolidated in a small pure, unit-tested helper. ### Tests - Rust: `validate_requested_row_groups` (single/non-monotonic/duplicate/empty/out-of-bounds/negative); the execution-layer invariant guard (all four unsupported inputs); a new optimizer test that a limit prevents count pushdown. - Python (native): `count_rows()` over explicit `row_groups=` using unequal row groups so subset/duplicate sums differ from the whole-file count, plus agreement with `agg(col.count())` and an assertion that the pushdown is actually taken. - Python (Ray): a 10-way split regression asserting `count_rows()` is not N× inflated. ## Related Issues Closes #7287 --------- Co-authored-by: fanng <“fanng@apache.org”>
main
13 hours ago

Latest Branches

CodSpeed Performance Gauge
0%
feat: add lineage section to plan JSON#7293
5 hours ago
47bae68
Lucas61000:issue-6496-plan-json-lineage
CodSpeed Performance Gauge
0%
2 hours ago
21a08d1
satya323:satyendra/mongodb-data-source-support
CodSpeed Performance Gauge
0%
7 hours ago
b0d8149
hfutatzhanghb:agent/opendal-hdfs-community
© 2026 CodSpeed Technology
Home Terms Privacy Docs