Latest Results
fix(parquet): honor row-group constraints in count_rows() count pushdown (#7288)
## Changes Made
`count_rows()`'s count-pushdown shortcut returned the whole-file row
count from parquet footer metadata and **ignored the ScanTask's
row-group constraint** (`ChunkSpec::Parquet`). This caused:
- **Explicit `row_groups=`**: `count_rows()` returned the whole-file
count instead of the selected groups' count (any runner).
- **Scan-task splitting on Ray** (`enable_scan_task_split_and_merge`,
the recommended large-file config): a file split into N subtasks had
each subtask return the whole-file count, so the sum was **N× the true
count** — a silent data-correctness error.
### Main fix
- `stream_parquet_count_pushdown` now takes the requested row groups and
sums only those. It reads the full footer and treats the `ChunkSpec`
indices as **positional** indices into it — mirroring the normal read
path, which also re-reads the full footer and ignores the embedded
(pruned) metadata, so original index == positional index in every case.
- `scan_task_reader.rs` extracts `ChunkSpec::Parquet` before entering
the shortcut and threads it through.
- Row-group validation is now shared with the normal read path via a new
`validate_requested_row_groups` helper in `helpers.rs` (order- and
duplicate-preserving: `[1,1,1]` counts three times; `[]` → 0;
out-of-bounds/negative → error), so both paths stay consistent.
### Defense in depth
The metadata-only count path cannot apply row-level processing. Built-in
sources never reach it with such pushdowns, but to protect against
custom `ScanOperator`s that bypass the optimizer's invariants:
- `PushDownAggregation::can_pushdown` now also requires
`pushdowns.limit.is_none()` (a limit is a global semantic; per-subtask
`min` then sum could overcount).
- The parquet count shortcut errors loudly — never silently miscounts,
never falls back (fallback would yield `SUM(null)` = null after
aggregation rewriting) — on a non-`All` count mode, a filter, a limit,
or delete rows. These four guards are consolidated in a small pure,
unit-tested helper.
### Tests
- Rust: `validate_requested_row_groups`
(single/non-monotonic/duplicate/empty/out-of-bounds/negative); the
execution-layer invariant guard (all four unsupported inputs); a new
optimizer test that a limit prevents count pushdown.
- Python (native): `count_rows()` over explicit `row_groups=` using
unequal row groups so subset/duplicate sums differ from the whole-file
count, plus agreement with `agg(col.count())` and an assertion that the
pushdown is actually taken.
- Python (Ray): a 10-way split regression asserting `count_rows()` is
not N× inflated.
## Related Issues
Closes #7287
---------
Co-authored-by: fanng <“fanng@apache.org”> Latest Branches
0%
hfutatzhanghb:agent/opendal-hdfs-community 0%
AnayGarodia:fix/5172-video-frames-mkv-webm 0%
XuQianJin-Stars:feature/hash-crypto-functions © 2026 CodSpeed Technology