Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

fix: Gravitino catalog S3 credential key mismatch and _has_table error handling - Support both hyphen and dot notation for S3 credential keys - _has_table only catches not-found errors, re-raises network/auth errors Closes #7200 Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
jiangxt2:fix/gravitino-error-handling
7 hours ago
feat(parquet): ownership-model reader prototype — bounded resident bytes via process-wide budget Prototype for the parquet OOM fix (RFC forthcoming). Adds an "owned" reader mode alongside the existing path (same binary, env-toggled A/B): - reader/budget.rs (new): process-wide ByteBudget. Strict-FIFO admission, sync reserve() + task-side acquire().await (avoids driver deadlock when the front RG blocks on a full output channel), oversized-RG exclusive admission, RAII permit release, planned vs actual metrics. 12 unit tests incl. the deadlock regression case. - reader/chunk_source.rs: RemoteChunkSourcePlan (immutable per-occurrence coalesced range plan, no GETs, no Bytes) + ResidentRowGroup (Bytes + budget permit, dropped together) + download_occurrence with response length validation. Legacy IO layout shared via rg_coalesced_layout(). - reader/mod.rs: owned driver — reservations registered in RG order (never awaited in the driver), per-RG RuntimeTask (abort-on-drop cascade for limit/cancel), predicate prefilter moved into the per-RG lifecycle, cross-RG limit truncation kept solely in the outer consumer. - reader/rg_processor.rs: RgAccess/RgReader::Resident — column decoders hold the Arc<ResidentRowGroup>, binding bytes+permit lifetime by type. Env knobs: DAFT_PARQUET_READER_MODE=legacy|owned (default owned; local files stay legacy), DAFT_PARQUET_RESIDENT_BUDGET_MB (default 256, 0 = unlimited), DAFT_PARQUET_RG_LOOKAHEAD (default 2), DAFT_PARQUET_MEM_VERBOSE. Measured (debug, minio): single-file peak RSS 1173-1337MB -> 379-391MB; 4-file 2879 -> 734MB (process-wide budget binds across files); predicate scan 2582 -> 645MB; compressed-resident ledger quantized to min(budget-fitting, lookahead) x RG size and returns to zero on completion and cancellation. Correctness: 21-case predicate matrix byte-identical legacy vs owned; dup/out-of-order RGs; limit boundaries; in-process cancel. Known gap (pre-existing upstream, both modes): predicate-prefilter column decoder JoinSet is never harvested — a decoder panic yields a silently truncated result. Fix planned as an independent PR. Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnr
FANNG1:parquet-owned-reader-prototype
12 hours ago
Merge branch 'main' into fix/partition-transform-predicate-pushdown
PerumalsamyR:fix/partition-transform-predicate-pushdown
16 hours ago
fix(parquet): honor row-group constraints in count_rows() count pushdown The count-pushdown shortcut for `count_rows()` returned the whole-file row count from parquet footer metadata and ignored the ScanTask's row-group constraint (`ChunkSpec::Parquet`). This caused two bugs: - Explicit `row_groups=`: `count_rows()` returned the whole-file count instead of the selected row groups' count (any runner). - Scan-task splitting (`enable_scan_task_split_and_merge`, Ray runner): a file split into N subtasks had each subtask return the whole-file count, so the sum was N× the true count — a silent data-correctness error on the officially recommended large-file config. Fix: the count shortcut now sums only the requested row groups. It reads the full footer and treats the `ChunkSpec` indices as positional indices into it — mirroring the normal read path, which also re-reads the full footer and ignores the embedded (pruned) metadata. Row-group validation is shared with the normal path via a new `validate_requested_row_groups` helper (order- and duplicate-preserving, empty → 0, out-of-bounds → error). Defense in depth (guards against custom ScanOperators that bypass the optimizer's invariants; built-in sources never hit these): - `PushDownAggregation::can_pushdown` now also requires `limit.is_none()`. - The parquet count shortcut errors loudly (never silently miscounts or falls back) when it sees a non-`All` count mode, a filter, a limit, or delete rows — none of which the metadata-only path can apply. Tests: Rust unit tests for row-group validation and the execution-layer invariant guard; a native `count_rows()` regression over explicit row_groups (incl. duplicate/non-monotonic/empty/out-of-bounds); and a Ray-runner 10-way split regression asserting no N× inflation. Claude-Session: https://claude.ai/code/session_01WXH9EJ2ySkFmDXJdkPmTBk
FANNG1:fix/count-rows-ignores-row-groups
24 hours ago
fix: predicate prefilter drops rows silently when a column decoder panics The two-phase predicate prefilter spawned per-column decoder tasks but discarded the returned JoinSet. When a decoder panicked, its dropped channel sender was indistinguishable from a normal stream end: the recv loop broke, the unprocessed rows were appended as a skip RowSelector, and the query returned successfully with silently missing rows. Fix ("EOF success barrier + fast cancel"): - On decoder EOF — the only loop exit that reports success without having seen every decoder finish — drop the receivers, then join all decoder tasks so panics surface as JoinError, and require that every selected row was processed before trusting the skip-padding. Receivers must close before joining: decoders block on tx.send (capacity-1 channels), so the order is deadlock-critical. - Limit early-stop and error paths keep abort-on-drop: decoders may still be blocked in column I/O and their remaining output cannot affect the query, so waiting on them would only delay cancellation. - Strengthen recv_one_chunk's lockstep protocol: all-Some is a chunk, all-None is EOF, and a Some/None mix (a decoder exited early while siblings still stream) is now an error instead of a silent EOF. The data-column paths already harvest via combine_stream + join_all; this closes the same gap in the prefilter phase. Tests: unit tests for the harvest helper and the lockstep protocol, plus end-to-end regression tests using a path-keyed, column-targeted test-only panic injection in spawn_col_decoders. The prefilter E2E test was verified to fail against the pre-fix behavior (query returned 0 rows successfully) and pass with the fix (query errors with JoinError). Closes #7285 Claude-Session: https://claude.ai/code/session_01H3HGmK8s37SmtzZDNBdSH7
FANNG1:fix/parquet-prefilter-decoder-panic
1 day ago
Merge branch 'main' into support_iceberg_rest_catalog_on_gravitino
kyo-tom:support_iceberg_rest_catalog_on_gravitino
3 days ago

Latest Branches

CodSpeed Performance Gauge
0%
fix: Gravitino catalog S3 credential key mismatch and _has_table error handling#7279
13 days ago
5d2e36a
jiangxt2:fix/gravitino-error-handling
CodSpeed Performance Gauge
0%
13 days ago
084bf20
jiangxt2:fix/sql-literal-translation
CodSpeed Performance Gauge
0%
8 hours ago
e2843a6
FANNG1:parquet-owned-reader-prototype
© 2026 CodSpeed Technology
Home Terms Privacy Docs