Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

Merge branch 'main' into support_iceberg_rest_catalog_on_gravitino
kyo-tom:support_iceberg_rest_catalog_on_gravitino
7 hours ago
feat(parquet): ownership-model reader prototype — bounded resident bytes via process-wide budget Prototype for the parquet OOM fix (RFC forthcoming). Adds an "owned" reader mode alongside the existing path (same binary, env-toggled A/B): - reader/budget.rs (new): process-wide ByteBudget. Strict-FIFO admission, sync reserve() + task-side acquire().await (avoids driver deadlock when the front RG blocks on a full output channel), oversized-RG exclusive admission, RAII permit release, planned vs actual metrics. 12 unit tests incl. the deadlock regression case. - reader/chunk_source.rs: RemoteChunkSourcePlan (immutable per-occurrence coalesced range plan, no GETs, no Bytes) + ResidentRowGroup (Bytes + budget permit, dropped together) + download_occurrence with response length validation. Legacy IO layout shared via rg_coalesced_layout(). - reader/mod.rs: owned driver — reservations registered in RG order (never awaited in the driver), per-RG RuntimeTask (abort-on-drop cascade for limit/cancel), predicate prefilter moved into the per-RG lifecycle, cross-RG limit truncation kept solely in the outer consumer. - reader/rg_processor.rs: RgAccess/RgReader::Resident — column decoders hold the Arc<ResidentRowGroup>, binding bytes+permit lifetime by type. Env knobs: DAFT_PARQUET_READER_MODE=legacy|owned (default owned; local files stay legacy), DAFT_PARQUET_RESIDENT_BUDGET_MB (default 256, 0 = unlimited), DAFT_PARQUET_RG_LOOKAHEAD (default 2), DAFT_PARQUET_MEM_VERBOSE. Measured (debug, minio): single-file peak RSS 1173-1337MB -> 379-391MB; 4-file 2879 -> 734MB (process-wide budget binds across files); predicate scan 2582 -> 645MB; compressed-resident ledger quantized to min(budget-fitting, lookahead) x RG size and returns to zero on completion and cancellation. Correctness: 21-case predicate matrix byte-identical legacy vs owned; dup/out-of-order RGs; limit boundaries; in-process cancel. Known gap (pre-existing upstream, both modes): predicate-prefilter column decoder JoinSet is never harvested — a decoder panic yields a silently truncated result. Fix planned as an independent PR. Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnr
FANNG1:parquet-owned-reader-prototype
2 days ago
Merge branch 'main' into fix/partition-transform-predicate-pushdown
PerumalsamyR:fix/partition-transform-predicate-pushdown
2 days ago
fix(parquet): honor row-group constraints in count_rows() count pushdown The count-pushdown shortcut for `count_rows()` returned the whole-file row count from parquet footer metadata and ignored the ScanTask's row-group constraint (`ChunkSpec::Parquet`). This caused two bugs: - Explicit `row_groups=`: `count_rows()` returned the whole-file count instead of the selected row groups' count (any runner). - Scan-task splitting (`enable_scan_task_split_and_merge`, Ray runner): a file split into N subtasks had each subtask return the whole-file count, so the sum was N× the true count — a silent data-correctness error on the officially recommended large-file config. Fix: the count shortcut now sums only the requested row groups. It reads the full footer and treats the `ChunkSpec` indices as positional indices into it — mirroring the normal read path, which also re-reads the full footer and ignores the embedded (pruned) metadata. Row-group validation is shared with the normal path via a new `validate_requested_row_groups` helper (order- and duplicate-preserving, empty → 0, out-of-bounds → error). Defense in depth (guards against custom ScanOperators that bypass the optimizer's invariants; built-in sources never hit these): - `PushDownAggregation::can_pushdown` now also requires `limit.is_none()`. - The parquet count shortcut errors loudly (never silently miscounts or falls back) when it sees a non-`All` count mode, a filter, a limit, or delete rows — none of which the metadata-only path can apply. Tests: Rust unit tests for row-group validation and the execution-layer invariant guard; a native `count_rows()` regression over explicit row_groups (incl. duplicate/non-monotonic/empty/out-of-bounds); and a Ray-runner 10-way split regression asserting no N× inflation. Claude-Session: https://claude.ai/code/session_01WXH9EJ2ySkFmDXJdkPmTBk
FANNG1:fix/count-rows-ignores-row-groups
2 days ago

Latest Branches

CodSpeed Performance Gauge
0%
feat(catalog): support Gravitino Iceberg REST catalogs#7245
8 hours ago
c0bb5cb
kyo-tom:support_iceberg_rest_catalog_on_gravitino
CodSpeed Performance Gauge
0%
14 days ago
5d2e36a
jiangxt2:fix/gravitino-error-handling
CodSpeed Performance Gauge
0%
14 days ago
084bf20
jiangxt2:fix/sql-literal-translation
© 2026 CodSpeed Technology
Home Terms Privacy Docs