Latest Results
feat(parquet): ownership-model reader prototype — bounded resident bytes via process-wide budget
Prototype for the parquet OOM fix (RFC forthcoming). Adds an "owned" reader
mode alongside the existing path (same binary, env-toggled A/B):
- reader/budget.rs (new): process-wide ByteBudget. Strict-FIFO admission,
sync reserve() + task-side acquire().await (avoids driver deadlock when
the front RG blocks on a full output channel), oversized-RG exclusive
admission, RAII permit release, planned vs actual metrics. 12 unit tests
incl. the deadlock regression case.
- reader/chunk_source.rs: RemoteChunkSourcePlan (immutable per-occurrence
coalesced range plan, no GETs, no Bytes) + ResidentRowGroup (Bytes +
budget permit, dropped together) + download_occurrence with response
length validation. Legacy IO layout shared via rg_coalesced_layout().
- reader/mod.rs: owned driver — reservations registered in RG order (never
awaited in the driver), per-RG RuntimeTask (abort-on-drop cascade for
limit/cancel), predicate prefilter moved into the per-RG lifecycle,
cross-RG limit truncation kept solely in the outer consumer.
- reader/rg_processor.rs: RgAccess/RgReader::Resident — column decoders
hold the Arc<ResidentRowGroup>, binding bytes+permit lifetime by type.
Env knobs: DAFT_PARQUET_READER_MODE=legacy|owned (default owned; local
files stay legacy), DAFT_PARQUET_RESIDENT_BUDGET_MB (default 256, 0 =
unlimited), DAFT_PARQUET_RG_LOOKAHEAD (default 2), DAFT_PARQUET_MEM_VERBOSE.
Measured (debug, minio): single-file peak RSS 1173-1337MB -> 379-391MB;
4-file 2879 -> 734MB (process-wide budget binds across files); predicate
scan 2582 -> 645MB; compressed-resident ledger quantized to
min(budget-fitting, lookahead) x RG size and returns to zero on completion
and cancellation. Correctness: 21-case predicate matrix byte-identical
legacy vs owned; dup/out-of-order RGs; limit boundaries; in-process cancel.
Known gap (pre-existing upstream, both modes): predicate-prefilter column
decoder JoinSet is never harvested — a decoder panic yields a silently
truncated result. Fix planned as an independent PR.
Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnrFANNG1:parquet-owned-reader-prototype fix: predicate prefilter drops rows silently when a column decoder panics
The two-phase predicate prefilter spawned per-column decoder tasks but
discarded the returned JoinSet. When a decoder panicked, its dropped
channel sender was indistinguishable from a normal stream end: the recv
loop broke, the unprocessed rows were appended as a skip RowSelector,
and the query returned successfully with silently missing rows.
Fix ("EOF success barrier + fast cancel"):
- On decoder EOF — the only loop exit that reports success without
having seen every decoder finish — drop the receivers, then join all
decoder tasks so panics surface as JoinError, and require that every
selected row was processed before trusting the skip-padding.
Receivers must close before joining: decoders block on tx.send
(capacity-1 channels), so the order is deadlock-critical.
- Limit early-stop and error paths keep abort-on-drop: decoders may
still be blocked in column I/O and their remaining output cannot
affect the query, so waiting on them would only delay cancellation.
- Strengthen recv_one_chunk's lockstep protocol: all-Some is a chunk,
all-None is EOF, and a Some/None mix (a decoder exited early while
siblings still stream) is now an error instead of a silent EOF.
The data-column paths already harvest via combine_stream + join_all;
this closes the same gap in the prefilter phase.
Tests: unit tests for the harvest helper and the lockstep protocol,
plus end-to-end regression tests using a path-keyed, column-targeted
test-only panic injection in spawn_col_decoders. The prefilter E2E test
was verified to fail against the pre-fix behavior (query returned 0
rows successfully) and pass with the fix (query errors with JoinError).
Closes #7285
Claude-Session: https://claude.ai/code/session_01H3HGmK8s37SmtzZDNBdSH7FANNG1:fix/parquet-prefilter-decoder-panic Latest Branches
0%
jiangxt2:fix/gravitino-error-handling 0%
jiangxt2:fix/sql-literal-translation 0%
FANNG1:parquet-owned-reader-prototype © 2026 CodSpeed Technology