Latest Results
feat(parquet): ownership-model reader prototype — bounded resident bytes via process-wide budget
Prototype for the parquet OOM fix (RFC forthcoming). Adds an "owned" reader
mode alongside the existing path (same binary, env-toggled A/B):
- reader/budget.rs (new): process-wide ByteBudget. Strict-FIFO admission,
sync reserve() + task-side acquire().await (avoids driver deadlock when
the front RG blocks on a full output channel), oversized-RG exclusive
admission, RAII permit release, planned vs actual metrics. 12 unit tests
incl. the deadlock regression case.
- reader/chunk_source.rs: RemoteChunkSourcePlan (immutable per-occurrence
coalesced range plan, no GETs, no Bytes) + ResidentRowGroup (Bytes +
budget permit, dropped together) + download_occurrence with response
length validation. Legacy IO layout shared via rg_coalesced_layout().
- reader/mod.rs: owned driver — reservations registered in RG order (never
awaited in the driver), per-RG RuntimeTask (abort-on-drop cascade for
limit/cancel), predicate prefilter moved into the per-RG lifecycle,
cross-RG limit truncation kept solely in the outer consumer.
- reader/rg_processor.rs: RgAccess/RgReader::Resident — column decoders
hold the Arc<ResidentRowGroup>, binding bytes+permit lifetime by type.
Env knobs: DAFT_PARQUET_READER_MODE=legacy|owned (default owned; local
files stay legacy), DAFT_PARQUET_RESIDENT_BUDGET_MB (default 256, 0 =
unlimited), DAFT_PARQUET_RG_LOOKAHEAD (default 2), DAFT_PARQUET_MEM_VERBOSE.
Measured (debug, minio): single-file peak RSS 1173-1337MB -> 379-391MB;
4-file 2879 -> 734MB (process-wide budget binds across files); predicate
scan 2582 -> 645MB; compressed-resident ledger quantized to
min(budget-fitting, lookahead) x RG size and returns to zero on completion
and cancellation. Correctness: 21-case predicate matrix byte-identical
legacy vs owned; dup/out-of-order RGs; limit boundaries; in-process cancel.
Known gap (pre-existing upstream, both modes): predicate-prefilter column
decoder JoinSet is never harvested — a decoder panic yields a silently
truncated result. Fix planned as an independent PR.
Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnrFANNG1:parquet-owned-reader-prototype fix: predicate prefilter drops rows silently when a column decoder panics
The two-phase predicate prefilter spawned per-column decoder tasks but
discarded the returned JoinSet. When a decoder panicked, its dropped
channel sender was indistinguishable from a normal stream end: the recv
loop broke, the unprocessed rows were appended as a skip RowSelector,
and the query returned successfully with silently missing rows.
Fix ("EOF success barrier + fast cancel"):
- On decoder EOF — the only loop exit that reports success without
having seen every decoder finish — drop the receivers, then join all
decoder tasks so panics surface as JoinError, and require that every
selected row was processed before trusting the skip-padding.
Receivers must close before joining: decoders block on tx.send
(capacity-1 channels), so the order is deadlock-critical.
- Limit early-stop and error paths keep abort-on-drop: decoders may
still be blocked in column I/O and their remaining output cannot
affect the query, so waiting on them would only delay cancellation.
- Strengthen recv_one_chunk's lockstep protocol: all-Some is a chunk,
all-None is EOF, and a Some/None mix (a decoder exited early while
siblings still stream) is now an error instead of a silent EOF.
The data-column paths already harvest via combine_stream + join_all;
this closes the same gap in the prefilter phase.
Tests: unit tests for the harvest helper and the lockstep protocol,
plus end-to-end regression tests using a path-keyed, column-targeted
test-only panic injection in spawn_col_decoders. The prefilter E2E test
was verified to fail against the pre-fix behavior (query returned 0
rows successfully) and pass with the fix (query errors with JoinError).
Closes #7285
Claude-Session: https://claude.ai/code/session_01H3HGmK8s37SmtzZDNBdSH7FANNG1:fix/parquet-prefilter-decoder-panic docs: add physical AI examples (hand tracking + reward scoring) (#7271)
## Changes Made
Adds two physical AI examples from
[daft-physical-ai](https://github.com/Eventual-Inc/daft-physical-ai) to
the docs, listed first in the nav and examples index: hand tracking,
then reward scoring. Each page is the executed `demo.md` copied from the
extension repo, with docs-site adaptations: title, intro naming the
extension, images moved to `docs/img/`, and (rewards) server-script
links made absolute to the extension repo.
- **Hand tracking**: verified by running all 10 Python blocks in order
in a fresh env; outputs match the committed ones (`detect=100%
mean_err=0.116 PCK@.1/.2/.3 = 49/84/96`).
- **Reward scoring**: needs a Robometer eval server (local-GPU and Modal
launchers linked from the page). Committed outputs come from a real run
against a Modal deployment, score-identical across the last three
regenerations.
- **Mobile header fix**: the `.DocSearch-Button` 11rem width override
applied at every viewport, overflowing phones into horizontal scroll -
now scoped to >=768px, and Material's dead search toggle removed.
Verified at 390/800/1440px.
## Screenshots (local `mkdocs` build)
Examples index, light and dark:
<img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/2d9b6c719/pr-screenshots/examples-index-light.png"
width="49%"> <img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/2d9b6c719/pr-screenshots/examples-index-dark.png"
width="49%">
Hand tracking, light and dark:
<img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/hand-tracking-top-light.png"
width="49%"> <img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/hand-tracking-top-dark.png"
width="49%">
Reward scoring, light and dark:
<img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/reward-scoring-top-light.png"
width="49%"> <img
src="https://raw.githubusercontent.com/Eventual-Inc/Daft/229bd66d7/pr-screenshots/reward-scoring-top-dark.png"
width="49%"> Latest Branches
0%
FANNG1:parquet-owned-reader-prototype 0%
PerumalsamyR:fix/partition-transform-predicate-pushdown 0%
FANNG1:fix/count-rows-ignores-row-groups © 2026 CodSpeed Technology