Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

Merge branch 'main' into support_iceberg_rest_catalog_on_gravitino

kyo-tom:support_iceberg_rest_catalog_on_gravitino

7 hours ago

fix(core): handle all-null list literals (#7168) ## Changes Made - Handle explicit list dtype construction when every list literal is null by creating an empty child series with the requested child dtype instead of concatenating an empty set of child series. - Build the all-null list child from the child physical dtype before casting back to the declared child dtype, so extension-typed children do not trigger a Rust panic. - Add regression tests for `Series.from_pylist([None, None], dtype=DataType.list(...))`, including string, extension, and nested extension child dtypes. This fixes the same conversion path used by legacy `@daft.udf` when a batch returns only `None` values for a declared list return dtype. ## Related Issues Closes #7167 ## Validation - `DAFT_RUNNER=native make test EXTRA_ARGS="-q tests/series/test_series.py"` passed: 69 tests. - Manual legacy UDF post-processing check via `daft.udf.legacy.run_udf` returned `List[String]` with `[None, None]`. Note: local full dataframe legacy UDF `collect()` timed out in this environment even for this minimal case, matching earlier local executor behavior unrelated to this conversion fix; the failing traceback path is covered by the regression test and `run_udf` check. Co-authored-by: wangzheyan <wangzheyan@bytedance.com>

main

21 hours ago

fix: Gravitino catalog S3 credential key mismatch and _has_table error handling - Support both hyphen and dot notation for S3 credential keys - _has_table only catches not-found errors, re-raises network/auth errors Closes #7200 Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/gravitino-error-handling

1 day ago

fix: SQL literal/IsIn/Decimal expressions fail to translate to SQL - Escape single quotes in Utf8 literal display_sql (O'Brien → O''Brien) - Implement Decimal and Uuid literal SQL generation - Implement IsIn expression SQL generation (col IN (v1, v2, v3)) - Support UnresolvedColumn in to_sql for pre-resolution expressions - Handle empty is_in([]) as constant-false (1 = 0) Closes #7197 Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/sql-literal-translation

1 day ago

style(parquet): satisfy clippy on the owned-reader prototype - gate BudgetPermit::bytes() behind cfg(test) (only tests read it) - include metrics in the manual ByteBudget Debug impl - iterate &mut waiters instead of .iter_mut() - drop the unused granted_now test helper - extract InflightRg type alias for the coordinator's in-flight deque No behavior change; 27 daft-parquet tests still pass. Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnr

FANNG1:parquet-owned-reader-prototype

1 day ago

feat(parquet): ownership-model reader prototype — bounded resident bytes via process-wide budget Prototype for the parquet OOM fix (RFC forthcoming). Adds an "owned" reader mode alongside the existing path (same binary, env-toggled A/B): - reader/budget.rs (new): process-wide ByteBudget. Strict-FIFO admission, sync reserve() + task-side acquire().await (avoids driver deadlock when the front RG blocks on a full output channel), oversized-RG exclusive admission, RAII permit release, planned vs actual metrics. 12 unit tests incl. the deadlock regression case. - reader/chunk_source.rs: RemoteChunkSourcePlan (immutable per-occurrence coalesced range plan, no GETs, no Bytes) + ResidentRowGroup (Bytes + budget permit, dropped together) + download_occurrence with response length validation. Legacy IO layout shared via rg_coalesced_layout(). - reader/mod.rs: owned driver — reservations registered in RG order (never awaited in the driver), per-RG RuntimeTask (abort-on-drop cascade for limit/cancel), predicate prefilter moved into the per-RG lifecycle, cross-RG limit truncation kept solely in the outer consumer. - reader/rg_processor.rs: RgAccess/RgReader::Resident — column decoders hold the Arc<ResidentRowGroup>, binding bytes+permit lifetime by type. Env knobs: DAFT_PARQUET_READER_MODE=legacy|owned (default owned; local files stay legacy), DAFT_PARQUET_RESIDENT_BUDGET_MB (default 256, 0 = unlimited), DAFT_PARQUET_RG_LOOKAHEAD (default 2), DAFT_PARQUET_MEM_VERBOSE. Measured (debug, minio): single-file peak RSS 1173-1337MB -> 379-391MB; 4-file 2879 -> 734MB (process-wide budget binds across files); predicate scan 2582 -> 645MB; compressed-resident ledger quantized to min(budget-fitting, lookahead) x RG size and returns to zero on completion and cancellation. Correctness: 21-case predicate matrix byte-identical legacy vs owned; dup/out-of-order RGs; limit boundaries; in-process cancel. Known gap (pre-existing upstream, both modes): predicate-prefilter column decoder JoinSet is never harvested — a decoder panic yields a silently truncated result. Fix planned as an independent PR. Claude-Session: https://claude.ai/code/session_01U4zNcFFh5qxC5RDC6PCZnr

FANNG1:parquet-owned-reader-prototype

2 days ago

Merge branch 'main' into fix/partition-transform-predicate-pushdown

PerumalsamyR:fix/partition-transform-predicate-pushdown

2 days ago

fix(parquet): honor row-group constraints in count_rows() count pushdown The count-pushdown shortcut for `count_rows()` returned the whole-file row count from parquet footer metadata and ignored the ScanTask's row-group constraint (`ChunkSpec::Parquet`). This caused two bugs: - Explicit `row_groups=`: `count_rows()` returned the whole-file count instead of the selected row groups' count (any runner). - Scan-task splitting (`enable_scan_task_split_and_merge`, Ray runner): a file split into N subtasks had each subtask return the whole-file count, so the sum was N× the true count — a silent data-correctness error on the officially recommended large-file config. Fix: the count shortcut now sums only the requested row groups. It reads the full footer and treats the `ChunkSpec` indices as positional indices into it — mirroring the normal read path, which also re-reads the full footer and ignores the embedded (pruned) metadata. Row-group validation is shared with the normal path via a new `validate_requested_row_groups` helper (order- and duplicate-preserving, empty → 0, out-of-bounds → error). Defense in depth (guards against custom ScanOperators that bypass the optimizer's invariants; built-in sources never hit these): - `PushDownAggregation::can_pushdown` now also requires `limit.is_none()`. - The parquet count shortcut errors loudly (never silently miscounts or falls back) when it sees a non-`All` count mode, a filter, a limit, or delete rows — none of which the metadata-only path can apply. Tests: Rust unit tests for row-group validation and the execution-layer invariant guard; a native `count_rows()` regression over explicit row_groups (incl. duplicate/non-monotonic/empty/out-of-bounds); and a Ray-runner 10-way split regression asserting no N× inflation. Claude-Session: https://claude.ai/code/session_01WXH9EJ2ySkFmDXJdkPmTBk

FANNG1:fix/count-rows-ignores-row-groups

2 days ago

Latest Branches

feat(catalog): support Gravitino Iceberg REST catalogs#7245

8 hours ago

c0bb5cb

kyo-tom:support_iceberg_rest_catalog_on_gravitino

fix: Gravitino catalog S3 credential key mismatch and _has_table error handling#7279

14 days ago

5d2e36a

jiangxt2:fix/gravitino-error-handling

fix: SQL literal/IsIn/Decimal expressions fail to translate to SQL#7278

14 days ago

084bf20

jiangxt2:fix/sql-literal-translation

Home Terms Privacy Docs