Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

fix(sql): use signed 64-bit Iceberg snapshot IDs

Solomon-mithra:fix/sql-iceberg-signed-snapshot-id

2 hours ago

feat(io): add basic ORC file reading support (#7576) ## Changes Made Add `daft.read_orc()` for reading raw ORC files into a Daft DataFrame, allowing ORC inputs to be processed directly without a separate format conversion step. The reader uses the existing Python `DataSource` / `DataSourceTask` APIs, Daft file I/O, and PyArrow's ORC Dataset scanner. - Support file paths, directories, glob patterns, and path lists. Search directories recursively for `*.orc` and deduplicate overlapping inputs. - Reuse `IOConfig`, including the planning context's default configuration, for local and remote file access. - Read rows lazily in execution tasks, using one task per file and configurable batch sizes. - Infer the schema from the first matched file. Align subsequent files by field name, fill missing fields with nulls, exclude extra fields, and apply Daft's existing conversion rules. - Support column projection while retaining fields required by filters. Filters and limits use Daft's existing execution operators. - Add public exports, unit and integration tests, API documentation, and an ORC usage guide. Full schema merging, stripe-level task splitting, and ORC-native predicate pruning are outside this PR. ### Example ```python import daft df = daft.read_orc("./data/*.orc", batch_size=65536) df.select("id", "label").show() ``` ### Testing Tests cover path handling, schema alignment, projection/filter interactions, batching, serialization, and resource cleanup. - Native ORC suite: **76 passed**. - Ray ORC suite with xdist and coverage: **76 passed**. - HTTP/S3-compatible integration suite: **9 passed**. - PyArrow 16.0.0 compatibility suite: **33 passed**. - Project pre-commit checks and MkDocs build passed. ### AI assistance Codex and Claude assisted with development and review. ## Related Issues Closes #7575 --------- Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

main

4 hours ago

style(parquet): use f64::mul_add to satisfy clippy suboptimal_flops The pipelined-predicate str_to_unit helper accumulated `acc += b * scale`, which trips the denied nursery lint clippy::suboptimal_flops. Rewrite it as `acc = f64::from(b).mul_add(scale, acc)`, the canonical fused form.

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

13 hours ago

Merge origin/main (8cba057) into fix/downscale-reaper-hardening Resolve the worker_manager.rs conflict by folding main's new Bisect autoscale strategy into this PR's per-owner demand model rather than keeping main's global bisect state: - BisectState now lives inside AutoscaleDemand (per owner), and the union of all owners' bundles is published via request_resources, so a bisecting plan no longer clobbers concurrent plans' demands. - Dropped main's global bisect_state / max_resources_requested and send_bundles_to_ray in favor of build_bundle_dicts + publish_union_and_refresh. - Kept main's 3-arg RayWorkerManager::new (strategy + bisect timeout) and AutoscaleStrategy parsing. Verified: cargo check -p daft-distributed --all-targets (exit 0) and cargo test -p daft-distributed --lib (100 passed, 0 failed).

hello-peter-tang:fix/downscale-reaper-hardening

13 hours ago

Merge branch 'main' into fix/bisect-round-robin-bundles

DogerW666:fix/bisect-round-robin-bundles

14 hours ago

fix(recordbatch): respect null_equals_nulls for statically-null semi/anti join keys hash_semi_anti_join short-circuited whenever an evaluated join key column had DataType::Null, returning every left row for anti joins and no rows for semi joins while ignoring null_equals_nulls entirely. Under null-safe equality (INTERSECT/EXCEPT semantics, EqNullSafe predicates), NULL keys must match NULL keys. After match_types_for_tables a key column is Null-typed only when both sides are Null, so the shortcut is now taken only when some statically null key has null_equals_nulls=false, where no pair can ever match. Otherwise the join falls through to the regular hash path, which already supports Null-dtype keys: NullArray::hash is deterministic and build_is_equal honors the per-column flag for (Null, Null) pairs. Fixes #7502

hello-peter-tang:fix/7502-semi-anti-null-safe-static-null-keys

14 hours ago

fix(optimizer): preserve input order observed by batch UDFs

hello-peter-tang:eliminate-redundant-sort

15 hours ago

fix(optimizer): exclude hidden partition metadata from data projections

hello-peter-tang:fix/6757-filter-columns-pushdown

15 hours ago

Latest Branches

fix(sql): use signed 64-bit Iceberg snapshot IDs#7590

2 hours ago

7206f26

Solomon-mithra:fix/sql-iceberg-signed-snapshot-id

perf(parquet): LM-pipelined predicate evaluation with stats-based group ordering#7488

14 hours ago

d23836e

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

fix(distributed): harden idle-worker downscaling against scheduler races#7344

14 hours ago

724a13d

hello-peter-tang:fix/downscale-reaper-hardening

Home Terms Privacy Docs