Eventual-Inc/Daft

Daft

Performance History

Latest Results

feat(io): support ignoring corrupt ORC files Add opt-in schema fallback and conservative ORC error filtering. Report skipped files through existing execution statistics while preserving emitted batches and unrelated error propagation. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:orc-ignore-corrupt-files

1 hour ago

feat(io): support Hive directory partitioning in read_orc Read Hive directory partition columns and prune ORC scan tasks using partition predicates. Preserve typed constants through projections and residual filtering, with reader tests and documentation. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:orc-hive-partitioning

2 hours ago

feat(io): support file_path_column in ORC reads Add a source path column through the existing ORC batch projection and cover schema, projection, filtering, URI, empty-file, and serialization behavior. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:orc-file-path-column

2 hours ago

feat(io): support explicit schema and schema hints in read_orc Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:orc-schema-hints

3 hours ago

fix(core): handle all-null literal batches for tensor/sparse_tensor/map Row-wise UDFs returning tensor, sparse_tensor, or map failed with "Need at least 1 series to perform concat" whenever every row in a batch was null (a UDF that returns None, or that raises on all rows). The Tensor, SparseTensor, and Map arms of `series_from_literals_iter` passed the all-None vec to `ListArray::from_series`, which concatenates zero child series and errors. Under `on_error="raise"` that error returned before the `errs` check at the end of the function, masking the UDF's real exception. Short-circuit to an all-null series of the target dtype when every value is None, mirroring the List fix from #7168. Execution then reaches the `errs` check, so `on_error="raise"` surfaces the real error while `on_error="ignore"` / all-None returns an all-null column. The Embedding and Image arms were already unaffected. Fixes #7584

hello-peter-tang:fix/7584-all-null-tensor-map-udf

5 hours ago

fix(parquet): apply positional deletes in count-only fast path (#7560) A read that projects zero file columns takes the count_only_stream fast path, which summed physical row-group sizes and ignored delete_rows, start_offset, and num_rows. On Iceberg merge-on-read tables this over-counted by including positionally-deleted rows. Derive the count-only total from build_base_selections -- the same selection machinery the decode path uses -- so offset, positional deletes, and the limit are applied consistently across both paths. Adds a regression test (200 rows over 4 row groups, 40 positional deletes -> 160 live rows) that failed 200 != 160 before the fix.

hello-peter-tang:fix/7560-count-only-positional-deletes

9 hours ago

fix(sql): use signed 64-bit Iceberg snapshot IDs

Solomon-mithra:fix/sql-iceberg-signed-snapshot-id

15 hours ago

feat(io): add basic ORC file reading support (#7576) ## Changes Made Add `daft.read_orc()` for reading raw ORC files into a Daft DataFrame, allowing ORC inputs to be processed directly without a separate format conversion step. The reader uses the existing Python `DataSource` / `DataSourceTask` APIs, Daft file I/O, and PyArrow's ORC Dataset scanner. - Support file paths, directories, glob patterns, and path lists. Search directories recursively for `*.orc` and deduplicate overlapping inputs. - Reuse `IOConfig`, including the planning context's default configuration, for local and remote file access. - Read rows lazily in execution tasks, using one task per file and configurable batch sizes. - Infer the schema from the first matched file. Align subsequent files by field name, fill missing fields with nulls, exclude extra fields, and apply Daft's existing conversion rules. - Support column projection while retaining fields required by filters. Filters and limits use Daft's existing execution operators. - Add public exports, unit and integration tests, API documentation, and an ORC usage guide. Full schema merging, stripe-level task splitting, and ORC-native predicate pruning are outside this PR. ### Example ```python import daft df = daft.read_orc("./data/*.orc", batch_size=65536) df.select("id", "label").show() ``` ### Testing Tests cover path handling, schema alignment, projection/filter interactions, batching, serialization, and resource cleanup. - Native ORC suite: **76 passed**. - Ray ORC suite with xdist and coverage: **76 passed**. - HTTP/S3-compatible integration suite: **9 passed**. - PyArrow 16.0.0 compatibility suite: **33 passed**. - Project pre-commit checks and MkDocs build passed. ### AI assistance Codex and Claude assisted with development and review. ## Related Issues Closes #7575 --------- Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

main

18 hours ago

Latest Branches

feat(io): support ignoring corrupt ORC files#7600

2 hours ago

5a50930

jiangxt2:orc-ignore-corrupt-files

feat(io): support Hive directory partitioning in read_orc#7599

2 hours ago

989096d

jiangxt2:orc-hive-partitioning

feat(io): support file_path_column in ORC reads#7598

3 hours ago

40a24d2

jiangxt2:orc-file-path-column

Home Terms Privacy Docs