Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

Fix unequal-partition sample regression test

johnquevedo:issue-5690-ray-sample-by-size

5 minutes ago

Merge branch 'Eventual-Inc:main' into main

ARDA7787:main

4 hours ago

feat(sql): support ignore_corrupt_files in read_csv (#7264) ## Changes Made Adds the `ignore_corrupt_files` option to the SQL `read_csv` table function. This closes the last gap in the `ignore_corrupt_files` parity matrix — Python `read_csv`, SQL `read_parquet` (#7133), and iceberg (#7130) already support it, but SQL `read_csv` did not. - Wire `ignore_corrupt_files` through `CsvScanBuilder` (field, setter, and `finish()` → `CsvSourceConfig`). - Parse the named arg in the `read_csv` table provider and add it to the allowed-args list. - Release the GIL around blocking schema inference so a skipped corrupt file's `log::warn!` (forwarded by pyo3-log) cannot deadlock on GIL re-acquisition. This mirrors the `read_parquet` fix in #7133 and the DataFrame scan path in `daft-scan/src/python.rs`. Usage: ```sql SELECT * FROM read_csv('/path/to/dir', ignore_corrupt_files => true) ``` ## Testing - New `tests/sql/test_sql_table_functions.py::test_sql_read_csv_ignore_corrupt_files` verifies corrupt files error by default and are skipped (with `skipped_corrupt_files` populated) when the option is set. - Existing `tests/sql/` and `tests/io/test_ignore_corrupt_files.py` suites pass; a 40× stress loop over corrupt-file schema inference confirmed no GIL deadlock. ## Related Issues Related to #7133 (SQL `read_parquet` `ignore_corrupt_files` + GIL fix). --------- Co-authored-by: Claude <noreply@anthropic.com>

main

5 hours ago

feat(sql): resolve read_iceberg io_config from table properties and default config (#7273) ## Changes Made The SQL `read_iceberg` table function silently ignored both the Iceberg table's embedded FileIO credentials and the context `default_io_config`, so SQL reads couldn't reach object storage that the equivalent Python `daft.read_iceberg(...)` call handled fine. Two root causes: - The Rust scan path (`iceberg_scan` in `scan_builder.rs`) built the `StorageConfig` from `io_config.unwrap_or_default()`, discarding any fallback. - `functions::args::parse_io_config` forced an empty `IOConfig` default when the SQL arg was unset, pre-empting the fallback entirely. This PR: - Introduces a shared `resolve_iceberg_io_config` helper in `daft/io/iceberg/_iceberg.py` implementing the standard precedence — **explicit `io_config` arg > table PyIceberg FileIO properties (S3/Azure/GCS creds + `oss://` alias) > context `default_io_config`**. - Calls that single helper from **both** the Python `read_iceberg` API and the Rust SQL scan path (resolved inside `Python::attach`, mirroring how the path already reuses `resolve_snapshot_id`), so the two implementations can't drift apart again. - Keeps the SQL `io_config` arg `None` when unset and drops the now-unused `parse_io_config` helper. ### Testing - `tests/io/iceberg/test_iceberg_io_config.py`: unit tests for the resolver's three-tier precedence (explicit wins / table-props fallback / default_io_config fallback). - `tests/sql/test_sql_read_iceberg.py`: end-to-end test asserting the context `default_io_config` reaches the scan's `StorageConfig` via `SELECT * FROM read_iceberg(...)`. - Local: `make build` OK; focused suites 12 passed / 1 skipped; `ruff` + `cargo fmt` + clippy clean on `daft-sql` and `daft-logical-plan`. ## Related Issues

main

5 hours ago

ci: rerun PR checks integration-test-ai failed on HuggingFace Hub rate limiting (HTTP 429 while fetching clip-vit-base-patch32/config.json), unrelated to this change.

jackylee-ch:sql-read-iceberg-io-config

6 hours ago

Merge branch 'main' into sql-read-csv-ignore-corrupt-files #7133 landed the shared block_on_io_runtime helper, which supersedes the inline GIL-release block this branch added to read_csv. Keep main's helper call and drop the local duplicate; the ignore_corrupt_files wiring is unchanged.

jackylee-ch:sql-read-csv-ignore-corrupt-files

7 hours ago

docs(iceberg): drop redundant io_config comment from scan_builder Address review feedback on #7273: the IOConfig precedence is already documented on resolve_iceberg_io_config in daft/io/iceberg/_iceberg.py, so the duplicate comment at the Rust call site is unnecessary.

jackylee-ch:sql-read-iceberg-io-config

7 hours ago

feat(sql): support read_parquet ignore_corrupt_files (#7133) ## Changes Made - Adds `ignore_corrupt_files` support to SQL `read_parquet`, forwarding the named argument into the Parquet scan config. - Adds SQL coverage for skipping corrupt Parquet files and reporting skipped files. - Fixes a GIL↔pyo3-log deadlock: the SQL `read_parquet`/`read_csv`/`read_json` providers ran async schema inference while holding the GIL, so the `ignore_corrupt_files` corrupt-file `log::warn!` (via pyo3-log) could deadlock waiting for the GIL. Schema inference now releases the GIL while blocking, mirroring the DataFrame scan path. This was the cause of the intermittent unit-test CI hang on this branch. Kept `faulthandler_timeout` as a GIL-proof CI watchdog. ## Related Issues Closes #7132 --------- Co-authored-by: Claude <noreply@anthropic.com>

main

7 hours ago

Latest Branches

feat: add Ray support for sample by size#7188

2 days ago

74fadd6

johnquevedo:issue-5690-ray-sample-by-size

-28%

feat(dataframe): add query profiling#7297

4 hours ago

e778b9b

ARDA7787:main

feat(sql): resolve read_iceberg io_config from table properties and default config#7273

7 hours ago

8eaa2aa

jackylee-ch:sql-read-iceberg-io-config

Home Terms Privacy Docs