Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

refactor: migrate IcebergScanOperator to DataSource - `IcebergScanOperator(ScanOperator)` → `IcebergDataSource(DataSource)` - `to_scan_tasks()` (sync iterator of `ScanTask`) → `async def get_tasks()` (async generator of `DataSourceTask`) - Data files are read via `DataSourceTask.parquet()`, which now accepts `iceberg_delete_files` so positional delete files flow through to the native Parquet reader - Count pushdown yields a metadata-only `_IcebergCountTask(DataSourceTask)`; `PyDataSourceWrapper` forwards `supports_count_pushdown()` from the Python source so the optimizer rule still applies - Partition pruning is performed inside `get_tasks` using `pushdowns.partition_filters` - SQL scan path (`scan_builder.rs`) constructs `IcebergDataSource` via `ScanOperatorHandle::from_data_source` Closes #7296
Lucas61000:issue-7296
2 hours ago
chore(format): fixed spacing of files
cpicazo8304:feat/bit-length
14 hours ago
feat(functions): add try_divide for PySpark parity (#7210) ## Summary Adds a `try_divide` scalar function: true division that returns null where the divisor is 0, instead of infinity, NaN, or an error. This mirrors Spark's `try_divide` semantics, where a zero divisor yields null for every numeric type. ## Why Part of the PySpark function parity effort tracked in #3793. Daft's `/` follows IEEE 754 (`1/0` gives `inf`, `0.0/0.0` gives `NaN`), and integer floor division by zero currently panics a worker thread, so a null-on-zero-divisor variant gives users a safe option that matches Spark. Follows the `pmod` (#6801) pattern: a Rust `ScalarUDF` in `src/daft-functions/src/numeric/` with a thin Python wrapper. ## Changes Made - `src/daft-functions/src/numeric/try_divide.rs`: new `TryDivide` ScalarUDF. Computes the quotient with Daft's existing `Series` division, then nulls rows where the divisor equals zero via a broadcast comparison and `if_else`. Return type comes from the same `InferDataType` division inference used by the `/` operator (integer inputs produce `Float64`). - `src/daft-functions/src/numeric/mod.rs`: register `TryDivide`. - `daft/functions/numeric.py`: `try_divide(dividend, divisor)` wrapper. - `daft/functions/__init__.py`: export `try_divide`. - `tests/recordbatch/numeric/test_numeric.py`: 7 tests covering ints, floats (including a `-0.0` divisor), unsigned ints, output dtype, broadcasting in both directions, and the non-numeric error. ## Behavior - `try_divide(1, 0)` returns null; `try_divide(6, 3)` returns `2.0` (`Float64`, same inference as `/`). - `try_divide(1.0, 0.0)`, `try_divide(-1.0, 0.0)`, and `try_divide(0.0, -0.0)` all return null, matching Spark, where the regular operators would give `inf`, `-inf`, and `NaN`. Daft's float equality is bitwise, so the implementation adds zero to the divisor before comparing (`-0.0 + 0.0` is `+0.0` in IEEE 754) to catch negative zero. - Nulls propagate; non-numeric inputs raise a `TypeError` at planning time. - Decimal inputs are rejected by the same `is_numeric` gate `pmod` uses (Daft's `is_numeric` excludes `Decimal128`). Spark's `try_divide` supports decimals; this can be lifted in a follow-up if decimal division support is wanted here. - Available in SQL through the function registry: `SELECT try_divide(1, 0)` returns null. ## Test Plan - `pytest tests/recordbatch/numeric/test_numeric.py`: 392 passed (7 new) - `pytest --doctest-modules daft/functions/numeric.py`: 5 passed - `cargo check -p daft-functions --lib` clean - `cargo fmt -p daft-functions --check` clean - `cargo clippy -p daft-functions --lib --no-deps` clean - End-to-end check through the DataFrame API and `daft.sql` ## Related Issues Part of #3793.
main
3 days ago
refactor(io): migrate Delta Lake scans to DataSource (#7300) ## Changes Made * Replace `DeltaLakeScanOperator` with `DeltaLakeDataSource` and route Python and SQL reads through the DataSource bridge. * Preserve partition pruning, file statistics, limit handling, storage configuration, and deletion-vector checks. * Allow `DataSourceTask.parquet` to accept a `ParquetSourceConfig` so Delta column-mapping field IDs continue to reach the native Parquet reader. Validation: * `uvx ruff@0.14.10 check daft/daft/__init__.pyi daft/io/source.py daft/io/delta_lake/_deltalake.py daft/io/delta_lake/delta_lake_scan.py` * `uvx ruff@0.14.10 format --check daft/daft/__init__.pyi daft/io/source.py daft/io/delta_lake/_deltalake.py daft/io/delta_lake/delta_lake_scan.py` * `cargo fmt --all -- --check` * `PYO3_PYTHON=python3 CARGO_TARGET_DIR=/tmp/daft-target-7294 cargo check -p daft-scan -p daft-logical-plan --features daft-scan/python,daft-logical-plan/python` No new tests were added because this is a behavior-preserving migration and the existing Delta Lake integration suite covers basic Python/SQL reads, pushdowns, limits, and column mapping. A local `maturin develop` build did not complete within the available WSL1 execution window, so runtime integration tests were not run locally; this draft PR will use the repository CI for that coverage before review. AI disclosure: This change was produced with material AI assistance. I reviewed the complete diff and verified the Rust/Python boundary with the checks listed above. ## Related Issues Fixes Eventual-Inc/Daft#7294 Signed-off-by: ihopenre-eng <247072151+ihopenre-eng@users.noreply.github.com> Co-authored-by: ihopenre-eng <247072151+ihopenre-eng@users.noreply.github.com>
main
3 days ago

Latest Branches

CodSpeed Performance Gauge
0%
refactor: migrate IcebergScanOperator to DataSource#7311
13 hours ago
3457147
Lucas61000:issue-7296
CodSpeed Performance Gauge
0%
6 hours ago
e591fc2
jiangxt2:fix/clickhouse-percentile
CodSpeed Performance Gauge
0%
13 days ago
a0ad850
cpicazo8304:feat/bit-length
© 2026 CodSpeed Technology
Home Terms Privacy Docs