Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

feat(functions): add try_divide for PySpark parity (#7210) ## Summary Adds a `try_divide` scalar function: true division that returns null where the divisor is 0, instead of infinity, NaN, or an error. This mirrors Spark's `try_divide` semantics, where a zero divisor yields null for every numeric type. ## Why Part of the PySpark function parity effort tracked in #3793. Daft's `/` follows IEEE 754 (`1/0` gives `inf`, `0.0/0.0` gives `NaN`), and integer floor division by zero currently panics a worker thread, so a null-on-zero-divisor variant gives users a safe option that matches Spark. Follows the `pmod` (#6801) pattern: a Rust `ScalarUDF` in `src/daft-functions/src/numeric/` with a thin Python wrapper. ## Changes Made - `src/daft-functions/src/numeric/try_divide.rs`: new `TryDivide` ScalarUDF. Computes the quotient with Daft's existing `Series` division, then nulls rows where the divisor equals zero via a broadcast comparison and `if_else`. Return type comes from the same `InferDataType` division inference used by the `/` operator (integer inputs produce `Float64`). - `src/daft-functions/src/numeric/mod.rs`: register `TryDivide`. - `daft/functions/numeric.py`: `try_divide(dividend, divisor)` wrapper. - `daft/functions/__init__.py`: export `try_divide`. - `tests/recordbatch/numeric/test_numeric.py`: 7 tests covering ints, floats (including a `-0.0` divisor), unsigned ints, output dtype, broadcasting in both directions, and the non-numeric error. ## Behavior - `try_divide(1, 0)` returns null; `try_divide(6, 3)` returns `2.0` (`Float64`, same inference as `/`). - `try_divide(1.0, 0.0)`, `try_divide(-1.0, 0.0)`, and `try_divide(0.0, -0.0)` all return null, matching Spark, where the regular operators would give `inf`, `-inf`, and `NaN`. Daft's float equality is bitwise, so the implementation adds zero to the divisor before comparing (`-0.0 + 0.0` is `+0.0` in IEEE 754) to catch negative zero. - Nulls propagate; non-numeric inputs raise a `TypeError` at planning time. - Decimal inputs are rejected by the same `is_numeric` gate `pmod` uses (Daft's `is_numeric` excludes `Decimal128`). Spark's `try_divide` supports decimals; this can be lifted in a follow-up if decimal division support is wanted here. - Available in SQL through the function registry: `SELECT try_divide(1, 0)` returns null. ## Test Plan - `pytest tests/recordbatch/numeric/test_numeric.py`: 392 passed (7 new) - `pytest --doctest-modules daft/functions/numeric.py`: 5 passed - `cargo check -p daft-functions --lib` clean - `cargo fmt -p daft-functions --check` clean - `cargo clippy -p daft-functions --lib --no-deps` clean - End-to-end check through the DataFrame API and `daft.sql` ## Related Issues Part of #3793.
main
10 hours ago
refactor(io): migrate Delta Lake scans to DataSource (#7300) ## Changes Made * Replace `DeltaLakeScanOperator` with `DeltaLakeDataSource` and route Python and SQL reads through the DataSource bridge. * Preserve partition pruning, file statistics, limit handling, storage configuration, and deletion-vector checks. * Allow `DataSourceTask.parquet` to accept a `ParquetSourceConfig` so Delta column-mapping field IDs continue to reach the native Parquet reader. Validation: * `uvx ruff@0.14.10 check daft/daft/__init__.pyi daft/io/source.py daft/io/delta_lake/_deltalake.py daft/io/delta_lake/delta_lake_scan.py` * `uvx ruff@0.14.10 format --check daft/daft/__init__.pyi daft/io/source.py daft/io/delta_lake/_deltalake.py daft/io/delta_lake/delta_lake_scan.py` * `cargo fmt --all -- --check` * `PYO3_PYTHON=python3 CARGO_TARGET_DIR=/tmp/daft-target-7294 cargo check -p daft-scan -p daft-logical-plan --features daft-scan/python,daft-logical-plan/python` No new tests were added because this is a behavior-preserving migration and the existing Delta Lake integration suite covers basic Python/SQL reads, pushdowns, limits, and column mapping. A local `maturin develop` build did not complete within the available WSL1 execution window, so runtime integration tests were not run locally; this draft PR will use the repository CI for that coverage before review. AI disclosure: This change was produced with material AI assistance. I reviewed the complete diff and verified the Rust/Python boundary with the checks listed above. ## Related Issues Fixes Eventual-Inc/Daft#7294 Signed-off-by: ihopenre-eng <247072151+ihopenre-eng@users.noreply.github.com> Co-authored-by: ihopenre-eng <247072151+ihopenre-eng@users.noreply.github.com>
main
12 hours ago
fix(flotilla): enable in-place restarts for RaySwordfishActor to avoid node-level compute loss RaySwordfishActor was created with Ray's default max_restarts=0, so any actor crash (e.g. OOM kill) was terminal: in-flight calls raised ActorDiedError, the dispatcher evicted the whole node from the worker pool, and the node's compute stayed lost until a later worker-refresh cycle re-created an actor on it. Meanwhile all requeued tasks piled onto the remaining nodes, risking cascading OOMs. Set max_restarts (default 4, matching MAX_UDFACTOR_ACTOR_RESTARTS in ray_actor_pool_udf.py) on the swordfish actor options so Ray restarts a crashed actor on the same node. While restarting, in-flight calls surface ActorUnavailableError, which already maps to RayTaskResult.worker_unavailable() -> the task is requeued WITHOUT evicting the node. max_task_retries intentionally stays 0: task re-execution is owned by Daft's dispatcher. The value is overridable via DAFT_SWORDFISH_ACTOR_MAX_RESTARTS (-1 = infinite, 0 = restore old behavior). Upstream community context (Eventual-Inc/Daft): - PR #4628 (feat(flotilla): Fault tolerance) added dispatcher-level rescheduling on WorkerDied/WorkerUnavailable but never configured actor restarts, leaving the WorkerUnavailable fast path unreachable for crashed actors. - PR #7116 / issue #7104 bumped the ray floor to >=2.11.0 specifically for ActorUnavailableError; this change makes that signal actually fire for crashed-but-restartable actors. Tests: - start_ray_workers passes max_restarts>0 by default and honors the env override (tests/ray/test_flotilla_worker_startup.py) - env parsing fallback + RaySwordfishTaskHandle actor-error mapping contract (tests/ray/test_flotilla_actor_fault_tolerance.py)
DogerW666:test-bug-fix
1 day ago

Latest Branches

CodSpeed Performance Gauge
0%
fix(io): support writing to hdfs:// URLs that include a port#7307
1 day ago
1b99b7c
Lucas61000:fix-hdfs-write-port
CodSpeed Performance Gauge
0%
fix(flotilla): enable in-place restarts for RaySwordfishActor to avoiโ€ฆ#7306
1 day ago
5b03dec
DogerW666:test-bug-fix
CodSpeed Performance Gauge
0%
feat: add rint and bround math functions#7222
1 day ago
0011d5d
cmirandavega:feat/rint-bround
ยฉ 2026 CodSpeed Technology
Home Terms Privacy Docs