Jay-ju:feature/udf-ray-options - Branch - Eventual-Inc/Daft

feat(udf): support ray_options and resource overrides in UDF v2

#5982

Comparing

Jay-ju:feature/udf-ray-options

(

2d0a0c2

) with

main

(

8e15689

)

Untouched: 24

Ignored: 4

Benchmarks

Passed

test_tpch_sql[1-in-memory-2]

tests/benchmarks/test_local_tpch.py

+5%

167 ms159.6 ms

test_tpch[1-in-memory-5]

tests/benchmarks/test_local_tpch.py

+2%

132.3 ms129.8 ms

test_tpch[1-in-memory-4]

tests/benchmarks/test_local_tpch.py

+2%

90.1 ms88.8 ms

test_show[1 Small File]

tests/benchmarks/test_interactive_reads.py

+1%

12.3 ms12.2 ms

test_tpch_sql[1-in-memory-7]

tests/benchmarks/test_local_tpch.py

+1%

121.6 ms120.4 ms

test_tpch_sql[1-in-memory-8]

tests/benchmarks/test_local_tpch.py

+1%

135.2 ms134.1 ms

test_tpch_sql[1-in-memory-4]

tests/benchmarks/test_local_tpch.py

+1%

90.7 ms90.2 ms

test_tpch[1-in-memory-3]

tests/benchmarks/test_local_tpch.py

+1%

125.8 ms125.2 ms

test_tpch[1-in-memory-10]

tests/benchmarks/test_local_tpch.py

190.8 ms189.9 ms

test_tpch_sql[1-in-memory-5]

tests/benchmarks/test_local_tpch.py

119.4 ms118.9 ms

test_tpch_sql[1-in-memory-9]

tests/benchmarks/test_local_tpch.py

265.5 ms264.6 ms

test_tpch[1-in-memory-8]

tests/benchmarks/test_local_tpch.py

148.3 ms147.9 ms

test_tpch_sql[1-in-memory-6]

tests/benchmarks/test_local_tpch.py

33.4 ms33.3 ms

test_tpch_sql[1-in-memory-10]

tests/benchmarks/test_local_tpch.py

183.6 ms183.2 ms

test_tpch[1-in-memory-2]

tests/benchmarks/test_local_tpch.py

56.1 ms56 ms

test_tpch_sql[1-in-memory-1]

tests/benchmarks/test_local_tpch.py

406.7 ms406.3 ms

test_tpch_sql[1-in-memory-3]

tests/benchmarks/test_local_tpch.py

118.9 ms118.8 ms

test_tpch[1-in-memory-6]

tests/benchmarks/test_local_tpch.py

32.6 ms32.6 ms

test_tpch[1-in-memory-9]

tests/benchmarks/test_local_tpch.py

271.9 ms272.1 ms

test_iter_rows_first_row[1 Small File]

tests/benchmarks/test_interactive_reads.py

35.4 ms35.4 ms

test_tpch[1-in-memory-7]

tests/benchmarks/test_local_tpch.py

128.1 ms128.7 ms

test_tpch[1-in-memory-1]

tests/benchmarks/test_local_tpch.py

405.6 ms407.5 ms

test_explain[100 Small Files]

tests/benchmarks/test_interactive_reads.py

-1%

12.1 ms12.2 ms

test_count[100 Small Files]

tests/benchmarks/test_interactive_reads.py

-1%

56.6 ms57.2 ms

Ignored

test_count[1 Small File]

tests/benchmarks/test_interactive_reads.py

Ignored

3.7 ms3.7 ms

test_explain[1 Small File]

tests/benchmarks/test_interactive_reads.py

Ignored

2.1 ms2.1 ms

test_iter_rows_first_row[100 Small Files]

tests/benchmarks/test_interactive_reads.py

Ignored

-24%

187 ms245.9 ms

test_show[100 Small Files]

tests/benchmarks/test_interactive_reads.py

Ignored

22.8 ms22.7 ms

Commits

Click on a commit to change the comparison range

Base

main

8e15689

+0.44%

feat(udf): support ray_options and resource overrides in UDF v2 This PR adds support for passing `ray_options` and overriding resource requirements in Daft UDF v2 (`@daft.func` and `@daft.cls`). Key changes: - Update `row_wise_udf` and `batch_udf` bindings to accept `ray_options`, `memory_bytes`, and `cpus`. - Add `override_options` and `with_concurrency` methods to `Func` class in `daft/udf/udf_v2.py` for dynamic configuration. - Propagate these options from Python to the Rust Logical Plan. - Update type stubs (`.pyi`) to match new Rust signatures. - Add integration tests verifying both `explain()` output and execution results. Verified with `pre-commit` and new test cases.

2d0a0c2

7 days ago

by Jay-ju

Home Terms Privacy Docs