colin/streaming-physical-writes - Branch - Eventual-Inc/Daft

[FEAT] Streaming physical writes for native executor

#2992Merged

Comparing

colin/streaming-physical-writes

(

b67499e

) with

main

(

301cd48

)

Untouched: 14

Ignored: 4

Benchmarks

Passed

test_tpch[1-in-memory-native-1]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-1]

+2%

449.8 ms439.4 ms

test_tpch[1-in-memory-native-10]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-10]

+1%

196.4 ms193.6 ms

test_tpch[1-in-memory-native-2]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-2]

+1%

99.3 ms98.2 ms

test_explain[100 Small Files]

tests/benchmarks/test_interactive_reads.py::test_explain[100 Small Files]

+1%

7 ms6.9 ms

test_count[100 Small Files]

tests/benchmarks/test_interactive_reads.py::test_count[100 Small Files]

+1%

127.6 ms126.7 ms

test_tpch[1-in-memory-native-8]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-8]

340.8 ms339.8 ms

test_tpch[1-in-memory-native-4]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-4]

144.1 ms143.7 ms

test_tpch[1-in-memory-native-7]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-7]

131.7 ms131.6 ms

test_iter_rows_first_row[1 Small File]

tests/benchmarks/test_interactive_reads.py::test_iter_rows_first_row[1 Small File]

101.9 ms101.8 ms

test_tpch[1-in-memory-native-6]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-6]

28.8 ms28.8 ms

test_show[1 Small File]

tests/benchmarks/test_interactive_reads.py::test_show[1 Small File]

-1%

13.1 ms13.2 ms

test_tpch[1-in-memory-native-5]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-5]

-1%

376.6 ms380.5 ms

test_tpch[1-in-memory-native-3]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-3]

-1%

138.7 ms140.3 ms

test_tpch[1-in-memory-native-9]

tests/benchmarks/test_local_tpch.py::test_tpch[1-in-memory-native-9]

-2%

366.2 ms374.1 ms

Ignored

test_iter_rows_first_row[100 Small Files]

tests/benchmarks/test_interactive_reads.py::test_iter_rows_first_row[100 Small Files]

Ignored

9.4 s9.5 s

test_show[100 Small Files]

tests/benchmarks/test_interactive_reads.py::test_show[100 Small Files]

Ignored

16.2 ms16.2 ms

test_count[1 Small File]

tests/benchmarks/test_interactive_reads.py::test_count[1 Small File]

Ignored

-3%

4.1 ms4.2 ms

test_explain[1 Small File]

tests/benchmarks/test_interactive_reads.py::test_explain[1 Small File]

Ignored

1.8 ms1.8 ms

Commits

Click on a commit to change the comparison range

Base

main

301cd48

-0.23%

physical

4d5af1a

1 year ago

+0.2%

iceberg fix

823e889

1 year ago

+0.03%

rename to factory

3854cd2

1 year ago

-0.27%

dont use table str for hash

56c8416

11 months ago

+0.25%

Merge branch main into colin/streaming-physical-writes

4db34f0

11 months ago

-0.4%

feedback

9ed8e84

11 months ago

-0.34%

add comments

16f3f7d

11 months ago

+0.46%

make write and close abstract

11d9244

11 months ago

+0.46%

agg fix

85ab44c

11 months ago

-0.57%

move buffer

0ce9dcc

11 months ago

-0.23%

remove unused deps

03dcfcb

11 months ago

+0.34%

if we move all file writer stuff to daft writer then the ray runner can use it too

bc9c7fa

11 months ago

+0.71%

address feedback

072ae6e

11 months ago

-0.48%

add unit tests for batch and file writers

b67499e

11 months ago

Home Terms Privacy Docs