Avatar for the pola-rs user
pola-rs
polars
BlogDocsChangelog

feat(python): Reduce scan_csv() (and friends') memory usage when using BytesIO

#20649Merged
Comparing
itamarst:scan_csv-memory-usage-bytesio-take-2
(
fe2aff1
) with
main
(
93d0b10
)
CodSpeed Performance Gauge
0%
Improvements
0
Regressions
0
Untouched
41
New
0
Dropped
0
Ignored
1

Benchmarks

Passed

test_groupby_h2oai_q3
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q3
CodSpeed Performance Gauge
+3%
2.1 ms
2.1 ms
test_pdsh_q3
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q3
CodSpeed Performance Gauge
+3%
9.5 ms
9.3 ms
test_filter1
py-polars/tests/benchmark/test_filter.py::test_filter1
CodSpeed Performance Gauge
+3%
764.1 µs
744.5 µs
test_groupby_h2oai_q5
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q5
CodSpeed Performance Gauge
+2%
1.8 ms
1.8 ms
test_single_inequality
py-polars/tests/benchmark/test_join_where.py::test_single_inequality
CodSpeed Performance Gauge
+2%
77.3 ms
76.1 ms
test_groupby_h2oai_q8
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q8
CodSpeed Performance Gauge
+2%
2.6 ms
2.5 ms
test_write_read_scan_large_csv
py-polars/tests/benchmark/test_io.py::test_write_read_scan_large_csv
CodSpeed Performance Gauge
+1%
37.2 ms
36.7 ms
test_pdsh_q16
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q16
CodSpeed Performance Gauge
+1%
12.3 ms
12.2 ms
test_pdsh_q2
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q2
CodSpeed Performance Gauge
+1%
4.5 ms
4.5 ms
test_pdsh_q10
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q10
CodSpeed Performance Gauge
+1%
9.6 ms
9.5 ms
test_groupby_h2oai_q1
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q1
CodSpeed Performance Gauge
+1%
2 ms
2 ms
test_pdsh_q4
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q4
CodSpeed Performance Gauge
+1%
7 ms
7 ms
test_pdsh_q21
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q21
CodSpeed Performance Gauge
+1%
143.6 ms
142.7 ms
test_pdsh_q1
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q1
CodSpeed Performance Gauge
0%
25.8 ms
25.7 ms
test_to_numpy_series_zero_copy
py-polars/tests/benchmark/interop/test_numpy.py::test_to_numpy_series_zero_copy
CodSpeed Performance Gauge
0%
127.9 µs
127.8 µs
test_strict_inequalities
py-polars/tests/benchmark/test_join_where.py::test_strict_inequalities
CodSpeed Performance Gauge
0%
183.8 ms
183.8 ms
test_groupby_h2oai_q6
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q6
CodSpeed Performance Gauge
0%
9.5 ms
9.5 ms
test_to_numpy_series_chunked
py-polars/tests/benchmark/interop/test_numpy.py::test_to_numpy_series_chunked
CodSpeed Performance Gauge
0%
273.7 µs
273.6 µs
test_to_numpy_series_with_nulls
py-polars/tests/benchmark/interop/test_numpy.py::test_to_numpy_series_with_nulls
CodSpeed Performance Gauge
0%
326.8 µs
326.9 µs
test_pdsh_q22
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q22
CodSpeed Performance Gauge
0%
13.5 ms
13.5 ms
test_pdsh_q6
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q6
CodSpeed Performance Gauge
0%
4.8 ms
4.9 ms
test_pdsh_q18
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q18
CodSpeed Performance Gauge
0%
12.3 ms
12.3 ms
test_datetime_range_fast_slow_paths
py-polars/tests/unit/functions/range/test_datetime_range.py::test_datetime_range_fast_slow_paths
CodSpeed Performance Gauge
0%
288.1 ms
288.4 ms
test_groupby_h2oai_q9
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q9
CodSpeed Performance Gauge
0%
20.3 ms
20.3 ms
test_non_strict_inequalities
py-polars/tests/benchmark/test_join_where.py::test_non_strict_inequalities
CodSpeed Performance Gauge
0%
188.2 ms
188.5 ms
test_pdsh_q19
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q19
CodSpeed Performance Gauge
0%
9.8 ms
9.8 ms
test_pdsh_q12
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q12
CodSpeed Performance Gauge
0%
9 ms
9 ms
test_pdsh_q13
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q13
CodSpeed Performance Gauge
-1%
21.9 ms
22 ms
test_pdsh_q20
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q20
CodSpeed Performance Gauge
-1%
8.4 ms
8.4 ms
test_groupby_h2oai_q10
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q10
CodSpeed Performance Gauge
-1%
5.8 ms
5.8 ms
test_pdsh_q17
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q17
CodSpeed Performance Gauge
-1%
7.5 ms
7.6 ms
test_pdsh_q9
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q9
CodSpeed Performance Gauge
-1%
27.9 ms
28.3 ms
test_pdsh_q8
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q8
CodSpeed Performance Gauge
-2%
8.1 ms
8.2 ms
test_pdsh_q5
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q5
CodSpeed Performance Gauge
-2%
6.6 ms
6.7 ms
test_pdsh_q7
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q7
CodSpeed Performance Gauge
-2%
10 ms
10.2 ms
test_pdsh_q14
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q14
CodSpeed Performance Gauge
-2%
4 ms
4.1 ms
test_pdsh_q11
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q11
CodSpeed Performance Gauge
-3%
3.5 ms
3.7 ms
test_groupby_h2oai_q7
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q7
CodSpeed Performance Gauge
-3%
1.9 ms
2 ms
test_groupby_h2oai_q2
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q2
CodSpeed Performance Gauge
-5%
3.8 ms
4 ms
test_groupby_h2oai_q4
py-polars/tests/benchmark/test_group_by.py::test_groupby_h2oai_q4
CodSpeed Performance Gauge
-6%
1.8 ms
1.9 ms
test_pdsh_q15
py-polars/tests/benchmark/test_pdsh.py::test_pdsh_q15
CodSpeed Performance Gauge
-7%
3.6 ms
3.8 ms

Ignored

test_filter2Ignored
py-polars/tests/benchmark/test_filter.py::test_filter2
CodSpeed Performance Gauge
-3%
1.1 ms
1.1 ms

Commits

Click on a commit to change the comparison range
Base
main
93d0b10
-1%
Put sleep in correct location
cfc5b69
18 days ago
by pythonspeed
+1%
Reformat
fa57f53
18 days ago
by pythonspeed
-2%
Handle BytesIO in better location for scan_csv and friends.
b422136
17 days ago
by pythonspeed
+1%
Newer cargo fmt.
fe2aff1
14 days ago
by pythonspeed
© 2025 CodSpeed Technology
Home Terms PrivacyDocs