Avatar for the vortex-data user
vortex-data
vortex
BlogDocsChangelog

Perf: Stream conversion from Parquet to Vortex to prevent OOM

#6228Merged
Comparing
ct/p2v-oom
(
fb02ae5
) with
develop
(
0d593a7
)
CodSpeed Performance Gauge
0%
Untouched
1138
Skipped
1265
Archived
119

Benchmarks

2522 total
encode_varbin[(10000, 4)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+8%
1.6 ms1.5 ms
set_indices_arrow_buffer[128]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+5%
1.2 µs1.2 µs
decode_primitives[u8, (1000, 2)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+4%
18.6 µs17.9 µs
true_count_arrow_buffer[128]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+3%
1,005.3 ns976.1 ns
train_compressor[(10000, 16, 4)]
encodings/fsst/benches/fsst_compress.rs
CodSpeed Performance Gauge
+3%
4.8 ms4.7 ms
train_compressor[(10000, 16, 8)]
encodings/fsst/benches/fsst_compress.rs
CodSpeed Performance Gauge
+2%
4.8 ms4.7 ms
decode_primitives[f32, (1000, 2)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+2%
18.8 µs18.4 µs
encode_varbin[(1000, 32)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+2%
236.8 µs232.8 µs
take_map[(0.05, 1.0)]
vortex-array/benches/take_patches.rs
CodSpeed Performance Gauge
+2%
3.1 ms3.1 ms
encode_varbin[(1000, 128)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+1%
271.7 µs267.9 µs
decode_varbin[(1000, 128)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+1%
29.1 µs28.7 µs
append_buffer_arrow_buffer[128]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+1%
4.5 µs4.4 µs
take_map[(0.01, 1.0)]
vortex-array/benches/take_patches.rs
CodSpeed Performance Gauge
+1%
2.5 ms2.5 ms
decode_primitives[f32, (1000, 8)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+1%
18.8 µs18.5 µs
append_n_arrow_buffer[128]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+1%
2.7 µs2.6 µs
push_n_vortex_buffer[u64, 128]
vortex-buffer/benches/vortex_buffer.rs
CodSpeed Performance Gauge
+1%
2.9 µs2.8 µs
set_indices_arrow_buffer[1024]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+1%
2.9 µs2.9 µs
push_n_vortex_buffer[u32, 128]
vortex-buffer/benches/vortex_buffer.rs
CodSpeed Performance Gauge
+1%
2.9 µs2.9 µs
take_map[(0.005, 0.05)]
vortex-array/benches/take_patches.rs
CodSpeed Performance Gauge
+1%
175.1 µs173.4 µs
append_n_arrow_buffer[2048]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+1%
3.1 µs3.1 µs
decode_primitives[f32, (10000, 2)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+1%
54.1 µs53.6 µs
encode_primitives[i64, (1000, 512)]
vortex-array/benches/dict_compress.rs
CodSpeed Performance Gauge
+1%
215.1 µs213.2 µs
append_n_arrow_buffer[1024]
vortex-buffer/benches/vortex_bitbuffer.rs
CodSpeed Performance Gauge
+1%
3.3 µs3.3 µs
expand_buffer[u8, (256, 0.1)]
vortex-compute/benches/expand_buffer.rs
CodSpeed Performance Gauge
+1%
3.3 µs3.3 µs
old_bp_prim_test_between[i16, 2048]
encodings/fastlanes/benches/compute_between.rs::bitpack
CodSpeed Performance Gauge
+1%
124.1 µs123.1 µs

Commits

Click on a commit to change the comparison range
Base
develop
0d593a7
-0.02%
stream conversion from parquet to vortex
fb02ae5
27 days ago
by connortsui20
© 2026 CodSpeed Technology
Home Terms Privacy Docs