add-varbin-compressor - Branch - vortex-data/vortex

feat: add VarBin compressor to btrblocks string compressor

#5360Closed

Comparing

add-varbin-compressor

(

ccd0736

) with

develop

(

76aeed6

)

Untouched: 1077

New: 7

Archived: 943

Benchmarks

2027 total

decompress[("datetime_for_bp", 0x4640060)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A34.9 ms

decompress[("dict_fsst_varbin_bp_string", 0x463f3a0)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A14.5 ms

decompress[("alp_for_bp_f64", 0x463d1a0)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A24.2 ms

decompress[("dict_varbinview_string", 0x463dbc0)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A14.7 ms

decompress[("for_bp_u64", 0x463ca50)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A2.5 ms

decompress[("dict_fsst_varbin_string", 0x463ef00)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A14.5 ms

decompress[("runend_for_bp_u32", 0x463e050)]

vortex/benches/common_encoding_tree_throughput.rs

N/A

N/A2 ms

encode_varbin[(10000, 8)]

vortex-array/benches/dict_compress.rs

+4%

1,012.8 µs971.7 µs

encode_varbin[(1000, 8)]

vortex-array/benches/dict_compress.rs

+3%

163 µs158.4 µs

push_arrow_buffer[128]

vortex-buffer/benches/vortex_buffer.rs

+3%

1.2 µs1.2 µs

map_new_output[128]

vortex-buffer/benches/vortex_buffer.rs

+1%

2.2 µs2.1 µs

take_map[(0.05, 1.0)]

vortex-array/benches/take_patches.rs

+1%

1.9 ms1.9 ms

take_map[(0.005, 0.1)]

vortex-array/benches/take_patches.rs

+1%

187.4 µs185.6 µs

encode_varbin[(1000, 32)]

vortex-array/benches/dict_compress.rs

+1%

165.3 µs164 µs

take_map[(0.01, 0.1)]

vortex-array/benches/take_patches.rs

+1%

205.3 µs203.8 µs

take_map[(0.05, 0.5)]

vortex-array/benches/take_patches.rs

+1%

1 ms1 ms

take_map[(0.01, 0.5)]

vortex-array/benches/take_patches.rs

+1%

773 µs768 µs

take_struct_simple

vortex-array/benches/take_struct.rs

+1%

75.5 µs75 µs

fsst_decompress_string

vortex/benches/single_encoding_throughput.rs

+1%

38.4 ms38.2 ms

encode_varbinview[(1000, 512)]

vortex-array/benches/dict_compress.rs

213.5 µs212.5 µs

chunked_dict_primitive_into_canonical[f32, (1000, 10, 100)]

vortex-array/benches/chunked_dict_builder.rs

852.7 µs848.8 µs

take_struct_sequential_indices

vortex-array/benches/take_struct.rs

60.3 µs60 µs

take_indices[(1000, 4)]

encodings/runend/benches/run_end_compress.rs

31.9 µs31.8 µs

push_arrow_buffer[1024]

vortex-buffer/benches/vortex_buffer.rs

7.8 µs7.8 µs

canonical_into_non_nullable[(10000, 1, 0.01)]

encodings/fastlanes/benches/canonicalize_bench.rs

40.8 µs40.7 µs

Commits

Click on a commit to change the comparison range

Base

develop

76aeed6

+0.04%

feat: add VarBin compressor to btrblocks string compressor

ab71b6b

3 months ago

by joseph-isaacs

-0.04%

refactor: use to_arrow_preferred for VarBinView to VarBin conversion

3ddbe35

3 months ago

by joseph-isaacs

-14.69%

fix: use into_arrow with specific DataType for VarBin conversion

0415270

3 months ago

by joseph-isaacs

+14.71%

fix: assign VarBinScheme to code 5 without renumbering existing schemes

a6e48f6

3 months ago

by joseph-isaacs

fix: correct typo offest -> offset

5dff651

3 months ago

by joseph-isaacs

-0.02%

fix: correctly calculate offset array size based on string bytes

e17e8b8

3 months ago

by joseph-isaacs

fix: return 0.0 compression ratio to prevent automatic selection

ae0840a

3 months ago

by joseph-isaacs

11e3db8

3 months ago

by joseph-isaacs

-0.07%

bfff5ac

3 months ago

by joseph-isaacs

8472086

3 months ago

by joseph-isaacs

fix: update Python tests to handle string vs string_view encoding differences

ccd0736

3 months ago

by joseph-isaacs

Home Terms Privacy Docs