vortex-data/vortex

vortex

Performance History

Latest Results

perf(array): gather dict byte values straight into the offset builder `Dict::append_to_builder` took the dictionary values to full logical length via `take_canonical` and then appended that result, so exporting a dictionary-encoded string column allocated a full-length intermediate array which was immediately copied into the builder a second time. Dictionaries are usually far smaller than the arrays they encode, so when the target is a `DynVarBinBuilder` and the dtype is Utf8 or Binary, materialize only the values and gather by code. Resolving a value is then an O(1) read out of the views slice and data buffers, both hoisted out of the loop, and null codes are filled a run at a time with `for_each_set_index`. One fewer full-length allocation and copy. The new branch keeps the same guards as the generic path it precedes -- non-empty, primitive codes, not definitely-all-null codes -- and adds an explicit dtype check, so anything it does not handle still falls through unchanged. A code pointing at a null dictionary entry appends a null, matching the canonical route. The accompanying benchmark is sized to fit the CI time budget, and at that size the change is within run-to-run noise, so it stands as a regression guard rather than as evidence of a speedup. Correctness is asserted against `execute::<Canonical>` so the gather and canonical routes are compared directly. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

claude/dict-varbin-append

2 hours ago

perf(array): hoist per-row work out of append_varbinview `DynVarBinBuilder::append_varbinview` is the fallback every string encoding without a direct branch reaches through canonicalization, including Dict, Sparse, RunEnd, and anything on the default `VTable::append_to_builder`. Converting views to offsets has to visit each value, but the work around that walk does not have to be per-row: - `bytes_at` re-resolved the views slice and returned an owned `ByteBuffer` on every call. Resolve the views and data buffer slices once and borrow from them instead. Inlined values now come straight out of the view rather than round-tripping the views handle through `into_byte_buffer().slice_ref(..)`. - Validity was walked a bit at a time with `bit_buffer().iter().enumerate()`. Use `for_each_set_index`, which processes a `u64` word at a time, and fill the gaps between set bits with `append_n_nulls`. - The byte total is a sum over the fixed-width views and needs no value access, so the data buffer is sized once via a new `reserve_data` rather than reallocating as it grows. `reserve_exact` takes a row count and cannot serve this purpose. The accompanying benchmark is sized to fit the CI time budget, and at those sizes the change is within run-to-run noise, so it stands as a regression guard rather than as evidence of a speedup. Output is unchanged, which the extended tests assert against the source array. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

claude/varbin-builder-bulk-append

2 hours ago

DCO Remediation Commit for forwardxu <forwardxu@apache.org> I, forwardxu <forwardxu@apache.org>, hereby add my Signed-off-by to this commit: a523c12a3b27449b6eb360752374418e7e62caec I, forwardxu <forwardxu@apache.org>, hereby add my Signed-off-by to this commit: 1e0e128e59547b8cd4fbb9d708a4d9f9ef9645f3 I, forwardxu <forwardxu@apache.org>, hereby add my Signed-off-by to this commit: dc238e1ce3b93afbd6139cc38e1b35b5a4ae7f53 Signed-off-by: forwardxu <forwardxu@apache.org>

XuQianJin-Stars:feature/opendal-cos

2 hours ago

perf(array): gather dict byte values straight into the offset builder `Dict::append_to_builder` took the dictionary values to full logical length via `take_canonical` and then appended that result, so exporting a dictionary-encoded string column allocated a full-length intermediate array which was immediately copied into the builder a second time. Dictionaries are usually far smaller than the arrays they encode, so when the target is a `DynVarBinBuilder` and the dtype is Utf8 or Binary, materialize only the values and gather by code. Resolving a value is then an O(1) read out of the views slice and data buffers, both hoisted out of the loop, and null codes are filled a run at a time with `for_each_set_index`. The new branch keeps the same guards as the generic path it precedes -- non-empty, primitive codes, not definitely-all-null codes -- and adds an explicit dtype check, so anything it does not handle still falls through unchanged. A code pointing at a null dictionary entry appends a null, matching the canonical route. On measurement: at the benchmark size here (4096 rows) this is within noise against the previous implementation. The saving is the skipped full-length intermediate, which only dominates once allocation cost does; it is clearly visible at 65536 rows but that overshoots the benchmark time budget, so the tracked shapes do not demonstrate it. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

claude/dict-varbin-append

2 hours ago

chore(arrow): trim guard comments Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

claude/arrow-target-dtype-check

2 hours ago

perf(array): hoist per-row work out of append_varbinview `DynVarBinBuilder::append_varbinview` is the fallback every string encoding without a direct branch reaches through canonicalization, including Dict, Sparse, RunEnd, and anything on the default `VTable::append_to_builder`. Converting views to offsets has to visit each value, but the work around that walk does not have to be per-row: - `bytes_at` re-resolved the views slice and returned an owned `ByteBuffer` on every call. Resolve the views and data buffer slices once and borrow from them instead. Inlined values now come straight out of the view rather than round-tripping the views handle through `into_byte_buffer().slice_ref(..)`. - Validity was walked a bit at a time with `bit_buffer().iter().enumerate()`. Use `for_each_set_index`, which processes a `u64` word at a time, and fill the gaps between set bits with `append_n_nulls`. - The byte total is a sum over the fixed-width views and needs no value access, so the data buffer is sized once via a new `reserve_data` rather than reallocating as it grows. `reserve_exact` takes a row count and cannot serve this purpose. On measurement: at the benchmark sizes here (4096 and 16384) this is within noise against the previous implementation. The data buffer reservation is what carries the win, and it only becomes measurable once reallocation cost starts to dominate. Toggling only the `reserve_data` call on an otherwise identical build at 65536 rows moves `heap_all_valid` from 2.37ms to 637us. That size overshoots the benchmark time budget, so it is not tracked here. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

claude/varbin-builder-bulk-append

2 hours ago

fix(deps): update lance benchmark dependencies to v59

renovate/major-lance-bench

2 hours ago

develop

2 hours ago

Latest Branches

perf(array): gather dict byte values straight into the offset builder#8995

3 hours ago

3e48f58

claude/dict-varbin-append

-21%

perf(array): hoist per-row work out of append_varbinview#8994

3 hours ago

8110b99

claude/varbin-builder-bulk-append

feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API#8845

2 hours ago

6b37b3d

XuQianJin-Stars:feature/opendal-cos

Home Terms Privacy Docs