Avatar for the vortex-data user
vortex-data
vortex
BlogDocsChangelog

Performance History

Latest Results

perf: remove implicit `ListViewArray` rebuild during `take` and `filter` operations (#8048) ## Summary Removes implicit rebuilds from `ListViewArray` `take` and `filter` compute kernels, adds density-introspection methods to support deciding when to rebuild explicitly at materialization boundaries, and defers rebuilding until array export to duckdb and arrow. **Motivation.** The previous code rebuilt the elements buffer eagerly on every `take` or `filter` whose row-fraction dropped below `REBUILD_DENSITY_THRESHOLD`. In an execution tree like `take → take → ...`, an eager mid-pipeline rebuild costs an allocation and a full copy of referenced ranges that the next operator may immediately sparsify away. The row-fraction heuristic was also inaccurate. It doesn't account for per-row size variance, unreferenced elements, and duplicate references. Instead, `ListViewArray::estimate_density` uses sum of `sizes` instead of row-fraction. This will overestimate density when there are overlapping references, but it is typically preferable to not compact. **Changes:** - **Drop implicit rebuild from `TakeReduce::take` and `TakeExecute::take`** - **Drop implicit rebuild from `filter_listview`** - **Add methods to calculate reference density**: `compute_referenced_elements_mask`, `compute_density`, and `estimate_density` - **Estimate density and conditionally rebuild on export boundaries for duckdb and arrow** ## API Changes Adds `ListViewArray::estimate_density`, `ListViewArray::compute_density`, and `ListViewArray::compute_referenced_elements_mask`. ## Testing - New `vortex-array/src/arrays/listview/tests/density.rs` --------- Signed-off-by: Matthew Katz <katz@spiraldb.com> Co-authored-by: Matt Katz <mattkatz@Matts-MacBook-Pro.local> Co-authored-by: Matt Katz <mattkatz@Matts-MBP.localdomain> Co-authored-by: Matthew Katz <katz@spiraldb.com>
develop
6 hours ago

Latest Branches

CodSpeed Performance Gauge
+11%
Fuse validity-aware min/max with the value conversion in primitive cast#8074
1 day ago
7f4dfd7
claude/fast-f-to-t-cast-7JrNR
CodSpeed Performance Gauge
-5%
add vortex-geo crate and WKB extension type#7722
1 day ago
0d0cec8
aduffy/geo-v0
CodSpeed Performance Gauge
+14%
1 day ago
6a44f51
claude/great-edison-jrGY0
© 2026 CodSpeed Technology
Home Terms Privacy Docs