Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

style: apply rustfmt
everettVT/hf-bucket-errors
26 minutes ago
perf(inline-agg): pack two-string-column keys into u64 for typed FNV grouping (#6924) ## Summary Extends #6585 item 3 (and builds on PR #6748). For the two-string-column groupby shape with sufficiently long string keys, this change packs the two u32 symbol IDs produced by `symbolize_column` into a single u64 key and groups against a typed `FnvHashMap<u64, u32>` in a tight integer loop. No per-row comparator closure, no `IndexHash`, no dynamic-typed `Series` equality dispatch. PR #6748 already symbolizes Utf8/Binary group-by columns, but the symbolized columns are still fed into the generic multi-column hash path (`agg_generic_hash_path`), which keeps the comparator-closure overhead. This PR captures the remaining benefit for two-string-col shapes via a dedicated typed map. Grouping semantics and final query results are unchanged. Short-string shapes (e.g. TPC-H Q1's CHAR(1) `l_returnflag` / `l_linestatus`) fall through to the existing generic hash path on raw strings, exactly as before this PR. ## Why The symbolization in #6748 reduces per-row work but still pays the multi-column hash + comparator-closure cost on the symbolized columns. For the two-string-col shape specifically, two u32 symbols pack losslessly into a u64, which means we can drop the entire generic hash machinery and use a typed `FnvHashMap<u64, u32>` instead. That's the slice this PR captures. ## Conceptual example ``` Input rows: After symbolization: Packed u64 key: key1 | key2 key1_sym | key2_sym (key1_sym << 32) | key2_sym --------|-------- ---------|--------- -------------------------- "alice" | "red" 0 | 0 0x00000000_00000000 "bob" | "red" 1 | 0 0x00000001_00000000 "alice" | "blue" 0 | 1 0x00000000_00000001 "alice" | "red" 0 | 0 0x00000000_00000000 ``` The two symbol spaces sit in disjoint 32-bit halves of the u64 key, so distinct `(sym0, sym1)` pairs always yield distinct packed keys. Null-equals-null is preserved: when a column has nulls, `symbolize_column` reserves symbol ID 0 for null and starts non-null IDs at 1, so both-null rows share a unique packed key and never collide with non-null rows. ## Changes Made - New `agg_packed_u64_path` for exactly two Utf8/Binary group-by columns: symbolize each column into a `Vec<u32>`, pack the pair into a u64, group with `FnvHashMap<u64, u32>` using the same `Vacant`/`Occupied` pattern as the existing single-column fast paths - Reuses `agg_symbolized_path`'s `MIN_AVG_STRING_BYTES_PER_ROW = 16` gate: tallies bytes across both string cols up front and bails when avg bytes per row falls below threshold. Short-string shapes (TPC-H Q1) fall through to the generic hash path on raw strings; for very short keys the symbolize pass costs more than the saved per-row hash/compare work, even with the typed u64 map - New `symbolize_string_col` helper returns `Ok(Some(Vec<u32>))` for Utf8/Binary and `Ok(None)` otherwise; isolates per-dtype null/value-accessor wiring - Dispatch in `agg_groupby_inline` tries the packed-u64 path first for multi-column shapes, then falls through to the existing `agg_symbolized_path`, then to `agg_generic_hash_path`. All other multi-column shapes (3+ columns, int×string, pure int multi-col) are unchanged - New `bench_packed_u64_two_strings` Rust-level benchmark covering two-Utf8-column shapes at varying cardinalities across the count/sum/min/max/count+sum agg matrix ## Benchmarks Long-string two-column shapes (avg bytes per row ≥ 16), measured by running the same benchmark twice on the same machine: once with this PR's packed-u64 dispatch enabled, once with it disabled (falling through to PR #6748's existing `agg_symbolized_path` / `agg_generic_hash_path`). All numbers best-of-10 after 3 warmups, Rust nightly --release, `--test-threads=1`, Linux (WSL). | agg | rows | distinct | PR #6748 inline (ms) | this PR (ms) | speedup | |---|---|---|---|---|---| | count | 1.2M | 8 × 4 = 32 | 61.69 | 57.05 | 1.08x | | sum | 1.2M | 8 × 4 = 32 | 65.88 | 56.05 | 1.18x | | min | 1.2M | 8 × 4 = 32 | 62.78 | 57.66 | 1.09x | | max | 1.2M | 8 × 4 = 32 | 63.70 | 56.23 | 1.13x | | count+sum | 1.2M | 8 × 4 = 32 | 64.76 | 56.75 | 1.14x | | count | 1.2M | 64 × 32 = 2048 | 62.43 | 55.56 | 1.12x | | sum | 1.2M | 64 × 32 = 2048 | 62.62 | 62.70 | 1.00x | | min | 1.2M | 64 × 32 = 2048 | 64.45 | 54.75 | 1.18x | | max | 1.2M | 64 × 32 = 2048 | 63.78 | 56.90 | 1.12x | | count+sum | 1.2M | 64 × 32 = 2048 | 66.10 | 55.77 | 1.19x | | count | 5M | 8 × 4 = 32 | 293.04 | 246.37 | 1.19x | | sum | 5M | 8 × 4 = 32 | 265.15 | 235.03 | 1.13x | | min | 5M | 8 × 4 = 32 | 263.15 | 235.38 | 1.12x | | max | 5M | 8 × 4 = 32 | 267.26 | 238.82 | 1.12x | | count+sum | 5M | 8 × 4 = 32 | 262.31 | 240.04 | 1.09x | | count | 5M | 1000 × 100 = 100k | 268.71 | 241.55 | 1.11x | | sum | 5M | 1000 × 100 = 100k | 269.30 | 243.80 | 1.10x | | min | 5M | 1000 × 100 = 100k | 267.40 | 246.93 | 1.08x | | max | 5M | 1000 × 100 = 100k | 269.19 | 245.68 | 1.10x | | count+sum | 5M | 1000 × 100 = 100k | 270.75 | 239.63 | 1.13x | Across all 20 measured long-string shapes the packed-u64 path is 1.06x-1.19x faster than PR #6748's inline path, with a single 1.00x outlier in the 1.2M × 2048 sum case. This is the "missing benefit" the comparator-closure / `IndexHash` dispatch was hiding. Short-string shapes (avg < 16 bytes per row), TPC-H Q1 included, fall through to the existing generic hash path on raw strings and behave identically to main. An earlier revision of this PR did not gate the packed-u64 path and CodSpeed flagged a -11.22% regression on `test_tpch_sql[1-in-memory-10]`; the gate added in `ee0657c` restores that workload to baseline. Reproduction: ```bash cargo test -p daft-recordbatch --release -- bench_packed_u64_two_strings --nocapture --ignored --test-threads=1 ``` ## Test Plan - 5 new `test_inline_packed_u64_*` cases comparing inline output to the fallback path: - Utf8 × Utf8, no nulls - Utf8 × Utf8, nulls in both columns - Binary × Binary - Utf8 × Binary - Short-string CHAR(1) shape (TPC-H Q1), exercises the gate's skip branch (routes through generic hash path) - All existing `inline_agg` tests still pass (48 total, 0 failures) - New `bench_packed_u64_two_strings` benchmark exercises the path end-to-end at scale across the count/sum/min/max/count+sum matrix ## Related Issues Part of #6585 (deeper specialization of Item 3, on top of PR #6748).
main
3 hours ago
Disable test for now
arpatilmh:abhijeet/deltalake-s3-conditional-put
14 hours ago
fix(native): avoid Python event loop for result iteration
everySympathy:codex/fix-native-executor-result-loop
15 hours ago

Latest Branches

CodSpeed Performance Gauge
0%
fix: normalize HF bucket tree paths#7218
1 hour ago
6c3a7c9
everettVT/hf-bucket-errors
CodSpeed Performance Gauge
0%
fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames#7229
2 hours ago
95a194f
AnayGarodia:fix/5172-video-frames-mkv-webm
CodSpeed Performance Gauge
0%
2 hours ago
31d0108
AnayGarodia:fix/5539-oss-storage-options
© 2026 CodSpeed Technology
Home Terms Privacy Docs