Latest Results
feat(vortex-buffer): arch-specialized SIMD bool packing for collect_bool
Replace the bit-at-a-time shift-OR loop behind collect_bool_word /
collect_bool_words with byte-to-bit packing kernels specialized per
architecture, without changing any public signature:
- x86-64 AVX-512BW: one vptestmb packs 64 bools into a u64
- x86-64 AVX2: two vpcmpeqb + vpmovmskb halves
- x86-64 SSE2 (baseline): four pcmpeqb + pmovmskb quarters
- aarch64 NEON (baseline): ushl by bit position + addp reduction tree
- other targets and Miri: branch-free SWAR multiply fallback
The API has two tiers. The default (collect_bool / collect_bool_words,
backed by collect_bool_words_inline) compiles the word loop once with
the widest kernel enabled at compile time (SSE2/NEON on stock targets;
AVX2/AVX-512 under -C target-cpu=native) so the predicate, the
[bool; 64] materialization, and the pack all inline into the caller.
The opt-in tier (collect_bool_multiversioned /
collect_bool_words_multiversioned) compiles the loop with the predicate
inside once per CPU feature level and selects a clone by runtime
feature detection; its docs state the caller's assertion — the
predicate must be trivially cheap, because the #[target_feature] call
boundary deoptimizes non-trivial predicates (an early all-multiversioned
draft regressed FSST fsst_prefix 13-63% this way; the pack kernel choice
itself moves that DFA-bound workload by at most ~4%).
Route the call sites whose predicates are bounds-check-free gathers or
comparisons through the multiversioned tier: From<&[bool]> / From<&[u8]>,
ByteBool boolean decoding, and the primitive and decimal between kernels
(now reading via get_unchecked, sound because the predicate only
receives indices 0..len — a contract now documented on every entry
point together with the bounds-check performance note).
Measured on AVX-512 hardware (divan, 64Ki bits, medians): the fused
word loop with a bool-gather predicate runs 14-24x faster than the old
scalar loop per SIMD level (SSE2 1.7us, AVX2 1.2us, AVX-512 1.1us vs
26.5us); BitBufferMut::from(&[bool]) improves ~43x; a between-shaped
predicate doubles under the multiversioned tier (5.5us vs 11.2us);
fsst_prefix is unchanged or slightly better than before the change.
Generated asm verified per kernel (the fused AVX-512 loop is vmovdqu64 +
vptestmb + kmovq per 64 bools; NEON is 14 instructions, its addp tree
additionally validated against a reference model on 10,004 random
inputs). New benches cover each kernel and both tiers; under CodSpeed
only the shipped entry points are tracked (from_bool_slice,
collect_bool_u32_gt).
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcsiAHXQc4e11yYcP3a3NXclaude/collect-bool-arch-optimizations-bo09ws feat: add CpuKernel for one-time CPU-feature kernel dispatch
Hot kernels re-ran cached CPU-feature checks on every call: each
is_x86_feature_detected! is an atomic load + bit test per call, and sites
like select_in_chunk chained three of them per 64-byte chunk. aarch64
paths were separate cfg blocks at every call site.
Add vortex_buffer::CpuKernel<F>, a LazyLock-like slot for function
pointers: the selector passed to CpuKernel::new runs once on the first
get(), and every later call is a relaxed atomic load, a never-taken
predicted branch, and an indirect call. No macros; the two
transmute_copy calls between F and the atomic pointer are contained in
the type, whose constructor statically asserts F is pointer-sized.
Selectors model both dispatch dimensions in plain code: one
cfg(target_arch) block per architecture that returns early (runtime
feature probes as an if-chain inside), then the portable default as the
plain tail - no cfg(not(any(...))) negation blocks anywhere. An arm that
returns unconditionally (aarch64 NEON needs no probe) makes the tail
unreachable there, silenced by an allow(unreachable_code) scoped to just
the tail block. Kernel types are unsafe fn pointers so #[target_feature]
kernels coerce directly by name with a single narrow unsafe call per
site.
Tiny inputs are gated before the dispatch so they keep the old fully
inlined code path: count_ones_aligned calls the scalar kernel directly
below 32 bytes (where SIMD never engaged anyway), and scan_chunks calls
the per-architecture unconditional kernel directly for two chunks or
fewer. Without these gates the dispatch indirection dominates
sub-SIMD-threshold workloads: CodSpeed flagged true_count[128] (~-30%)
and rank_single[(1024, 0.1)]; cachegrind measured the rank regression at
+27 instructions per call and parity after the gates (37.70M vs 37.71M
baseline instructions for 100k rank calls).
Converted sites: fastlanes transpose_bits/untranspose_bits, vortex-buffer
scan_chunks/select_in_chunk/select_in_word/count_ones_aligned,
vortex-array filter_bitbuffer_by_mask, and vortex-mask
intersect_bit_buffers_dispatch/intersect_rank_indices_dispatch.
Intentionally left: primitive take (already one-time via LazyLock dyn
kernel), intersect_mask_driven_dispatch (generic over an iterator type,
so its instantiations cannot share one fn-pointer slot), and the
compile-time IS_CONST_LANE_WIDTH constant.
The count_ones any-length wrappers carry #[target_feature] so the SIMD
kernels inline into them, avoiding an extra call level, and the
vortex_bitbuffer and rank bench mains pre-resolve the kernel slots
(matching the existing CPUID pre-warm precedent) so no single-shot
benchmark measurement includes first-call selection.
benches/cpu_dispatch.rs compares dispatch overhead per call against the
same #[inline(never)] kernel. Fastest-of-3 per 1024 calls: direct 2.44us,
CpuKernel 2.44us (indistinguishable, confirmed by disassembly: load +
fused null test + indirect jmp), LazyLock<fn> 3.04us, and three cached
feature checks per call 4.54us. All touched crates also cargo check
clean on aarch64-unknown-linux-gnu.
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUFxXZLv4xxEqEYcoGWhfJclaude/vortex-aarch-pattern-abstraction-s3javb feat(vortex-buffer): arch-specialized SIMD bool packing for collect_bool
Replace the bit-at-a-time shift-OR loop behind collect_bool_word /
collect_bool_words with byte-to-bit packing kernels specialized per
architecture, without changing any public signature:
- x86-64 AVX-512BW: one vptestmb packs 64 bools into a u64
- x86-64 AVX2: two vpcmpeqb + vpmovmskb halves
- x86-64 SSE2 (baseline): four pcmpeqb + pmovmskb quarters
- aarch64 NEON (baseline): ushl by bit position + addp reduction tree
- other targets and Miri: branch-free SWAR multiply fallback
The API has two tiers. The default (collect_bool / collect_bool_words,
backed by collect_bool_words_inline) compiles the word loop once with
the widest kernel enabled at compile time (SSE2/NEON on stock targets;
AVX2/AVX-512 under -C target-cpu=native) so the predicate, the
[bool; 64] materialization, and the pack all inline into the caller.
The opt-in tier (collect_bool_multiversioned /
collect_bool_words_multiversioned) compiles the loop with the predicate
inside once per CPU feature level and selects a clone by runtime
feature detection; its docs state the caller's assertion — the
predicate must be trivially cheap, because the #[target_feature] call
boundary deoptimizes non-trivial predicates (an early all-multiversioned
draft regressed FSST fsst_prefix 13-63% this way; the pack kernel choice
itself moves that DFA-bound workload by at most ~4%).
Route the call sites whose predicates are bounds-check-free gathers or
comparisons through the multiversioned tier: From<&[bool]> / From<&[u8]>,
ByteBool boolean decoding, and the primitive and decimal between kernels
(now reading via get_unchecked, sound because the predicate only
receives indices 0..len — a contract now documented on every entry
point together with the bounds-check performance note).
Measured on AVX-512 hardware (divan, 64Ki bits, medians): the fused
word loop with a bool-gather predicate runs 14-24x faster than the old
scalar loop per SIMD level (SSE2 1.7us, AVX2 1.2us, AVX-512 1.1us vs
26.5us); BitBufferMut::from(&[bool]) improves ~43x; a between-shaped
predicate doubles under the multiversioned tier (5.5us vs 11.2us);
fsst_prefix is unchanged or slightly better than before the change.
Generated asm verified per kernel (the fused AVX-512 loop is vmovdqu64 +
vptestmb + kmovq per 64 bools; NEON is 14 instructions, its addp tree
additionally validated against a reference model on 10,004 random
inputs). New benches cover each kernel and both tiers; under CodSpeed
only the shipped entry points are tracked (from_bool_slice,
collect_bool_u32_gt).
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcsiAHXQc4e11yYcP3a3NXclaude/collect-bool-arch-optimizations-bo09ws feat(vortex-buffer): arch-specialized SIMD bool packing for collect_bool
Replace the bit-at-a-time shift-OR loop behind collect_bool_word /
collect_bool_words with byte-to-bit packing kernels specialized per
architecture, without changing any public signature:
- x86-64 AVX-512BW: one vptestmb packs 64 bools into a u64
- x86-64 AVX2: two vpcmpeqb + vpmovmskb halves
- x86-64 SSE2 (baseline): four pcmpeqb + pmovmskb quarters
- aarch64 NEON (baseline): ushl by bit position + addp reduction tree
- other targets and Miri: branch-free SWAR multiply fallback
The API has two tiers. The default (collect_bool / collect_bool_words,
backed by collect_bool_words_inline) compiles the word loop once with
the widest kernel enabled at compile time (SSE2/NEON on stock targets;
AVX2/AVX-512 under -C target-cpu=native) so the predicate, the
[bool; 64] materialization, and the pack all inline into the caller.
The opt-in tier (collect_bool_multiversioned /
collect_bool_words_multiversioned) compiles the loop with the predicate
inside once per CPU feature level and selects a clone by runtime
feature detection; its docs state the caller's assertion — the
predicate must be trivially cheap, because the #[target_feature] call
boundary deoptimizes non-trivial predicates (an early all-multiversioned
draft regressed FSST fsst_prefix 13-63% this way; the pack kernel choice
itself moves that DFA-bound workload by at most ~4%).
Route the call sites whose predicates are bounds-check-free gathers or
comparisons through the multiversioned tier: From<&[bool]> / From<&[u8]>,
ByteBool boolean decoding, and the primitive and decimal between kernels
(now reading via get_unchecked, sound because the predicate only
receives indices 0..len — a contract now documented on every entry
point together with the bounds-check performance note).
Measured on AVX-512 hardware (divan, 64Ki bits, medians): the fused
word loop with a bool-gather predicate runs 14-24x faster than the old
scalar loop per SIMD level (SSE2 1.7us, AVX2 1.2us, AVX-512 1.1us vs
26.5us); BitBufferMut::from(&[bool]) improves ~43x; a between-shaped
predicate doubles under the multiversioned tier (5.5us vs 11.2us);
fsst_prefix is unchanged or slightly better than before the change.
Generated asm verified per kernel (the fused AVX-512 loop is vmovdqu64 +
vptestmb + kmovq per 64 bools; NEON is 14 instructions, its addp tree
additionally validated against a reference model on 10,004 random
inputs). New benches cover each kernel and both tiers; under CodSpeed
only the shipped entry points are tracked (from_bool_slice,
collect_bool_u32_gt).
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcsiAHXQc4e11yYcP3a3NXclaude/collect-bool-arch-optimizations-bo09ws feat(vortex-buffer): arch-specialized SIMD bool packing for collect_bool
Replace the bit-at-a-time shift-OR loop behind collect_bool_word /
collect_bool_words with byte-to-bit packing kernels specialized per
architecture, without changing any public signature:
- x86-64 AVX-512BW: one vptestmb packs 64 bools into a u64
- x86-64 AVX2: two vpcmpeqb + vpmovmskb halves
- x86-64 SSE2 (baseline): four pcmpeqb + pmovmskb quarters
- aarch64 NEON (baseline): ushl by bit position + addp reduction tree
- other targets and Miri: branch-free SWAR multiply fallback
The API has two tiers. The default (collect_bool / collect_bool_words,
backed by collect_bool_words_inline) compiles the word loop once with
the widest kernel enabled at compile time (SSE2/NEON on stock targets;
AVX2/AVX-512 under -C target-cpu=native) so the predicate, the
[bool; 64] materialization, and the pack all inline into the caller.
The opt-in tier (collect_bool_multiversioned /
collect_bool_words_multiversioned) compiles the loop with the predicate
inside once per CPU feature level and selects a clone by runtime
feature detection; its docs state the caller's assertion — the
predicate must be trivially cheap, because the #[target_feature] call
boundary deoptimizes non-trivial predicates (an early all-multiversioned
draft regressed FSST fsst_prefix 13-63% this way; the pack kernel choice
itself moves that DFA-bound workload by at most ~4%).
Route the call sites whose predicates are bounds-check-free gathers or
comparisons through the multiversioned tier: From<&[bool]> / From<&[u8]>,
ByteBool boolean decoding, and the primitive and decimal between kernels
(now reading via get_unchecked, sound because the predicate only
receives indices 0..len — a contract now documented on every entry
point together with the bounds-check performance note).
Measured on AVX-512 hardware (divan, 64Ki bits, medians): the fused
word loop with a bool-gather predicate runs 14-24x faster than the old
scalar loop per SIMD level (SSE2 1.7us, AVX2 1.2us, AVX-512 1.1us vs
26.5us); BitBufferMut::from(&[bool]) improves ~43x; a between-shaped
predicate doubles under the multiversioned tier (5.5us vs 11.2us);
fsst_prefix is unchanged or slightly better than before the change.
Generated asm verified per kernel (the fused AVX-512 loop is vmovdqu64 +
vptestmb + kmovq per 64 bools; NEON is 14 instructions, its addp tree
additionally validated against a reference model on 10,004 random
inputs). New benches cover each kernel and both tiers; under CodSpeed
only the shipped entry points are tracked (from_bool_slice,
collect_bool_u32_gt).
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcsiAHXQc4e11yYcP3a3NXclaude/collect-bool-arch-optimizations-bo09ws Latest Branches
+17%
claude/collect-bool-arch-optimizations-bo09ws +14%
claude/vortex-aarch-pattern-abstraction-s3javb +10%
© 2026 CodSpeed Technology