vortex-data/vortex

vortex

Performance History

Latest Results

Key the metadata footer cache by the readable's name `NativeReadable.name()` is already required to be stable and unique, and is what the scan path keys Java readables by, so a metadata read has an identity to cache under too. Pass it down and drop the "no key, no caching" path from `multi::open_cached`: every caller now supplies a fallback key, used when the source itself reports no URI. The name is normalized the same way `openFiles` normalizes it before registering readables, so a metadata read and a scan of the same readable share one cache entry rather than landing on two. Also pass the readable's known length, so a read that resolves against a cached footer validates it against the file size instead of trusting it blindly. Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wt7vVoiJrjxHyDvy4XDjsW

rk/jnimetadata

6 minutes ago

Address review: writer errors, metadata validation tests, footer cache Follow-ups from review of the metadata JNI bindings: - `VortexWriter.Builder.build()` declared `throws IOException`, but every real failure arrived as the `RuntimeException` thrown across the JNI boundary, so a rejected metadata set escaped uncaught by `catch (IOException)`. Wrap native failures the way `writeBatch` and `finish` already do. This also routes the Spark writer's failures into its existing `IOException` handler, which logs the target path. - Cover `VortexWriteOptions::validate_metadata` in Rust: the empty-key and segment-count branches had no test in either language, and only the oversized-key branch was reachable from the Java suite. - Read metadata through the session footer cache. `readMetadata` opened the file with a bare `open_options()`, so reading metadata and then scanning the same file paid two footer reads. Factor the cached open out of `multi::open_file` into `multi::open_cached` and use it from both. Sources without a URI (the `NativeReadable` bridge) skip the cache, since they have no stable identity. - Release the per-entry local references in `extract_metadata`, matching `metadata_to_java` directly below it. Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wt7vVoiJrjxHyDvy4XDjsW

rk/jnimetadata

29 minutes ago

test(array): reword a comment the typo checker rejects `typos` splits `mis-sliced` on the hyphen and flags `mis`. Signed-off-by: Claude <noreply@anthropic.com>

claude/builders-canonical-children-9ze0t6

6 hours ago

refactor(array): canonicalize chunked structs and FSLs through the builder `pack_struct_chunks` and `swizzle_fixed_size_list_chunks` existed because `append_to_builder` used to decode a builder's children: without them, canonicalizing a `ChunkedArray` would concatenate every chunk's children instead of reusing them. Their doc comments describe what the builders now do on their own, so both are dead weight. Route both dtypes through the generic builder path, telling it to keep every chunk boundary. `ListArray` keeps its swizzle for now — that one also allocates its offsets and sizes through the session allocator, which the builders cannot yet do. Signed-off-by: Claude <noreply@anthropic.com>

claude/chunked-canonical-via-builder-9ze0t6

6 hours ago

perf(array): accumulate nested builder validity without a null buffer A nested builder learns about validity from two sources: one row at a time as scalars are appended, and a whole array's worth at a time as arrays are. Only the first needs a null buffer, but `LazyBitBufferBuilder` treated both the same, so every appended array had its validity executed into a `Mask` and its bits copied. `ValidityBuilder` keeps a whole array's validity as a run and concatenates the runs at the end, the way `Validity::concat` already does for `StructArray::try_concat`. `AllValid` and `AllInvalid` runs cost nothing, array-backed runs are bool arrays that are already built, and a builder that only ever saw uniform validity still answers from its nullability rather than producing a bool array. Runs shorter than the chunk threshold are still copied into the null buffer, so a builder fed one row at a time does not accumulate a run per row. `StructBuilder`, `ListBuilder`, `ListViewBuilder` and `FixedSizeListBuilder` use it; the leaf builders keep `LazyBitBufferBuilder`. Signed-off-by: Claude <noreply@anthropic.com>

claude/builders-lazy-validity-9ze0t6

6 hours ago

feat(array): let callers choose a nested builder's chunk threshold `ChildBuilder` copies appended arrays shorter than 64 values instead of giving them a chunk, which is right for the nested builders that append the elements of a single list, and wrong for a caller whose appended arrays are themselves chunks worth preserving. Add `ArrayBuilder::set_min_chunk_len`, a no-op for builders without array children, so such a caller can lower the threshold — to zero to keep every chunk boundary. The threshold is read when an array is appended and applies transitively to the children of a builder's children. Signed-off-by: Claude <noreply@anthropic.com>

claude/builders-min-chunk-len-9ze0t6

6 hours ago

test(array): reword a comment the typo checker rejects `typos` splits `mis-sliced` on the hyphen and flags `mis`. Signed-off-by: "Claude" <noreply@anthropic.com>

claude/builders-canonical-children-9ze0t6

6 hours ago

Guarantee 8-byte alignment in vx_alloc; cast instead of copy in bitpacked The bitpacked kernel copied packed words out per chunk, defending against alignment and endianness — neither of which actually applies in the guest: wasm32 is always little-endian (matching the serialized format), and every buffer reaches guest memory through vx_alloc, which we control. - vx_alloc now allocates with an explicit 8-byte-aligned Layout, and the alignment is documented as part of the ABI: host uploads (frames, raw buffers, child structs) always land aligned, so kernels can view typed data in place and Arrow int64 struct fields are naturally aligned. - The bitpacked kernel views the packed buffer as &[u32] via a checked align_to (no word copies), unpacks full in-range chunks directly into the output — mirroring the native decode_into fast path — and uses scratch only for a sliced first chunk and a partial trailer. Patches are applied in u32 space; the output converts to bytes with a single cast + copy. Also validated the no_std-by-default path the kernels should end on: with fastlanes-rs's num-traits edge patched to default-features = false (a one-line upstream fix), the identical kernel source builds fully no_std with the SDK runtime enabled at 37 KB instead of 53 KB, and passes the whole round-trip suite (this also exercises the SDK bump allocator through the real pipeline). fsst-rs needs a mechanical no_std-ification for the same win. Documented in the design doc with measured sizes. All 11 tests pass in the detached workspace; clippy clean. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TaUprJCAbZjuvv6S5MVt4s

claude/relaxed-johnson-8sll2h

6 hours ago

Latest Branches

+20%

Expose metadata reading and writing in vortex-jni#8963

7 minutes ago

b1b05ab

rk/jnimetadata

+32%

feat(array): builders no longer canonicalize their children [builders-child-stack]#8964

6 hours ago

b4cb8fb

claude/builders-canonical-children-9ze0t6

-31%

refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]#8967

8 hours ago

afed52f

claude/chunked-canonical-via-builder-9ze0t6

Home Terms Privacy Docs