vortex-data/vortex

Write wide legacy decimal fixtures at the canonical width The checked decimal constructors reject storage wider than the precision requires, so the compat fixtures that specified such a width can no longer build their written form. Keep the fixtures so their published files stay covered by `check`, which reads the wide form back through legacy narrowing, and write those columns at the canonical width instead. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zQPnFWRi7CWXHpgBE8CiF

ji/decimal-storage-contract

5 hours ago

Bound develop's new decimal tests by precision Tests added on develop built decimal storage wider than their precision admits, which the checked constructors now reject. Give each the storage its precision requires, derive expected arrays in the result's own storage, and cap needle widths at the precision bound. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zQPnFWRi7CWXHpgBE8CiF

ji/decimal-storage-contract

5 hours ago

perf(scan): announce every chunk's reads when a V2 scan starts Each chunk of a V2 scan runs on a driver of its own, so a chunk's announcements only let its reads coalesce with its own segments. The chunk planner now announces the filter's and the projection's segments over every chunk still to plan on its first compute, and the prepared scan holds those announcements for its lifetime, so the IO service can widen each read with the segments around it. On TPC-H SF1 with DataFusion over local files this cuts physical reads from 2199 to 447 (v1: 882). Setting VORTEX_SCAN_EARLY_ANNOUNCE=0 restores per-chunk announcements, for comparisons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

ji/segment-io-announce

6 hours ago

Rank patches in one pass over the mask's words when filtering Filtering patches galloped through the patches for every selected row whenever patches outnumbered selected rows. A V2 conjunct filters its columns by the rows the previous conjunct kept, and TPC-DS store_sales holds about one patch in five rows against a few selected rows in a hundred: each gallop then crosses a handful of patches at the cost of a few unpredictable branches, and the gallop was V2's hottest function in q28. Gallop only when the patches outnumber the mask indices 32 to one. Otherwise rank each patch in the mask's words in one pass: whole words are counted as the patches pass them, and the bits below a patch within its own word. This replaces the per-patch count_range call of the bitmap path too. A new bitpacking_filter case, filter_patched, measures a patched array under such a selection: one row in 27 goes from 42.3 to 33.6 us; one row in 400, which still gallops, is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>

ji/scan-v2-clickbench-tpcds

6 hours ago

feat(scan): run the V2 executor on the segment IO protocol Switches the planner-driven scan from the byte-range `vortex_io::request` protocol to `vortex_io::segments`, and serves it with a session-wide `FileSegmentIo` that reads through files' v1 segment sources. - vortex-scan: `Planner` and `Morsel` drop the `IoConsumer` supertrait, publish `NeedsIO(Vec<IoRequest>)`, and receive `deliver(key, result)` once per successful fetch. `Run`/`Driver` submit each root's requests to its `SegmentIoSession` and hold the handles; each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches. Fetches in flight are counted per session, since spawned roots share one. `Run::poll_io` replaces `Run::complete`. The scripted harness is a recording `SegmentIoSession`. - vortex-layout: `ScanAdaptor` fetches each segment once by `IoKey::segment` while any scan read of it is outstanding, and hands the result to every waiting read. Prune and filter hints name segment keys. `ScanFile` and `PreparedScan` carry the file's `RegisteredSource`, and `PreparedScan::io` is its service. `SegmentScanIo` and the cached byte-range index it needed are removed. - vortex-file: `FileSegmentIo`, one per `VortexSession`. Size requests are answered at submit; segment requests call `SegmentSource::request`, so they share the file's segment cache, request sharing and coalescing read driver. `VortexFile::scan_file` registers the file's segment source, and a caching file keeps one registration in its scan cache, renewed with its segment source. The V2 scan driver reads through `SegmentIoSession`. - vortex-datafusion, vortex-duckdb: their V2 drivers read through `SegmentIoSession`; the DataFusion opener shares a file's segment source across partitions through `VortexFile::with_segment_source`. - vortex-io: `request` is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

ji/v2-scan-segment-io

6 hours ago

feat(scan): run the V2 executor on the segment IO protocol Switches the planner-driven scan from the byte-range `vortex_io::request` protocol to `vortex_io::segments`, and serves it with a session-wide `FileSegmentIo` that reads through files' v1 segment sources. - vortex-scan: `Planner` and `Morsel` drop the `IoConsumer` supertrait, publish `NeedsIO(Vec<IoRequest>)`, and receive `deliver(key, result)` once per successful fetch. `Run`/`Driver` submit each root's requests to its `SegmentIoSession` and hold the handles; each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches. Fetches in flight are counted per session, since spawned roots share one. `Run::poll_io` replaces `Run::complete`. The scripted harness is a recording `SegmentIoSession`. - vortex-layout: `ScanAdaptor` fetches each segment once by `IoKey::segment` while any scan read of it is outstanding, and hands the result to every waiting read. Prune and filter hints name segment keys. `ScanFile` and `PreparedScan` carry the file's `RegisteredSource`, and `PreparedScan::io` is its service. `SegmentScanIo` and the cached byte-range index it needed are removed. - vortex-file: `FileSegmentIo`, one per `VortexSession`. Size requests are answered at submit; segment requests call `SegmentSource::request`, so they share the file's segment cache, request sharing and coalescing read driver. `VortexFile::scan_file` registers the file's segment source, and a caching file keeps one registration in its scan cache, renewed with its segment source. The V2 scan driver reads through `SegmentIoSession`. - vortex-datafusion, vortex-duckdb: their V2 drivers read through `SegmentIoSession`; the DataFusion opener shares a file's segment source across partitions through `VortexFile::with_segment_source`. - vortex-io: `request` is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

ji/segment-io

6 hours ago

feat(io): a segment-keyed IO protocol for planner-driven scans Adds `vortex_io::segments`, keyed by segment source and source request, next to the byte-range `vortex_io::request` protocol that the planner-driven scan still uses. - A request names `IoKey { source: SegmentSourceId, request: SourceRequest }`, where `SourceRequest ::= Size | Segment(id)`, with an `IoIntent`: `Fetch`, `Prefetch` or `Announce`. One service serves many segment sources, and reads resolve to a `SourceResult` (`Size(u64)` or `Bytes(BufferHandle)`). - `SegmentIoSession::submit` registers a whole batch before any read starts and returns one `IoHandle` per request. The session holds fetches and prefetches until they finish; an announcement is withdrawn by dropping its handles. - `poll_ready` reports every fetch's owner once it finishes and drives the session's prefetches, through the shared `FetchTracker` helper. `SegmentIoService::session` opens one session per scope, and `RegisteredSource` is a source's registration with a service. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

ji/segment-io-protocol

6 hours ago

Retain a sparse V2 fetch's reads and plan only its selected chunks A row-id fetch, such as DuckDB's late materialization in ClickBench q23, selects a few rows of each file, but each wide column segment it touches is needed by every chunk that selects a row of it. The chunks run one after another, so sharing a read only while another session holds it read those segments again: 3905 requests for 2312 segments, 849 MB read for 321 MB of segments. A scan whose selection keeps at most one row in 1024 now retains its reads for its lifetime, as V1's layout readers do. The DuckDB driver also planned every chunk of such a file as a part, most of them empty, and offered DuckDB a thread for each. It now keeps only the chunks with a selected row, each its own part. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8NsCJMx4C45Wf2Df24qzZ

ji/loving-edison-936mz7

6 hours ago

Performance History

Latest Results

Latest Branches