Latest Results
feat(bench-orchestrator): declarative benchmark matrix with profiles
Introduce a single declarative registry of SQL benchmarks plus composable
"profiles" that select which benchmarks run and how many targets each runs, so
CI matrices are derived from one source of truth instead of hand-maintained as
parallel pr_targets/develop_targets arrays and duplicated full/base matrices in
the workflow YAML.
- matrix.py: BenchmarkDef registry, TargetSet algebra (a shared STANDARD core
with opt-in `| df(...)` / opt-out `.only(...)` deltas), Predicate selectors,
target policies (all_targets / defaults), and a resolver that emits the
GitHub Actions `include:` array. data_formats is derived from the resolved
targets, and storage-invalid targets (remote lance) are dropped using the
existing support rules, so neither can drift from what actually runs.
- vx-bench matrix <profile>: emit a profile's matrix as JSON for the dynamic
matrix workflow pattern, or list the available profiles.
- Profiles: develop, pr, and nightly, plus sql-nvme, sql-s3, and tpch examples
that show how a custom benchmark action becomes one declarative entry.
- Tests: unit coverage for the algebra, selectors, and policies; a golden file
per profile; and parity tests proving the develop and nightly profiles
reproduce today's workflow matrices exactly (behavior-preserving). The pr
profile intentionally regularizes coverage -- it adds the arbitrarily-omitted
appian and tpch-s3-10 benchmarks and drops tpch's PR-only arrow lane -- which
addresses the consistency concerns in #4935.
- README: document the model and how a workflow consumes the matrix.
This is the source-of-truth module; wiring the workflows to consume
`vx-bench matrix` is a deliberate follow-up. No workflow YAML changes yet.
Signed-off-by: Connor Tsui <connor@spiraldb.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQg1cgPLKJg4QVJvE2xa3Q Latest Branches
0%
0%
-5%
© 2026 CodSpeed Technology