Latest Results
Move protobuf and flatbuffer definitions into their owning crates
## Rationale for this change
Closes #1906. The `.fbs` and `.proto` schemas all lived in
`vortex-flatbuffers` and `vortex-proto`, away from the types they
describe, with their generated Rust checked in.
Build-time generation existed before #557 but was removed because
`vortex-build` discovered include paths by walking workspace metadata
with `cargo_metadata` and reaching outside the package directory, neither
of which survives packaging, so publishing broke. Declaring the
dependencies explicitly avoids that.
## What changes are included in this PR?
Each schema moves to the crate that owns the types it describes, and is
compiled into `OUT_DIR` by that crate's `build.rs`. Nothing generated is
checked in:
| Schema | Crate | Module |
| -------------------------- | --------------- | ---------------------------- |
| `array.fbs`, `dtype.fbs` | `vortex-array` | `vortex_array::flatbuffers` |
| `dtype/scalar/expr.proto` | `vortex-array` | `vortex_array::proto` |
| `layout.fbs` | `vortex-layout` | `vortex_layout::flatbuffers` |
| `footer.fbs` | `vortex-file` | `vortex_file::flatbuffers` |
| `message.fbs` | `vortex-ipc` | `vortex_ipc::flatbuffers` |
`vortex-proto` and `vortex-flatbuffers` are both removed. The FlatBuffers
read/write traits move into `vortex_array::flatbuffers` alongside the
generated array and dtype bindings, which every crate that used them
already depended on.
The new `vortex-build` crate holds the shared build script helpers, and
`xtask` is deleted since code generation was its only job. A crate whose
schemas include another crate's names it explicitly:
vortex_build::flatbuffers()
.depends_on("vortex-array")
.compile(&["vortex-serde/message.fbs"]);
`depends_on` resolves the dependency's schema directory through Cargo's
`links` metadata, so a path dependency in the workspace and a package
unpacked from a registry behave identically. `flatc`'s `--include-prefix`
points cross-crate includes at a small `deps` module each consuming crate
provides by hand, so no schema is compiled twice.
The issue suggests generating into `src/flatbuffers/` and git-ignoring
it. Generating into `OUT_DIR` instead keeps the same property ā no
generated code in git ā without a build script writing into its own
package, which would invalidate the registry checksum for anyone
building a published crate.
`.proto` compilation uses `protox` rather than `protoc`, so protobuf
codegen needs no external tooling; the generated output is byte-identical
to what was checked in. `.fbs` compilation shells out to `flatc`, which
must be on `PATH` or named by `FLATC`, so CI installs it as part of the
shared Rust setup and the musl job pulls it from Alpine.
The flatbuffer back-compat check now flattens each revision's per-crate
schema directories into one tree before running `flatc --conform`.
## What APIs are changed? Are there any user-facing changes?
`vortex_proto::{dtype, scalar, expr}` becomes `vortex_array::proto::*`,
and both the flatbuffer traits and the generated modules move from
`vortex_flatbuffers::*` to `vortex_array::flatbuffers` and the crates
listed above. The `vortex` facade keeps `vortex::proto` and
`vortex::flatbuffers` pointing at the same items, and `vortex_array::
dtype`'s `proto` and `flatbuffers` re-exports are unchanged.
Building any Vortex crate from source now requires `flatc`, including for
downstream consumers and docs.rs.
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017zTmZ6ANxESomTnvkvk85tclaude/protobuf-codegen-build-rs-s897y8 Latest Branches
0%
claude/protobuf-codegen-build-rs-s897y8 +12%
ct/row-fn-associated-output -7%
ct/row-fn-binary-numeric-comparison Ā© 2026 CodSpeed Technology