Environment diffs in run comparisons
A benchmark can move without you touching a single line of code: a compiler upgrade, a new version of a system library, or a different CPU configuration can all shift the numbers. Until now, telling an environment-induced change apart from a real regression meant digging through CI logs. CodSpeed now collects detailed environment metadata with every run and diffs it whenever you compare two runs, so the likely cause is right there on the comparison page.
A benchmark comparison where only the Rust toolchain changed: Cargo and Rustc moved from 1.95.0 to 1.96.1, while LLVM, opt-level, and profile stayed put.
What you'll see
When the two environments differ, the comparison page shows a breakdown of exactly what changed:
- Toolchain versions: the language runtimes and compilers your integration uses (e.g. Python, Node.js, or Rust)
- Linked libraries: the system libraries your benchmarks were linked against, version by version
- CPU flags and hardware: differences in the underlying machine configuration
Anything identical on both sides is collapsed, so the diff only surfaces what actually changed between the base and head runs.
Available in the MCP server
The same environment diff ships in the
MCP server's compare_runs report,
so your AI assistant can factor environment changes into its analysis when
investigating a regression.
On by default
Environment metadata is collected automatically by recent runner versions, and the diff appears on the run comparison page whenever a difference is detected. Just open any run comparison and look for the environment section.