Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance.
Read our detailed post on how CI noise affects benchmark consistency.
CodSpeed instruments are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in Buildkite is to use the CodSpeed Runner directly.

​
Setup

For now, only the following OS and versions are supported on the agents:
  • Ubuntu 22.04 and later
  • Debian 12 and later

​
1. Setup the CODSPEED_TOKEN secret

First, you need to get your CodSpeed Token. There are multiple ways to retrieve it:
  • Once you enable a repository on CodSpeed, you’ll be prompted to copy the token
  • You can also find it on the repository settings page
Upload Token from the settings page
Token Scope: Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token.
Then, create a new pipeline secret with your preferred method with the name CODSPEED_TOKEN and the value of your token.

​
2. Install the codspeed-runner CLI in your agent

Install the CLI on your agent by running the following commands: 
CODSPEED_RUNNER_VERSION=<insert-version> # refer to https://github.com/CodSpeedHQ/runner/releases for available versions
curl -fsSL https://github.com/CodSpeedHQ/runner/releases/download/$CODSPEED_RUNNER_VERSION/codspeed-runner-installer.sh | bash
source "$HOME/.cargo/env"
It is important to source the $HOME/.cargo/env file to make sure that the codspeed-runner command will be available in the pipeline.
Refer to the releases page to see all available versions.

​
3. Create the benchmarks pipeline

In a new or existing pipeline, add a step that will run your benchmarks with codspeed-runner:
.buildkite/pipeline.yml
steps:
  - label: "Run benchmarks with CodSpeed"
    command: |
      ... # your build commands
      codspeed-runner --token=$CODSPEED_TOKEN -- <Insert your benchmark command here>

​
3. Check the results

Once the workflow is created, your pull requests will receive a performance report comment and will also receive some additional checks: Pull Request Result Pull Request Result

​
Next Steps

Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed.

Explore the Performance Metrics

Understand the performance metrics generated by CodSpeed

Enforce Performance Checks

Make sure you or team members never merge unexpected performance regressions

Dive in Trace Generation

Get detailed performance traces for your benchmarks

Shard the execution of your benchmarks

Run your benchmarks in parallel to speed up your CI