Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance.
Read our detailed post on how CI noise affects benchmark consistency.
CodSpeed instruments are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitLab CI is to use the CodSpeed Runner directly.

​
Setup

For now, only the following OS and versions are supported on the runners:
  • Ubuntu 22.04 and later
  • Debian 12 and later

​
1. Allow the Gitlab CI workflow to upload on CodSpeed

You need to get your CodSpeed Token. There are multiple ways to retrieve it:
  • Once you enable a repository on CodSpeed, you’ll be prompted to copy the token
  • You can also find it on the repository settings page
Upload Token from the settings page
Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token.
Then, create a new CI/CD variable in your repository with the name CODSPEED_TOKEN and the value of your token.
Set the CODSPEED_TOKEN variable as "Masked", but not "Protected", because it must be available on all branches.Upload Token from the settings page

​
2. Create the benchmarks job

The next step is to create a new job to run the benchmarks for your repository. For example, you can add this job to your existing pipeline by adding a section in your .gitlab-ci.yml file with the following content:
.gitlab-ci.yml
workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

codspeed:
  stage: test
  image: <your-docker-image> # for example: python:3.12 or rust:1.82
  before_script:
    # ...
    # Setup your environment here:
    # - Configure your Python/Rust/Node version
    # - Install your dependencies
    # - Build your benchmarks (if using a compiled language)
    # ...

    # Use the latest CodSpeed version
    - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh | bash -s -- --quiet
    # Or set a specific version. Refer to https://github.com/CodSpeedHQ/runner/releases for available versions. For example:
    # - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/download/v3.8.1/codspeed-runner-installer.sh | bash -s -- --quiet
  script:
    - codspeed run -- "<Insert your benchmark command here>"
Make sure to run the job on merge_request_event. This is required to have reports on merge requests correctly working.Learn more about baseline report selection.

​
Sample configurations

.gitlab-ci.yml
workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
codspeed:
  stage: test
  image: python:3.12
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh  | bash -s -- --quiet
    - source $HOME/.cargo/env
  script:
    - codspeed run -- pytest tests/ --codspeed
More info on how to setup Python benchmarks.
CodSpeed only currently supports docker images that are based on Ubuntu or Debian.

​
3. Check the results

Once the workflow is created, your merge requests will receive a performance report comment and will also receive some additional checks:
Merge Request Setup Comment

Merge Request Setup Comment with detected benchmarks

Merge Request Result

Merge Request Status Check with performance results

​
4. Next Steps

Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed.

Explore the Performance Metrics

Understand the performance metrics generated by CodSpeed

Enforce Performance Checks

Make sure you or team members never merge unexpected performance regressions

Dive in Trace Generation

Get detailed performance traces for your benchmarks

Shard the execution of your benchmarks

Run your benchmarks in parallel to speed up your CI

​
Advanced

​
Running benchmarks in parallel CI jobs

With GitlabCI, you can split your benchmarks across multiple jobs. For example with pytest:
.gitlab-ci.yml
workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

codspeed-1:
  stage: test
  image: python:3.12
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL
      https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh  |
      bash -s -- --quiet
    - source $HOME/.cargo/env
  script:
    - codspeed run -- pytest tests/ --codspeed --test-group=1 --test-group-count=2

codspeed-2:
  stage: test
  image: python:3.12
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL
      https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh  |
      bash -s -- --quiet
    - source $HOME/.cargo/env
  script:
    - codspeed run -- pytest tests/ --codspeed --test-group=2 --test-group-count=2
CodSpeed will only be able to aggregate results from your benchmarks if you split them within a single CI workflow.
Learn more about benchmark sharding and how to integrate with your CI provider.