Skip to main content
Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance.
Read our detailed post on how CI noise affects benchmark consistency.
CodSpeed instruments are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitLab CI is to use the CodSpeed Runner directly.
For now, only the following OS and versions are supported on the runners:
  • Ubuntu 22.04 and later
  • Debian 12 and later

​
1. Create the benchmarks job

Create a new job to run the benchmarks for your repository. For example, you can add this job to your existing pipeline by adding a section in your .gitlab-ci.yml file with the following content:
.gitlab-ci.yml
workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

codspeed:
  stage: test
  image: <your-docker-image> # for example: python:3.12 or rust:1.82
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    # ...
    # Setup your environment here:
    # - Configure your Python/Rust/Node version
    # - Install your dependencies
    # - Build your benchmarks (if using a compiled language)
    # ...

    # Use the latest CodSpeed version
    - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh | bash -s -- --quiet
    # Or set a specific version. Refer to https://github.com/CodSpeedHQ/runner/releases for available versions. For example:
    # - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/download/v4.2.1/codspeed-runner-installer.sh | bash -s -- --quiet
  script:
    - codspeed run --mode instrumentation -- "<Insert your benchmark command here>"
Make sure to run the job on merge_request_event. This is required to have reports on merge requests correctly working.Learn more about baseline report selection.

​
Sample configurations

  • Python
  • Rust
  • Node.js
.gitlab-ci.yml
workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
codspeed:
  stage: test
  image: python:3.12
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL https://github.com/CodSpeedHQ/runner/releases/latest/download/codspeed-runner-installer.sh  | bash -s -- --quiet
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode instrumentation -- pytest tests/ --codspeed
More info on how to setup Python benchmarks.
CodSpeed only currently supports docker images that are based on Ubuntu or Debian.

​
2. Check the results

Once the workflow is created, your merge requests will receive a performance report comment and will also receive some additional checks:
Merge Request Setup Comment

Merge Request Setup Comment with detected benchmarks

Merge Request Result

Merge Request Status Check with performance results

​
3. Next Steps

Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed.

Configure GitLab CI for CodSpeed

Learn how to configure authentication methods and advanced options for GitLab CI

Explore the Performance Metrics

Understand the performance metrics generated by CodSpeed

Enforce Performance Checks

Make sure you or team members never merge unexpected performance regressions

Dive in Trace Generation

Get detailed performance traces for your benchmarks

Shard the execution of your benchmarks

Run your benchmarks in parallel to speed up your CI