Skip to main content
Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard GitHub-hosted runners can exhibit significant performance variance.
Read our detailed post on how CI noise affects benchmark consistency.
CodSpeed instruments are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitHub Actions is to use the CodSpeed GitHub Action.

​
Setup

​
1. Create the benchmarks workflow

Create a new workflow to run the benchmarks for your repository. You can do this by creating the codspeed.yml file in the .github/workflows directory with the following content:
.github/workflows/codspeed.yml
name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    permissions: # optional for public repositories
      contents: read
      id-token: write # for OpenID Connect authentication with CodSpeed
    steps:
      - uses: actions/checkout@v5
      # ...
      # Setup your environment here:
      #  - Configure your Python/Rust/Node version
      #  - Install your dependencies
      #  - Build your benchmarks (if using a compiled language)
      # ...
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v4
        with:
          mode: instrumentation
          run: <Insert your benchmark command here>
The most important step of this workflow is the usage of CodSpeedHQ/action. This action will configure the CodSpeed environment and upload the benchmarks results.
Make sure to include pull_request in the on section of the workflow. This is required to have reports on pull requests correctly working.Learn more about baseline report selection.
Public repositoriesCodSpeed allows tokenless uploads for public repositories, allowing runs to be triggered from public forks directly.To enable this, you can simply omit the permissions section.Learn more about authentication methods.

​
Sample configurations

​
2. Check the results

Once the workflow is created, your pull requests will receive a performance report comment and will also receive some additional checks: Pull Request Result Pull Request Result

​
3. Next Steps

Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed.

Explore the Performance Metrics

Understand the performance metrics generated by CodSpeed

Enforce Performance Checks

Make sure you or team members never merge unexpected performance regressions

Dive in Trace Generation

Get detailed performance traces for your benchmarks

Shard the execution of your benchmarks

Run your benchmarks in parallel to speed up your CI

​
Authentication

CodSpeed needs to validate the authenticity of the benchmark results being uploaded. In order to upload benchmark results to CodSpeed, the GitHub Actions workflow needs to authenticate with CodSpeed. There are three supported methods for authentication:
  • OpenID Connect (OIDC): recommended way for both private and public repositories
  • Static CodSpeed tokens
  • Tokenless uploads (public repositories only)

​
OpenID Connect (OIDC) Authentication

CodSpeed recommends using OpenID Connect (OIDC) for authentication. Using this method, a token is generated on-the-fly during the workflow run. This token is then used to authenticate securely with CodSpeed without needing to store long-lived credentials, but grants no additional permissions to the workflow. To enable OIDC, add the following section to your workflow definition: 
permissions:
  contents: read # required to checkout the repository
  id-token: write # required to authenticate with CodSpeed
For more information, see the GitHub Actions OIDC documentation:

​
CodSpeed token (Classic)

While we recommend using OpenID Connect (OIDC) for improved security, you can use a static CodSpeed token for authentication. Retrieve your CodSpeed token from your repository settings page: Upload Token from the settings page
Token Scope: Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token.
Then, create a new encrypted secret in your repository with the name CODSPEED_TOKEN and the value of your token. Then pass the token explicitly to the action using the with key: 
- name: Run benchmarks
  uses: CodSpeedHQ/action@v4
  with:
    mode: instrumentation
    token: ${{ secrets.CODSPEED_TOKEN }}

​
Tokenless upload (public repositories only)

CodSpeed allows tokenless uploads for public repositories, allowing runs to be triggered from public forks directly. However, we still recommend using OIDC for improved security. To use tokenless uploads on public repositories, you can simply omit the token input and the whole permissions section when creating the benchmarks workflow.

​
Advanced

​
Defining environment variables

You can define environment variables for your benchmarks by using the env key in the section of the action: 
- name: Run benchmarks
  uses: CodSpeedHQ/action@v4
  env:
    MY_ENV_VAR: "my-value"
  with:
    mode: instrumentation

​
Running benchmarks in parallel CI jobs

With Github Actions, you can leverage matrix jobs to improve the performance of your benchmarks. For example, using pytest:
.github/workflows/codspeed.yml
name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks (Shard #\${{ matrix.shard }})
    runs-on: ubuntu-latest
    permissions: # optional for public repositories
      contents: read
      id-token: write # for OpenID Connect authentication with CodSpeed
    steps:
      - uses: actions/checkout@v5
      # ...
      # Setup your environment here:
      #  - Configure your Python/Rust/Node version
      #  - Install your dependencies
      #  - Build your benchmarks (if using a compiled language)
      # ...
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v4
        with:
          mode: instrumentation
          run: <Insert your benchmark command here>
CodSpeed will only be able to aggregate results from your benchmarks if you split them within a single CI workflow.
CodSpeed does not yet support running the same benchmarks multiple times. If your matrix has several dimensions (e.g. a runtime version), please ensure that each benchmark will only run once.If the same benchmark is run multiple times, you will receive the following comment on your pull request:Multiple Benchmark Variations Error Message
Learn more about benchmark sharding and how to integrate with your CI provider.

​
Using container images

When running benchmarks in a container (such as ubuntu:latest or other base images), git is typically not installed by default. Without git in the path, actions/checkout falls back to downloading a tarball of your code, which does not setup a local git repository. CodSpeed relies on git to determine the current commit hash and other metadata, so it’s important to have git installed and the repository checked out. 
jobs:
  benchmarks-in-container:
    runs-on: ubuntu-latest
    container:
      image: ubuntu:latest
    steps:
      - name: Install git before fetching repository
        run: |
          apt-get update
          apt-get install -y git
          git config --global --add safe.directory $GITHUB_WORKSPACE
      - name: Checkout code
        uses: actions/checkout@v4
The git config --global --add safe.directory command is required to prevent git from triggering file system permission errors when accessing the repository. More details can be found in this GitHub Issue

​
Compatibility

For now, only the following OS and versions are supported on the runners:
  • Ubuntu 22.04 and later
  • Debian 12 and later