CodSpeed CLI

The CodSpeed CLI allows you to benchmark any executable program and collect performance data. It is open source, and its source code is available on GitHub.

Getting Started

Installation

Install the CodSpeed CLI using the installation script:

curl -fsSL https://codspeed.io/install.sh | sh

After installation, authenticate with your CodSpeed account by running codspeed auth login.

Installing a specific version

To install a specific version:

curl -fsSL https://codspeed.io/v4.8.2/install.sh | sh

Replace v4.8.2 with the version you want to install.You can find all available versions on the releases page.

The CodSpeed CLI officially supports Ubuntu 20.04, 22.04, 24.04 and Debian 11, 12. Other Linux distributions can be used as long as the required tools for your chosen instruments are manually installed and available in your PATH.

Run your first command

Let’s run a simple benchmark using the walltime instrument:

terminal

$ codspeed exec -- sleep 1

►►► Running the benchmarks
Executing: sleep 1
Completed 2 warmup rounds
Warmup done, now performing 3 rounds

►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

sleep 1: 1.01 s

To see the full report, visit: https://codspeed.io/[...]

Configuration

You can define benchmarks and options in a codspeed.yml configuration file. Most importantly, items in the benchmarks list describe the commands that are run when using codspeed run.

Other supported filenames

The CodSpeed CLI looks for configuration files with the following names, by order of priority:

codspeed.yml
codspeed.yaml
.codspeed.yml
.codspeed.yaml

You can also specify a custom path with --config <path>.

Benchmark fields

exec

string

required

The command to execute. Can include arguments and flags.

name

string

A descriptive name for the benchmark. This is used to identify the benchmark in reports and results.When omitted, the command is used as the benchmark name.

options

object

Override global instrument options for this specific benchmark.See the Instruments section for available options for each instrument and their effects.

Running benchmarks

To run all benchmarks defined in your configuration file, specify the instrument with the -m flag:

terminal

$ codspeed run -m walltime

►►► Running the benchmarks

Executing: <commmand 1>
Completed 13 warmup rounds
Warmup done, now performing 20 rounds

Executing: <command 2>
A single warmup execution (517.ms) exceeded or met max_time (200ms).

Executing: <command 3>
Completed 23 warmup rounds
Warmup done, now performing 68 rounds

►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

┌──────────────┬─────────────┐
│ Benchmark    │ Measurement │
├──────────────┼─────────────┤
│ <command 1>  │ 517.23 ms   │
├──────────────┼─────────────┤
│ <command 2>  │ 14.10 ms    │
├──────────────┼─────────────┤
│ <command 3>  │ 6.91 ms     │
└──────────────┴─────────────┘

To see the full report, visit: https://codspeed.io/[...]

You can also use codspeed exec to run a single command:

terminal

$ codspeed exec -m walltime --warmup-time 1ms --max-time 2s -- ./my-binary --arg1 value --arg2

►►► Running the benchmarks
Executing: ./my-binary --arg1 value --arg2

Completed 1 warmup rounds
Warmup done, now performing 115 rounds


►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

./my-binary --arg1 value --arg2: 14.01 ms

To see the full report, visit: https://codspeed.io/[...]

Example configuration

codspeed.yml

$schema: https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json

# Global options applied to all benchmarks
options:
  warmup-time: "0.2s"
  max-time: 1s

# List of benchmarks to run
benchmarks:
  - name: "<command 1>"
    exec: ./my_binary --arg1 value --arg2
    # Override global options for this benchmark
    options:
      max-rounds: 20

  - name: "<command 2>"
    exec: ./my_binary2
    options:
      max-time: 200ms

  - name: "<command 3>"
    exec: ./my_binary3 --flag
    options:
      min-time: 200ms

As displayed in the above example, we provide a schema for the configuration file here.If your editor supports it, you can use the $schema field to enable autocompletion and validation for the codspeed.yml file.Please note that by default, the VSCode yaml language server does not support declaring the schema through the $schema field. One of the ways to declare the schema as is to add

codspeed.yaml

# yaml-language-server: $schema=https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json

More information in the yaml-language-server documentation.

Instruments

CodSpeed supports three instruments for measuring different aspects of your program’s performance.

Walltime

The walltime instrument measures real-world execution time. It runs your benchmark multiple times and collects timing data. See the Walltime documentation for more details.

How it works

Warmup phase: The benchmark runs repeatedly until the warmup time is reached. This allows the system to reach a steady state.
Measurement phase: Based on the average time from warmup, the CodSpeed CLI calculates how many rounds to run within the configured time bounds, then collects timing data for each round.

The walltime instrument requires sudo privileges to run because it needs to configure the kernel so that linux perf can collect all the necessary events.We plan to fine tune permissions in the future to reduce permissions to the strict minimum.

Profiling for interpreted languages (e.g., Python, JavaScript) is not currently supported with the walltime instrument.We are currently working on it and it’s coming soon!

Walltime Options

warmup_time

duration

default:"1s"

Time spent in warmup runs. Set to 0 to disable warmup.

min_time

duration

Minimum time spent running measurement rounds.

max_time

duration

default:"3s"

Maximum time spent running measurement rounds.

min_rounds

integer

Minimum number of measurement rounds.

max_rounds

integer

Maximum number of measurement rounds.

If there is a conflict between time and round constraints, the CodSpeed CLI prioritizes satisfying the maximum bounds first.

Example

CLI
Configuration file

terminal

$ codspeed exec -m walltime --warmup-time 100ms --max-time 1s -- <command>

►►► Running the benchmarks
Executing: <command>

Completed 7 warmup rounds
Warmup done, now performing 63 rounds


►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

<command>: 14.15 ms

To see the full report, visit:
https://codspeed.io/[...]

codspeed.yml

benchmarks:
  - name: My command
    exec: <command>
    options:
      warmup_time: 100ms
      max_time: 1s

terminal

$ codspeed run

►►► Running the benchmarks
Executing: My command

Completed 7 warmup rounds
Warmup done, now performing 63 rounds



►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

<command>: 14.15 ms

To see the full report, visit:
https://codspeed.io/[...]

Memory

The memory instrument tracks heap allocations during benchmark execution. Unlike walltime, the benchmark runs only once while memory usage is recorded. See the Memory documentation for more details. It measures:

Peak memory usage
Total allocated memory
Allocation count
Average allocation size

See Supported Allocators for the list of allocators that can be tracked.

The memory instrument requires sudo privileges to run because it uses eBPF to track memory allocations.We plan to fine tune permissions in the future to reduce permissions to the strict minimum.

Example

terminal

$ codspeed exec -m memory -- <command>

►►► Running the benchmarks
Executing: <command>


►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

<command>: 7.8 MB

To see the full report, visit: https://codspeed.io/[...]

Simulation

The simulation instrument uses CPU simulation to measure performance. The benchmark runs only once and CPU behavior is simulated, providing consistent measurements independent of system load. See the CPU Simulation documentation for more details. This is the same instrument used by CodSpeed integrations when running in CI.

Profiling for interpreted languages (e.g., Python, JavaScript) is not currently supported with the simulation instrument.We are currently working on it and it’s coming soon!

Limitations

Statically linked executables are not currently supported with the simulation instrument. If this is a limitation for your use case, please open a GitHub issue. Additionally, the simulation instrument currently does not follow child processes. Support for this is planned for a future release.

Example

terminal

$ codspeed exec -m simulation -- <my_command>

►►► Running the benchmarks
Executing: <my_command>

►►► Uploading results
Linked repository: [...]
Performance data uploaded

►►► Benchmark results

<my_command>: 26.74 ms

To see the full report, visit: https://codspeed.io/[...]

Setting instrument with `codspeed use`

If you do not want to always explicitly specify the instrument with -m when running benchmarks, you can set an instrument for your whole shell session by using

terminal

$ codspeed use simulation

This will set the simulation instrument as the default for all subsequent codspeed run and codspeed exec commands in the current shell session. You can check the currently selected instrument with

terminal

$ codspeed show
Simulation

Even if you ran codspeed use <instrument>, you can still override the instrument for a specific command by using the -m flag with codspeed run or codspeed exec.

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development.

If you want more details on how to configure the CodSpeed action, you can check out the Continuous Reporting section.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

Contrary to other CI usages, the run input is intentionally omitted here.This is what defines whether the action will run benchmarks defined in your configuration file, or a benchmark that makes use of one of the CodSpeed integrations.

Manual tool installation

On Ubuntu and Debian, the CodSpeed CLI automatically installs the tools required by each instrument. On other distributions, you need to install them manually.

Walltime

The walltime instrument uses linux perf for profiling, which is enabled by default. Refer to your distribution’s documentation for how to install the perf tool. If you don’t need profiling data, you can disable it by setting the CODSPEED_PERF_ENABLED environment variable to false.

Simulation

The simulation instrument requires CodSpeed’s fork of Valgrind. You need to build and install it from source. Prerequisites: git, autoconf, automake, make, and a C compiler.

git clone https://github.com/CodSpeedHQ/valgrind-codspeed.git
cd valgrind-codspeed
./autogen.sh
./configure
make -j
make install

After installation, verify that the CodSpeed version of Valgrind is in your PATH:

$ valgrind --version
valgrind-3.26.0.codspeed

The valgrind binary in your PATH must be the CodSpeed version for the simulation instrument to work correctly.

Memory

The memory instrument requires a Linux kernel with eBPF support. No additional tool installation is needed, the CLI will download and load the necessary tools automatically.

Next Steps

Profiling

Learn how to use flame graphs and profiling data to optimize your code

Instruments

Dive into instruments and learn how to measure different aspects of your program’s performance

Get Started

For Agents

Instruments

Integrations

Explore Features

Getting Started

Installation

Run your first command

Configuration

Benchmark fields

Running benchmarks

Example configuration

Instruments

Walltime

How it works

Walltime Options

Example

Memory

Example

Simulation

Limitations

Example

Setting instrument with `codspeed use`

Running the benchmarks in your CI

Manual tool installation

Walltime

Simulation

Memory

Next Steps

Profiling

Instruments

Get Started

For Agents

Instruments

Integrations

Explore Features

​Getting Started

​Installation

​Run your first command

​Configuration

​Benchmark fields

​Running benchmarks

​Example configuration

​Instruments

​Walltime

​How it works

​Walltime Options

​Example

​Memory

​Example

​Simulation

​Limitations

​Example

​Setting instrument with codspeed use

​Running the benchmarks in your CI

​Manual tool installation

​Walltime

​Simulation

​Memory

​Next Steps

Profiling

Instruments

Getting Started

Installation

Run your first command

Configuration

Benchmark fields

Running benchmarks

Example configuration

Instruments

Walltime

How it works

Walltime Options

Example

Memory

Example

Simulation

Limitations

Example

Setting instrument with `codspeed use`

Running the benchmarks in your CI

Manual tool installation

Walltime

Simulation

Memory

Next Steps