Writing benchmarks with vitest-bench

Installation

First install the plugin @codspeed/vitest-plugin and vitest (if not already installed):

The CodSpeed plugin is compatible with vitest v3.2 and above. Vitest 5 requires @codspeed/vitest-plugin>=6.0.0. This page uses the Vitest 5 syntax. For the Vitest 3 and 4 syntax, see the @codspeed/vitest-plugin README.

npm install --save-dev @codspeed/vitest-plugin vitest

Usage

Creating benchmarks

Let’s create a fibonacci function and benchmark it with Vitest’s bench fixture and the CodSpeed plugin:

TypeScript
JavaScript

src/fibo.bench.ts

import { test } from "vitest";

function fibonacci(n: number): number {
  if (n < 2) {
    return n;
  }
  return fibonacci(n - 1) + fibonacci(n - 2);
}

test("fibo", async ({ bench }) => {
  await bench("fibo 10", () => {
    fibonacci(10);
  }).run();
  await bench("fibo 15", () => {
    fibonacci(15);
  }).run();
});

src/fibo.bench.js

import { test } from "vitest";

function fibonacci(n) {
  if (n < 2) {
    return n;
  }
  return fibonacci(n - 1) + fibonacci(n - 2);
}

test("fibo", async ({ bench }) => {
  await bench("fibo 10", () => {
    fibonacci(10);
  }).run();
  await bench("fibo 15", () => {
    fibonacci(15);
  }).run();
});

Here, a few things are happening:

We create a simple recursive fibonacci function.
We create a vitest test "fibo" that receives the bench fixture.
We declare two benchmarks with bench() and run each with .run(), benching our fibonacci function for 10 and 15. CodSpeed reports each bench() as its own benchmark, e.g., src/fibo.bench.ts::fibo::fibo 10.

To compare several implementations side by side, pass the benchmarks to bench.compare instead of calling .run() on each one:

test("fibo 15", async ({ bench }) => {
  await bench.compare(
    bench("recursive", () => {
      recursiveFibonacci(15);
    }),
    bench("iterative", () => {
      iterativeFibonacci(15);
    })
  );
});

CodSpeed still reports each bench() as its own benchmark, e.g., src/compare.bench.ts::fibo 15::recursive. Create or update the vitest.config.ts file to use the CodSpeed runner:

vitest.config.ts

import codspeedPlugin from "@codspeed/vitest-plugin";
import { defineConfig } from "vitest/config";

export default defineConfig({
  plugins: [codspeedPlugin()],
  // ...
});

Testing the benchmarks locally

Now, we can run our benchmarks locally to make sure everything is working as expected:

npx vitest bench --run

yarn vitest bench --run

pnpm vitest bench --run

This will run the benchmarks and output the results in the terminal. Add --reporter=verbose to display the results of each benchmark (for example, with pnpm):

terminal

$ pnpm vitest bench --run --reporter=verbose
 RUN  v5.0.4

 ✓ |bench| src/fibo.bench.ts > fibo 4020ms
   name               hz     min     max    mean     p75     p99    p995    p999     rme  samples
   fibo 10  2,678,987.94  0.0003  0.5480  0.0004  0.0004  0.0005  0.0005  0.0008  ±0.21%  2615096

   name             hz     min     max    mean     p75     p99    p995    p999     rme  samples
   fibo 15  259,938.79  0.0035  0.0813  0.0039  0.0040  0.0049  0.0063  0.0207  ±0.12%   255585

 Test Files  1 passed (1)
      Tests  1 passed (1)

Troubleshooting errorsIf you encounter an error like Failed to resolve "@codspeed/vitest-plugin" from "vitest.config.ts", you might need to rename the vitest.config.ts file to vitest.config.mts.You can find more information in the Vite documentation on importing ESM package.

And… Congrats🎉, CodSpeed is installed in your benchmarking suite! When not used in the CI environment or with the CLI, CodSpeed will fallback to using the default vitest. You can now run those benchmarks in your CI to get consistent performance measurements.

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development.

If you want more details on how to configure the CodSpeed action, you can check out the Continuous Reporting section.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

.github/workflows/codspeed.yml

name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
      - name: Install dependencies
        run: npm install
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: simulation
          run: npm exec vitest bench

Recipes

Running benchmarks in parallel CI jobs

To parallelize your benchmarks, you can use the shard options from vitest. For example with pnpm on Github Actions:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: ["1/5","2/5","3/5","4/5","5/5"]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
      - name: Install dependencies
        run: npm install
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: simulation
          run: pnpm vitest bench --run --shard=${{ matrix.shard }}

Same benchmark with different variationsFor now, you cannot run the same benchmarks several times within the same run. If the same benchmark is run multiple times, you will receive the following comment on your pull request:

Multiple Benchmark Variations Error Message

Learn more about benchmark sharding and how to integrate with your CI provider.

To combine measurement modes like simulation and memory, check out the documentation on running multiple instruments serially.

Limitations

Async code execution profiling

Execution profiles for async code can sometimes be unreliable, as profiling tools may lose stack trace information due to the event loop.

Get Started

For Agents

Instruments

Integrations

Explore Features

Writing benchmarks with vitest-bench

Installation

Usage

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Recipes

Running benchmarks in parallel CI jobs

Limitations

Async code execution profiling

​Installation

​Usage

​Creating benchmarks

​Testing the benchmarks locally

​Running the benchmarks in your CI

​Recipes

​Running benchmarks in parallel CI jobs

​Limitations

​Async code execution profiling

Installation

Usage

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Recipes

Running benchmarks in parallel CI jobs

Limitations

Async code execution profiling