Writing benchmarks with vitest-bench

​

Installation

First install the plugin @codspeed/vitest-plugin and vitest (if not already installed):

npm install --save-dev @codspeed/vitest-plugin vitest

​

Usage

​

Creating benchmarks

Let’s create a fibonacci function and benchmark it with vitest and the CodSpeed plugin:

import { bench, describe } from "vitest";

function fibonacci(n: number): number {
  if (n < 2) {
    return n;
  }
  return fibonacci(n - 1) + fibonacci(n - 2);
}

describe("fibo", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });
  bench("fibo 15", () => {
    fibonacci(15);
  });
});

import { bench, describe } from "vitest";

function fibonacci(n: number): number {
  if (n < 2) {
    return n;
  }
  return fibonacci(n - 1) + fibonacci(n - 2);
}

describe("fibo", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });
  bench("fibo 15", () => {
    fibonacci(15);
  });
});

import { bench, describe } from "vitest";

function fibonacci(n) {
  if (n < 2) {
    return n;
  }
  return fibonacci(n - 1) + fibonacci(n - 2);
}

describe("fibo", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });
  bench("fibo 15", () => {
    fibonacci(15);
  });
});

Here, a few things are happening:

• We create a simple recursive fibonacci function.
• We create a new vitest suite "fibo" with two benchmarks, benching our fibonacci function for 10 and 15.

Create or update the vitest.config.ts file to use the CodSpeed runner:

import codspeedPlugin from "@codspeed/vitest-plugin";
import { defineConfig } from "vitest/config";

export default defineConfig({
  plugins: [codspeedPlugin()],
  // ...
});

​

Testing the benchmarks locally

Now, we can run our benchmarks locally to make sure everything is working as expected:

npx vitest bench --run

This will run the benchmarks and output the results in the terminal (for example, with pnpm):

$ pnpm vitest bench --run
[CodSpeed] bench detected but no instrumentation found, falling back to default vitest runner
 RUN  v1.0.3 ...

✓ benches/flat.bench.ts (2) 1521ms
✓ fibo (2) 1520ms
name min max mean rme samples
· fibo 10 0.0005 1.0279 0.0006 ±0.50% 880344 fastest
· fibo 15 0.0052 0.0594 0.0057 ±0.08% 86967

BENCH Summary

fibo 10 - benches/flat.bench.ts > fibo 10.12x faster than fibo 15

And… Congrats🎉, CodSpeed is installed in your benchmarking suite! Locally, CodSpeed will fall back to vitest since the instrumentation is only available in the CI environment for now.

You can now run those benchmarks in your CI to get consistent performance measurements.

​

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to detect the CI environment and properly configure the instrumented environment.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

name: CodSpeed

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    steps:
      - uses: "actions/checkout@v4"
      - uses: "actions/setup-node@v3"
      - name: Install dependencies
        run: npm install
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: npm exec vitest bench
          token: ${{ secrets.CODSPEED_TOKEN }}

​

Recipes

​

Running benchmarks in parallel CI jobs

To parallelize your benchmarks, you can use the shard options from vitest.

For example with pnpm on Github Actions:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: ["1/5","2/5","3/5","4/5","5/5"]
    steps:
      - uses: "actions/checkout@v4"
      - uses: "actions/setup-node@v3"
      - name: Install dependencies
        run: npm install
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: pnpm vitest bench --run --shard=${{ matrix.shard }}
          token: ${{ secrets.CODSPEED_TOKEN }}

Learn more about benchmark sharding and how to integrate with your CI provider.