Installation
First install the plugin
@codspeed/tinybench-plugin
and tinybench
(if not already installed):
npm install --save-dev tinybench @codspeed/tinybench-plugin
Usage
Creating benchmarks
Let’s create a fibonacci function and benchmark it with tinybench and the
CodSpeed plugin:
import { withCodSpeed } from "@codspeed/tinybench-plugin";
import { Bench } from "tinybench";
function fibonacci(n: number): number {
if (n < 2) {
return n;
}
return fibonacci(n - 1) + fibonacci(n - 2);
}
const bench = withCodSpeed(new Bench());
bench
.add("fibonacci10", () => {
fibonacci(10);
})
.add("fibonacci15", () => {
fibonacci(15);
});
await bench.run();
console.table(bench.table());
import { withCodSpeed } from "@codspeed/tinybench-plugin";
import { Bench } from "tinybench";
function fibonacci(n: number): number {
if (n < 2) {
return n;
}
return fibonacci(n - 1) + fibonacci(n - 2);
}
const bench = withCodSpeed(new Bench());
bench
.add("fibonacci10", () => {
fibonacci(10);
})
.add("fibonacci15", () => {
fibonacci(15);
});
await bench.run();
console.table(bench.table());
import { Bench } from "tinybench";
import { withCodSpeed } from "@codspeed/tinybench-plugin";
function fibonacci(n) {
if (n < 2) {
return n;
}
return fibonacci(n - 1) + fibonacci(n - 2);
}
const bench = withCodSpeed(new Bench());
bench
.add("fibonacci10", () => {
fibonacci(10);
})
.add("fibonacci15", () => {
fibonacci(15);
});
await bench.run();
console.table(bench.table());
Noticed the .mjs
extension? This is because we’re using the ESM module format.
Saving our file with the .js
extension would have worked as well, but we would
have needed to add "type": "module"
to our package.json
file to instruct
Node.js to use the ESM module format.
If you’re working with CommonJS modules, you can totally use the require
syntax for importing the libraries.
Here, a few things are happening:
-
We create a simple recursive fibonacci function.
-
We create a new Bench
instance with CodSpeed support by using the
withCodSpeed
helper. This step is critical to enable CodSpeed on
your benchmarks.
-
We add two benchmarks to the suite and launch it, benching our fibonacci
function for 10 and 15.
Testing the benchmarks locally
Now, we can run our benchmarks locally to make sure everything is working as
expected:
TypeScript runner
To run the .ts
file directly, we recommend using
esbuild-register
. It allows
running TypeScript & ESM files directly with Node.js.
npm install --save-dev esbuild-register
$ node -r esbuild/register benches/bench.ts
[CodSpeed] 2 benches detected but no instrumentation found
[CodSpeed] falling back to tinybench
┌─────────┬───────────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ Average Time (ns) │ Margin │
├─────────┼───────────────┼───────────────────┼──────────┤
│ 0 │ 'fibonacci10' │ 552.4139857896414 │ '±0.18%' │
│ 1 │ 'fibonacci15' │ 5633.276191749634 │ '±0.14%' │
└─────────┴───────────────┴───────────────────┴──────────┘
TypeScript runner
To run the .ts
file directly, we recommend using
esbuild-register
. It allows
running TypeScript & ESM files directly with Node.js.
npm install --save-dev esbuild-register
$ node -r esbuild/register benches/bench.ts
[CodSpeed] 2 benches detected but no instrumentation found
[CodSpeed] falling back to tinybench
┌─────────┬───────────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ Average Time (ns) │ Margin │
├─────────┼───────────────┼───────────────────┼──────────┤
│ 0 │ 'fibonacci10' │ 552.4139857896414 │ '±0.18%' │
│ 1 │ 'fibonacci15' │ 5633.276191749634 │ '±0.14%' │
└─────────┴───────────────┴───────────────────┴──────────┘
$ node benches/bench.mjs
[CodSpeed] 2 benches detected but no instrumentation found
[CodSpeed] falling back to tinybench
┌─────────┬───────────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ Average Time (ns) │ Margin │
├─────────┼───────────────┼───────────────────┼──────────┤
│ 0 │ 'fibonacci10' │ 552.4139857896414 │ '±0.18%' │
│ 1 │ 'fibonacci15' │ 5633.276191749634 │ '±0.14%' │
└─────────┴───────────────┴───────────────────┴──────────┘
And… Congrats🎉, CodSpeed is installed in your benchmarking suite! Locally,
CodSpeed will fall back to tinybench since the instrumentation is only available
in the CI environment for now.
You can now
run those benchmarks in your CI to get
consistent performance measurements.
Integrating into a bigger project, multiple benchmark files
Often time you will not be writing your benchmarks in a single file. Indeed, it
can become quite difficult to maintain a single file with all your benchmarks as
your project grows.
For these kind of situations, we recommend the following approach. Let’s say you
have a file structure like this, in a project with TypeScript:
.
├── bench
│ ├── fibo.bench.ts
│ ├── foobarbaz.bench.ts
│ └── index.bench.ts
├── package.json
├── src
│ ├── fibonacci.ts
│ └── foobarbaz.ts
└── tsconfig.json
- The
src
directory contains the source code of the project. Here we have two
files, fibonacci.ts
and foobarbaz.ts
.
- The
bench
directory contains the benchmarks for the project. There is a file
for each source file that defines benchmarks for it.
- The
bench/index.bench.ts
file is the entry point for the benchmarks. It
imports all the other benchmark files and runs them.
import { Bench } from "tinybench";
import { iterativeFibonacci } from "../../src/fibonacci";
export function registerFiboBenchmarks(bench: Bench) {
bench
.add("test_iterative_fibo_10", () => {
iterativeFibonacci(10);
})
.add("test_iterative_fibo_100", () => {
iterativeFibonacci(100);
});
}
Here we define a function that takes an instance of Bench
as a parameter and
then adds some benchmarks to it. This will allow us to add benchmarks to the
same suite from multiple files.
import { withCodSpeed } from "@codspeed/tinybench-plugin";
import { Bench } from "tinybench";
import { registerFiboBenchmarks } from "./fibo.bench";
import { registerFoobarbazBenchmarks } from "./foobarbaz.bench";
export const bench = withCodSpeed(new Bench());
(async () => {
registerFiboBenchmarks(bench);
registerFoobarbazBenchmarks(bench);
await bench.run();
console.table(bench.table());
})();
Here all the functions registering benchmarks are executed to import all the
benchmarks from the different files.
To run the benchmarks, use the following command:
node -r esbuild-register bench/index.bench.ts
Running the benchmarks in your CI
To generate performance reports, you need to run the benchmarks in your CI. This
allows CodSpeed to detect the CI environment and properly configure the
instrumented environment.
Here is an example of a GitHub Actions workflow that runs the benchmarks and
reports the results to CodSpeed on every push to the main
branch and every
pull request:
.github/workflows/codspeed.yml
name: CodSpeed
on:
push:
branches:
- "main" # or "master"
pull_request:
# `workflow_dispatch` allows CodSpeed to trigger backtest
# performance analysis in order to generate initial data.
workflow_dispatch:
jobs:
benchmarks:
name: Run benchmarks
runs-on: ubuntu-latest
steps:
- uses: "actions/checkout@v4"
- uses: "actions/setup-node@v3"
- name: Install dependencies
run: npm install
- name: Run benchmarks
uses: CodSpeedHQ/action@v3
with:
run: node benches/bench.mjs
token: ${{ secrets.CODSPEED_TOKEN }}