Writing Benchmarks in Kotlin

The JVM integration is still in early development, and only the walltime instrument is currently supported.If you have any feedback, please reach out to us via Discord or email our support.

Integrating CodSpeed into your Kotlin project works through a fork of JMH (Java Microbenchmark Harness). You write standard JMH benchmarks in Kotlin and swap in the CodSpeed JMH fork as a dependency. When running in CI with CodSpeed, the results are automatically collected and reported. JMH generates its measurement harness from the compiled @Benchmark classes, so the setup is close to the Java one. Kotlin only changes how benchmark classes have to be declared, and which generator a Maven build has to run.

Installation

CodSpeed provides a fork of JMH that collects walltime results and sends them to CodSpeed.

Gradle
Maven

Add the CodSpeed JMH fork as a Git submodule:

git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm

Then include it as a composite build in your settings.gradle.kts, with dependency substitution to redirect JMH dependencies to the CodSpeed fork:

settings.gradle.kts

includeBuild("third-party/codspeed-jvm/jmh-fork") {              
    dependencySubstitution {                          
        substitute(module("org.openjdk.jmh:jmh-core"))           
            .using(project(":jmh-core"))              
        substitute(module("org.openjdk.jmh:jmh-generator-annprocess")) 
            .using(project(":jmh-generator-annprocess")) 
    }                                                 
}                                                     

We recommend using the JMH Gradle Plugin as it handles benchmark compilation and provides the jmh task. It generates the harness from the compiled Kotlin classes in src/jmh/kotlin, so no annotation processing setup is needed. Add it to your build.gradle.kts:

build.gradle.kts

plugins {
    kotlin("jvm") version "2.1.20"
    id("me.champeau.jmh") version "0.7.3"
}

kotlin {
    jvmToolchain(21)
}

Add the CodSpeed JMH fork as a Git submodule and publish it to your local Maven repository:

git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
cd third-party/codspeed-jvm
./gradlew -p jmh-fork publishToMavenLocal

We’re planning to publish to Maven Central in the future. If you need this, please reach out via Discord or email.Then replace the JMH dependencies in your pom.xml with the CodSpeed fork. Maven resolves from ~/.m2/repository by default. Kotlin projects use the JMH bytecode generator rather than the annotation processor, so the fork has to be substituted in two places: the jmh-core dependency and the generator used by exec-maven-plugin.

pom.xml

<properties>
    <jmh.version>1.37</jmh.version>
    <jmh.version>0.2.1</jmh.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <groupId>io.codspeed.jmh</groupId>
        <artifactId>jmh-core</artifactId>
        <version>${jmh.version}</version>
    </dependency>
</dependencies>

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <dependencies>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <groupId>io.codspeed.jmh</groupId>
            <artifactId>jmh-generator-bytecode</artifactId>
            <version>${jmh.version}</version>
        </dependency>
    </dependencies>
</plugin>

Your Maven project must be configured to compile Kotlin sources, run the JMH generator over the compiled classes, and package an executable benchmark JAR. The jmh-kotlin-benchmark-archetype from JMH generates a project with all of this wired up:

mvn archetype:generate -DinteractiveMode=false \
  -DarchetypeGroupId=org.openjdk.jmh \
  -DarchetypeArtifactId=jmh-kotlin-benchmark-archetype \
  -DarchetypeVersion=1.37 \
  -DgroupId=com.example -DartifactId=kotlin-benchmarks -Dversion=1.0

Creating benchmarks

Write your benchmarks using standard JMH annotations:

FibBenchmark.kt

package bench

import org.openjdk.jmh.annotations.Benchmark

open class FibBenchmark {

    @Benchmark
    fun fib(): Long = fib(30)

    private fun fib(n: Int): Long =
        if (n <= 1) n.toLong() else fib(n - 1) + fib(n - 2)
}

JMH benchmarks must return their result or use Blackhole.consume() to prevent the JVM from eliminating dead code. All examples on this page return the computed value.

Testing the benchmarks locally

To run the benchmarks with CodSpeed locally, first install the codspeed runner:

curl -fsSL https://codspeed.io/install.sh | sh

Then run your benchmarks with CodSpeed:

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development.

If you want more details on how to configure the CodSpeed action, you can check out the Continuous Reporting section.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

Gradle
Maven

.github/workflows/codspeed.yml

name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed
    steps:
      - uses: actions/checkout@v5
        with:
          submodules: recursive
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: walltime
          run: ./gradlew jmh

.github/workflows/codspeed.yml

name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed
    steps:
      - uses: actions/checkout@v5
        with:
          submodules: recursive
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
      - name: Install CodSpeed JMH fork
        run: cd third-party/codspeed-jvm && ./gradlew -p jmh-fork publishToMavenLocal
      - name: Build benchmark JAR
        run: mvn package -q
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: walltime
          run: java -jar target/benchmarks.jar

Advanced usage

JMH provides many features for writing expressive benchmarks. Below is a selection that can be useful in CodSpeed benchmarks.

Parameterized benchmarks

Use @Param to run the same benchmark with different input values. The annotated property has to be a var, since JMH assigns it before the benchmark runs and rejects final fields:

ParamBenchmark.kt

package bench

import org.openjdk.jmh.annotations.Benchmark
import org.openjdk.jmh.annotations.Param
import org.openjdk.jmh.annotations.Scope
import org.openjdk.jmh.annotations.State

@State(Scope.Benchmark)
open class ParamBenchmark {

    @Param("10", "20", "30")
    var n: Int = 0

    @Benchmark
    fun fib(): Long = fib(n)

    private fun fib(n: Int): Long =
        if (n <= 1) n.toLong() else fib(n - 1) + fib(n - 2)
}

Shared state

Use @State to share setup logic across benchmarks and control the scope of the state object:

StateBenchmark.kt

package bench

import org.openjdk.jmh.annotations.Benchmark
import org.openjdk.jmh.annotations.Scope
import org.openjdk.jmh.annotations.Setup
import org.openjdk.jmh.annotations.State

@State(Scope.Benchmark)
open class StateBenchmark {

    private lateinit var numbers: List<Int>

    @Setup
    fun setup() {
        numbers = (0 until 1000).toList()
    }

    @Benchmark
    fun sum(): Int = numbers.sum()
}

JMH annotations behave the same in Kotlin as in Java. For a deeper dive, see the dedicated guide:

How to Benchmark Java with JMH

An in-depth guide to writing JMH benchmarks: project setup, annotations, parameters, common pitfalls, and CodSpeed CI integration.

Compatibility

JDK 21 or later is required.
Benchmark classes must not be final, since JMH subclasses them to generate the measurement harness.
All standard JMH annotations are supported.
Only the Walltime instrument is supported.
CodSpeed uses a custom mode for all benchmarks to collect statistically significant results. Any @BenchmarkMode annotations in your code are ignored.

If you run into issues or require certain features, please open an issue or join our Discord to get help.

Next steps

Example repository

The CodSpeed JVM repository with example JMH benchmarks.

How to Benchmark Java with JMH

An in-depth guide to writing JMH benchmarks with CodSpeed.

Walltime instrument

Learn more about the Walltime instrument and how to use it.

Dive into performance changes

Learn more about profiling and how to read flame graphs.

Get Started

For Agents

Instruments

Integrations

Explore Features

Writing Benchmarks in Kotlin

Installation

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Advanced usage

Parameterized benchmarks

Shared state

How to Benchmark Java with JMH

Compatibility

Next steps

Example repository

How to Benchmark Java with JMH

Walltime instrument

Dive into performance changes

​Installation

​Creating benchmarks

​Testing the benchmarks locally

​Running the benchmarks in your CI

​Advanced usage

​Parameterized benchmarks

​Shared state

How to Benchmark Java with JMH

​Compatibility

​Next steps

Example repository

How to Benchmark Java with JMH

Walltime instrument

Dive into performance changes

Installation

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Advanced usage

Parameterized benchmarks

Shared state

Compatibility

Next steps