Writing Benchmarks in Java

The Java integration is still in early development, and only the walltime instrument is currently supported.If you have any feedback, please reach out to us via Discord or email our support.

Integrating CodSpeed into your Java project works through a fork of JMH (Java Microbenchmark Harness). You write standard JMH benchmarks and swap in the CodSpeed JMH fork as a dependency. When running in CI with CodSpeed, the results are automatically collected and reported.

Installation

CodSpeed provides a fork of JMH that collects walltime results and sends them to CodSpeed.

Gradle
Maven

Add the CodSpeed JMH fork as a Git submodule:

git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm

Then include it as a composite build in your settings.gradle.kts, with dependency substitution to redirect JMH dependencies to the CodSpeed fork:

settings.gradle.kts

includeBuild("third-party/codspeed-jvm/jmh-fork") {              
    dependencySubstitution {                          
        substitute(module("org.openjdk.jmh:jmh-core"))           
            .using(project(":jmh-core"))              
        substitute(module("org.openjdk.jmh:jmh-generator-annprocess")) 
            .using(project(":jmh-generator-annprocess")) 
    }                                                 
}                                                     

We recommend using the JMH Gradle Plugin as it handles benchmark compilation and provides the jmh task. Add it to your build.gradle.kts:

build.gradle.kts

plugins {
    java
    id("me.champeau.jmh") version "0.7.2"
}

Add the CodSpeed JMH fork as a Git submodule and publish it to your local Maven repository:

git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
cd third-party/codspeed-jvm
./gradlew -p jmh-fork publishToMavenLocal

Then replace the JMH dependencies in your pom.xml with the CodSpeed fork. Maven resolves from ~/.m2/repository by default. The repository entry below is only necessary if your local repository is at a non-default location:

pom.xml

<repositories>
    <repository>
        <id>local</id>
        <url>file://${user.home}/.m2/repository</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <version>1.37</version>
        <groupId>io.codspeed.jmh</groupId>
        <version>0.1.0</version>
        <artifactId>jmh-core</artifactId>
    </dependency>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <version>1.37</version>
        <groupId>io.codspeed.jmh</groupId>
        <version>0.1.0</version>
        <artifactId>jmh-generator-annprocess</artifactId>
    </dependency>
</dependencies>

Your Maven project must be configured to produce an executable benchmark JAR. See the official JMH setup guide for instructions.

We’re planning to publish to Maven Central in the future. If you need this, please reach out via Discord or email.

Creating benchmarks

Write your benchmarks using standard JMH annotations:

FibBenchmark.java

package bench;

import org.openjdk.jmh.annotations.*;

public class FibBenchmark {

  @Benchmark
  public long fib() {
    return fib(30);
  }

  private static long fib(int n) {
    if (n <= 1) return n;
    return fib(n - 1) + fib(n - 2);
  }
}

JMH benchmarks must return their result or use Blackhole.consume() to prevent the JVM from eliminating dead code. All examples on this page return the computed value.

For an in-depth tutorial on JMH, see the How to Benchmark Java with JMH guide.

Testing the benchmarks locally

To run the benchmarks with CodSpeed locally, first install the codspeed runner:

curl -fsSL https://codspeed.io/install.sh | sh

Then run your benchmarks with CodSpeed:

Gradle
Maven

terminal

$ codspeed run --mode walltime -- ./gradlew jmh
►►► Running the benchmarks
# JMH version: 0.1.0
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Benchmark: bench.FibBenchmark.fib

# Run progress: 0.00% complete, ETA 00:01:40
# Warmup Iteration   1: 112.541 ns/op
...

Benchmark              Mode  Cnt      Score      Error  Units
FibBenchmark.fib       avgt    5    110.342 ±    2.153  ns/op

terminal

$ mvn package -q
$ codspeed run --mode walltime -- java -jar target/benchmarks.jar
►►► Running the benchmarks
# JMH version: 0.1.0
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Benchmark: bench.FibBenchmark.fib

# Run progress: 0.00% complete, ETA 00:01:40
# Warmup Iteration   1: 112.541 ns/op
...

Benchmark              Mode  Cnt      Score      Error  Units
FibBenchmark.fib       avgt    5    110.342 ±    2.153  ns/op

Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development.

If you want more details on how to configure the CodSpeed action, you can check out the Continuous Reporting section.

Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the main branch and every pull request:

Gradle
Maven

Advanced usage

JMH provides many features for writing expressive benchmarks. Below is a selection that can be useful in CodSpeed benchmarks.

Parameterized benchmarks

Use @Param to run the same benchmark with different input values:

ParamBenchmark.java

package bench;

import org.openjdk.jmh.annotations.*;

@State(Scope.Benchmark)
public class ParamBenchmark {

  @Param({"10", "20", "30"})
  int n;

  @Benchmark
  public long fib() {
    return fib(n);
  }

  private static long fib(int n) {
    if (n <= 1) return n;
    return fib(n - 1) + fib(n - 2);
  }
}

Shared state

Use @State to share setup logic across benchmarks and control the scope of the state object:

StateBenchmark.java

package bench;

import org.openjdk.jmh.annotations.*;
import java.util.ArrayList;
import java.util.List;

@State(Scope.Benchmark)
public class StateBenchmark {

  List<Integer> list;

  @Setup
  public void setup() {
    list = new ArrayList<>();
    for (int i = 0; i < 1000; i++) {
      list.add(i);
    }
  }

  @Benchmark
  public int sum() {
    return list.stream().mapToInt(Integer::intValue).sum();
  }
}

For a deeper dive into JMH, see the dedicated guide:

How to Benchmark Java with JMH

An in-depth guide to writing JMH benchmarks: project setup, annotations, parameters, common pitfalls, and CodSpeed CI integration.

Compatibility

JDK 17 or later is required.
All standard JMH annotations are supported.
Only the Walltime instrument is supported
CodSpeed uses a custom mode for all benchmarks to collect statistically significant results. Any @BenchmarkMode annotations in your code are ignored.

If you run into issues or require certain features, please open an issue or join our Discord to get help.

Next steps

Example repository

The CodSpeed JVM repository with example JMH benchmarks.

How to Benchmark Java with JMH

An in-depth guide to writing JMH benchmarks with CodSpeed.

Walltime instrument

Learn more about the Walltime instrument and how to use it.

Dive into performance changes

Learn more about profiling and how to read flame graphs.

Get Started

For Agents

Instruments

Integrations

Explore Features

Writing Benchmarks in Java

Installation

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Advanced usage

Parameterized benchmarks

Shared state

How to Benchmark Java with JMH

Compatibility

Next steps

Example repository

How to Benchmark Java with JMH

Walltime instrument

Dive into performance changes

​Installation

​Creating benchmarks

​Testing the benchmarks locally

​Running the benchmarks in your CI

​Advanced usage

​Parameterized benchmarks

​Shared state

How to Benchmark Java with JMH

​Compatibility

​Next steps

Example repository

How to Benchmark Java with JMH

Walltime instrument

Dive into performance changes

Installation

Creating benchmarks

Testing the benchmarks locally

Running the benchmarks in your CI

Advanced usage

Parameterized benchmarks

Shared state

Compatibility

Next steps