> ## Documentation Index
> Fetch the complete documentation index at: https://codspeed.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Writing Benchmarks in Scala

> Create benchmarks for your Scala codebase using Java Microbenchmark Harness

export const codspeedJvmVersion = "0.2.1";

<Note>
  The JVM integration is still in early development, and **only the [walltime
  instrument](/docs/instruments/walltime) is currently supported**.

  If you have any feedback, please reach out to us via
  [Discord](https://discord.gg/MxpaCfKSqF) or
  [email our support](mailto:contact@codspeed.io).
</Note>

Integrating CodSpeed into your Scala project works through a fork of
[JMH](https://github.com/openjdk/jmh) (Java Microbenchmark Harness). You write
standard JMH benchmarks in Scala and swap in the CodSpeed JMH fork as a
dependency. When running in CI with CodSpeed, the results are automatically
collected and reported.

JMH generates its measurement harness from the compiled `@Benchmark` classes, so
no annotation processing setup is needed. Scala only changes how benchmark
classes have to be declared, and which generator a Maven build has to run.

<Info>
  This page covers Gradle and Maven. If you benchmark with
  [sbt-jmh](https://github.com/sbt/sbt-jmh), let us know on
  [Discord](https://discord.gg/MxpaCfKSqF) so we can prioritize it.
</Info>

## Installation

CodSpeed provides a fork of JMH that collects walltime results and sends them to
CodSpeed.

<Tabs>
  <Tab title="Gradle">
    Add the CodSpeed JMH fork as a Git submodule:

    ```bash theme={null}
    git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
    ```

    Then include it as a composite build in your `settings.gradle.kts`, with
    dependency substitution to redirect JMH dependencies to the CodSpeed fork:

    ```kotlin settings.gradle.kts theme={null}
    includeBuild("third-party/codspeed-jvm/jmh-fork") {              // [!code ++]
        dependencySubstitution {                          // [!code ++]
            substitute(module("org.openjdk.jmh:jmh-core"))           // [!code ++]
                .using(project(":jmh-core"))              // [!code ++]
            substitute(module("org.openjdk.jmh:jmh-generator-annprocess")) // [!code ++]
                .using(project(":jmh-generator-annprocess")) // [!code ++]
        }                                                 // [!code ++]
    }                                                     // [!code ++]
    ```

    We recommend using the
    [JMH Gradle Plugin](https://github.com/melix/jmh-gradle-plugin) as it handles
    benchmark compilation and provides the `jmh` task. It generates the harness from
    the compiled Scala classes in `src/jmh/scala`, which need the Scala library
    added to the `jmh` configuration:

    ```kotlin build.gradle.kts theme={null}
    plugins {
        scala
        id("me.champeau.jmh") version "0.7.3"// [!code ++]
    }

    java {
        toolchain {
            languageVersion.set(JavaLanguageVersion.of(21))// [!code ++]
        }
    }

    dependencies {
        implementation("org.scala-lang:scala3-library_3:3.3.6")
        jmh("org.scala-lang:scala3-library_3:3.3.6")// [!code ++]
    }
    ```
  </Tab>

  <Tab title="Maven">
    Add the CodSpeed JMH fork as a Git submodule and publish it to your local Maven
    repository:

    ```bash theme={null}
    git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
    cd third-party/codspeed-jvm
    ./gradlew -p jmh-fork publishToMavenLocal
    ```

    We're planning to publish to Maven Central in the future. If you need this,
    please reach out via [Discord](https://discord.gg/MxpaCfKSqF) or
    [email](mailto:contact@codspeed.io).

    Then replace the JMH dependencies in your `pom.xml` with the CodSpeed fork.
    Maven resolves from `~/.m2/repository` by default. Scala projects drive the JMH
    bytecode generator, so the fork has to be substituted in two places: the
    `jmh-core` dependency and the generator used by `exec-maven-plugin`:

    ```xml pom.xml theme={null}
    <properties>
        <jmh.version>1.37</jmh.version>// [!code --]
        <jmh.version>0.2.1</jmh.version>// [!code ++]
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>// [!code --]
            <groupId>io.codspeed.jmh</groupId>// [!code ++]
            <artifactId>jmh-core</artifactId>
            <version>${jmh.version}</version>
        </dependency>
    </dependencies>

    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>org.openjdk.jmh</groupId>// [!code --]
                <groupId>io.codspeed.jmh</groupId>// [!code ++]
                <artifactId>jmh-generator-bytecode</artifactId>
                <version>${jmh.version}</version>
            </dependency>
        </dependencies>
    </plugin>
    ```

    <Note>
      Your Maven project must be configured to compile Scala sources, run the JMH
      generator over the compiled classes, and package an executable benchmark JAR.
      The `jmh-scala-benchmark-archetype` from JMH generates a project with all of
      this wired up:

      ```bash theme={null}
      mvn archetype:generate -DinteractiveMode=false \
        -DarchetypeGroupId=org.openjdk.jmh \
        -DarchetypeArtifactId=jmh-scala-benchmark-archetype \
        -DarchetypeVersion=1.37 \
        -DgroupId=com.example -DartifactId=scala-benchmarks -Dversion=1.0
      ```
    </Note>

    <Warning>
      The archetype pins Scala 2.11.8 and `scala-maven-plugin` 3.2.2, which cannot
      compile against JDK 21. Update both, and point the dependency at the matching
      standard library:

      ```xml pom.xml theme={null}
      <dependency>
          <groupId>org.scala-lang</groupId>
          <artifactId>scala-library</artifactId>// [!code --]
          <artifactId>scala3-library_3</artifactId>// [!code ++]
          <version>${scala.stdLib.version}</version>
      </dependency>

      <properties>
          <scala.stdLib.version>2.11.8</scala.stdLib.version>// [!code --]
          <scala.stdLib.version>3.3.6</scala.stdLib.version>// [!code ++]
          <scala.mavenPlugin.version>3.2.2</scala.mavenPlugin.version>// [!code --]
          <scala.mavenPlugin.version>4.9.2</scala.mavenPlugin.version>// [!code ++]
      </properties>
      ```
    </Warning>
  </Tab>
</Tabs>

## Creating benchmarks

Write your benchmarks using standard JMH annotations:

```scala FibBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.Benchmark

class FibBenchmark {

  @Benchmark
  def fib(): Long = fib(30)

  private def fib(n: Int): Long =
    if (n <= 1) n.toLong else fib(n - 1) + fib(n - 2)
}
```

<Warning>
  Benchmarks must live in a `class`, not an `object`. JMH subclasses the
  benchmark class to generate the measurement harness, and an `object` compiles
  to a final class, which fails the build with `Benchmark classes should not be
      final`.
</Warning>

<Info>
  JMH benchmarks must return their result or use `Blackhole.consume()` to
  prevent the JVM from eliminating dead code. All examples on this page return
  the computed value.
</Info>

## Testing the benchmarks locally

To run the benchmarks with CodSpeed locally, first install the `codspeed`
runner:

```bash theme={null}
curl -fsSL https://codspeed.io/install.sh | sh
```

Then run your benchmarks with CodSpeed:

<Tabs>
  <Tab title="Gradle">
    <CodeBlock language="shellsession" filename="terminal" icon="square-terminal">
      {[
              "$ codspeed run --mode walltime -- ./gradlew jmh",
              "►►► Running the benchmarks",
              `# JMH version: ${codspeedJvmVersion}`,
              "# VM version: JDK 21.0.9, OpenJDK 64-Bit Server VM, 21.0.9+10",
              "# Warmup: 3 iterations, 1 s each",
              "# Measurement: 3 iterations, 1 s each",
              "# Benchmark: bench.FibBenchmark.fib",
              "",
              "# Run progress: 0.00% complete, ETA 00:00:18",
              "# Warmup Iteration   1: 286.841 ops/s",
              "...",
              "",
              "Benchmark          Mode  Cnt    Score    Error  Units",
              "FibBenchmark.fib  thrpt    9  305.213 ± 44.639  ops/s",
            ].join("\n")}
    </CodeBlock>
  </Tab>

  <Tab title="Maven">
    <CodeBlock language="shellsession" filename="terminal" icon="square-terminal">
      {[
              "$ mvn package -q",
              "$ codspeed run --mode walltime -- java -jar target/benchmarks.jar",
              "►►► Running the benchmarks",
              `# JMH version: ${codspeedJvmVersion}`,
              "# VM version: JDK 21.0.9, OpenJDK 64-Bit Server VM, 21.0.9+10",
              "# Warmup: 3 iterations, 1 s each",
              "# Measurement: 3 iterations, 1 s each",
              "# Benchmark: com.example.MyBenchmark.fib",
              "",
              "# Run progress: 0.00% complete, ETA 00:00:18",
              "# Warmup Iteration   1: 279.817 ops/s",
              "...",
              "",
              "Benchmark         Mode  Cnt    Score   Error  Units",
              "MyBenchmark.fib  thrpt    9  284.008 ± 4.488  ops/s",
            ].join("\n")}
    </CodeBlock>
  </Tab>
</Tabs>

## Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This
allows CodSpeed to automatically run benchmarks and warn you about regressions
during development.

<Tip>
  If you want more details on how to configure the CodSpeed action, you can check
  out the [Continuous Reporting section](/docs/integrations/ci).
</Tip>

Here is an example of a GitHub Actions workflow that runs the benchmarks and
reports the results to CodSpeed on every push to the `main` branch and every
pull request:

<Tabs>
  <Tab title="Gradle">
    ```yaml .github/workflows/codspeed.yml icon="github" theme={null}
    name: CodSpeed Benchmarks

    on:
      push:
        branches:
          - "main" # or "master"
      pull_request:
      # `workflow_dispatch` allows CodSpeed to trigger backtest
      # performance analysis in order to generate initial data.
      workflow_dispatch:

    jobs:
      benchmarks:
        name: Run benchmarks
        runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
        permissions: # optional for public repositories
          contents: read # required for actions/checkout
          id-token: write # required for OIDC authentication with CodSpeed
        steps:
          - uses: actions/checkout@v5
            with:
              submodules: recursive
          - uses: actions/setup-java@v4
            with:
              distribution: temurin
              java-version: 21
          - name: Run the benchmarks
            uses: CodSpeedHQ/action@v5
            with:
              mode: walltime
              run: ./gradlew jmh
    ```
  </Tab>

  <Tab title="Maven">
    ```yaml .github/workflows/codspeed.yml icon="github" theme={null}
    name: CodSpeed Benchmarks

    on:
      push:
        branches:
          - "main" # or "master"
      pull_request:
      # `workflow_dispatch` allows CodSpeed to trigger backtest
      # performance analysis in order to generate initial data.
      workflow_dispatch:

    jobs:
      benchmarks:
        name: Run benchmarks
        runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
        permissions: # optional for public repositories
          contents: read # required for actions/checkout
          id-token: write # required for OIDC authentication with CodSpeed
        steps:
          - uses: actions/checkout@v5
            with:
              submodules: recursive
          - uses: actions/setup-java@v4
            with:
              distribution: temurin
              java-version: 21
          - name: Install CodSpeed JMH fork
            run: cd third-party/codspeed-jvm && ./gradlew -p jmh-fork publishToMavenLocal
          - name: Build benchmark JAR
            run: mvn package -q
          - name: Run the benchmarks
            uses: CodSpeedHQ/action@v5
            with:
              mode: walltime
              run: java -jar target/benchmarks.jar
    ```
  </Tab>
</Tabs>

## Advanced usage

JMH provides many features for writing expressive benchmarks. Below is a
selection that can be useful in CodSpeed benchmarks.

### Parameterized benchmarks

Use `@Param` to run the same benchmark with different input values. Scala passes
the values as an `Array`, and the annotated field has to be a `var`, since JMH
assigns it before the benchmark runs and rejects final fields:

```scala ParamBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.{Benchmark, Param, Scope, State}

@State(Scope.Benchmark)
class ParamBenchmark {

  @Param(Array("10", "20", "30"))
  var n: Int = 0

  @Benchmark
  def fib(): Long = fib(n)

  private def fib(k: Int): Long =
    if (k <= 1) k.toLong else fib(k - 1) + fib(k - 2)
}
```

### Shared state

Use `@State` to share setup logic across benchmarks and control the scope of the
state object:

```scala StateBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.{Benchmark, Scope, Setup, State}

@State(Scope.Benchmark)
class StateBenchmark {

  private var numbers: Vector[Int] = Vector.empty

  @Setup
  def setup(): Unit = {
    numbers = (0 until 1000).toVector
  }

  @Benchmark
  def sum(): Int = numbers.sum
}
```

JMH annotations behave the same in Scala as in Java. For a deeper dive, see the
dedicated guide:

<Card title="How to Benchmark Java with JMH" icon="java" href="/docs/guides/how-to-benchmark-java-with-jmh">
  An in-depth guide to writing JMH benchmarks: project setup, annotations,
  parameters, common pitfalls, and CodSpeed CI integration.
</Card>

## Compatibility

* **JDK 21 or later** is required.
* Benchmark classes must not be final, since JMH subclasses them to generate the
  measurement harness.
* All standard JMH annotations are supported.
* Only the [Walltime instrument](/docs/instruments/walltime) is supported.
* CodSpeed uses a custom mode for all benchmarks to collect statistically
  significant results. Any `@BenchmarkMode` annotations in your code are
  ignored.

If you run into issues or require certain features, please
[open an issue](https://github.com/CodSpeedHQ/codspeed-jvm/issues) or
[join our Discord](https://discord.com/invite/MxpaCfKSqF) to get help.

## Next steps

<CardGroup>
  <Card title="Example repository" icon="github" href="https://github.com/CodSpeedHQ/codspeed-jvm">
    The CodSpeed JVM repository with example JMH benchmarks.
  </Card>

  <Card title="How to Benchmark Java with JMH" icon="java" href="/docs/guides/how-to-benchmark-java-with-jmh">
    An in-depth guide to writing JMH benchmarks with CodSpeed.
  </Card>

  <Card title="Walltime instrument" icon="stopwatch" href="/docs/instruments/walltime">
    Learn more about the Walltime instrument and how to use it.
  </Card>

  <Card title="Dive into performance changes" icon="bars-sort" href="/docs/features/profiling">
    Learn more about profiling and how to read flame graphs.
  </Card>
</CardGroup>