# MCP Server Source: https://codspeed.io/docs/ai/mcp Connect AI assistants to your CodSpeed performance data using the Model Context Protocol. The CodSpeed MCP server gives AI-powered tools direct access to your performance data — benchmark runs, comparisons, and flamegraphs — so you can investigate regressions, explore profiling results, and review performance changes without leaving your editor or chat interface. The server follows the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) specification and is hosted by CodSpeed at ``` https://mcp.codspeed.io/mcp ``` ## Getting started Install the plugin from the official Claude plugins marketplace: ```sh theme={null} /plugin install codspeed@claude-plugins-official ``` The plugin installs both the MCP server and [agent skills](/docs/ai/skills) automatically. Alternatively, you can add the MCP server directly: ```sh theme={null} claude mcp add --transport http CodSpeed https://mcp.codspeed.io/mcp ``` Add the following to your `.cursor/mcp.json` file: ```json title=".cursor/mcp.json" theme={null} { "mcpServers": { "CodSpeed": { "url": "https://mcp.codspeed.io/mcp" } } } ``` Cursor will prompt you to authenticate with CodSpeed via OAuth on first use. Add the following to your `~/.codeium/windsurf/mcp_config.json` file: ```json title="mcp_config.json" theme={null} { "mcpServers": { "CodSpeed": { "serverUrl": "https://mcp.codspeed.io/mcp" } } } ``` Add the CodSpeed MCP server and authenticate: ```sh theme={null} codex mcp add CodSpeed --url https://mcp.codspeed.io/mcp codex mcp login CodSpeed ``` Add the following to your `.vscode/mcp.json` file: ```json title=".vscode/mcp.json" theme={null} { "servers": { "CodSpeed": { "type": "http", "url": "https://mcp.codspeed.io/mcp" } } } ``` 1. Open **Settings** → **MCP**. 2. Click **Add MCP Server**. 3. Enter the following: * **Name**: `CodSpeed` * **URL**: `https://mcp.codspeed.io/mcp` 4. Save and authenticate with CodSpeed when prompted. ### Alternative: auto-detect with `add-mcp` If your tool supports it, you can use [`add-mcp`](https://github.com/neondatabase/add-mcp) to automatically detect installed agents and configure them: ```sh theme={null} npx add-mcp https://mcp.codspeed.io/mcp --name CodSpeed ``` ## Example prompts Once connected, you can ask your AI assistant questions like: * "Explain the regression on the `feat/my-great-feature` branch." * "Make my `foo_bar` function faster." * "What are the hottest functions in the `bench_foo` benchmark?" * "Analyze the flamegraph for `bench_parse` and refactor the hot path." * "Compare the flamegraphs of `bench_serialize` between `main` and `feat/new-encoder` and explain what changed." * "Find the bottleneck in `bench_api_handler` and open a PR to fix it." Because the MCP server runs inside coding agents, your assistant can cross-reference flamegraph hot spots with your actual source code — then suggest or apply optimizations directly. ## Available tools The CodSpeed MCP server exposes seven tools: List all CodSpeed-enabled repositories that you have access to. Returns repository names, visibility, and descriptions. ```js title="Query" theme={null} list_repositories({}); ``` ```json title="Response" theme={null} { "repositories": [ { "repository": "CodSpeedHQ/codspeed", "visibility": "public", "description": "CodSpeed is the all-in-one performance testing toolkit. Optimize code performance and catch regressions early." }, { "repository": "CodSpeedHQ/pytest-codspeed", "visibility": "public", "description": "A pytest plugin to create benchmarks" }, { "repository": "CodSpeedHQ/codspeed-rust", "visibility": "public", "description": "Crates to benchmark your Rust code" } ] } ``` List recent performance runs for a repository. Returns run IDs, commit hashes, status, event type, branch, and PR information. ```js title="Query" theme={null} list_runs({ repository: "CodSpeedHQ/codspeed", limit: 5 }); ``` ```json title="Response" theme={null} { "runs": [ { "id": "6a315367cef14236ed18cd7b", "commit": { "hash": "7c50010379a4cb2ff1f550e7e8c9790b4391bc8e", "message": "fix(valgrind): skip rustup-wrapped proxy in trace-children" }, "date": "2026-06-16T13:45:11.844Z", "status": "completed", "event": "push", "branch": "main" }, { "id": "6a315096823c2ab665d8a12b", "commit": { "hash": "707a61cf1ea55936a2405eabbeddf6b494c44757", "message": "fix(valgrind): skip rustup-wrapped proxy in trace-children" }, "date": "2026-06-16T13:33:10.082Z", "status": "completed", "event": "pull_request", "branch": "cod-2850-codspeed-cli-fails-with-rustup-proxy-error", "pullRequest": { "number": 405, "title": "fix(valgrind): skip rustup-wrapped proxy in trace-children" } } ] } ``` Inspect a single performance run and its benchmark results. Shows benchmark names, identifiers, and values. Accepts a run ID, branch, or defaults to the latest run. ```js title="Query" theme={null} get_run({ repository: "CodSpeedHQ/codspeed", run_id: "6a315367cef14236ed18cd7b", }); ``` ```text title="Response" theme={null} --- repository: CodSpeedHQ/codspeed run_id: 6a315367cef14236ed18cd7b branch: main commit: 7c50010 date: 2026-06-16T13:45:11.844Z --- ## Benchmarks (7 total) | Mode | Benchmark Name | Benchmark URI | Value | | ---------- | ----------------------- | ------------------------------------------------------------------------ | -------- | | Memory | `sleep 1` | `exec_harness::sleep 1` | 15.6 KB | | WallTime | `sleep 1` | `exec_harness::sleep 1` | 1 s | | Simulation | `sleep 1` | `exec_harness::sleep 1` | 126.4 µs | | Simulation | `write_events[10000]` | `crates/runner-shared/benches/memtrack_writer.rs::write_events[10000]` | 20 ms | | Simulation | `write_events[100000]` | `crates/runner-shared/benches/memtrack_writer.rs::write_events[100000]` | 207.3 ms | | Simulation | `write_events[500000]` | `crates/runner-shared/benches/memtrack_writer.rs::write_events[500000]` | 1 s | | Simulation | `write_events[1000000]` | `crates/runner-shared/benches/memtrack_writer.rs::write_events[1000000]` | 2.1 s | ``` Compare two performance runs and return a markdown performance report. Shows benchmark-level comparisons including improvements, regressions, and new or missing benchmarks. When the two runs ran in different environments (CPU, OS, runtime version, linked libraries), an "Environment Differences" section is added first. ```js title="Query" theme={null} compare_runs({ repository: "CodSpeedHQ/pytest-codspeed", head_run_id: "6a19d91790da9981fdd07f33", base_run_id: "6a19cc7a6edb082bd37ca2d1", }); ``` ```text title="Response" theme={null} --- repository: CodSpeedHQ/pytest-codspeed head_run_id: 6a19d91790da9981fdd07f33 head_commit: 7c66604 base_run_id: 6a19cc7a6edb082bd37ca2d1 base_commit: 5db906e impact: -0.011035181366613855 --- ## Environment Differences > ⚠️ The head and base runs were executed in different environments. Performance > differences may be caused by environment changes rather than code changes. ### Hardware - **CPU**: `AMD EPYC 9V74 80-Core Processor` → `AMD EPYC 7763 64-Core Processor` _…additional environment differences omitted_ ## Summary | Status | Count | | --------------- | ----: | | ⚡ Improvements | 17 | | ❌ Regressions | 20 | | ✅ Unchanged | 252 | ## Benchmark Results | Status | Mode | Benchmark Name | `BASE` | `HEAD` | Change | | ------ | -------- | --------------------------------- | ------- | -------- | ------- | | ❌ | WallTime | `test_recursive_fibo_20` | 5.3 ms | 5.9 ms | -10.48% | | ❌ | WallTime | `test_multiprocessing_map[10000]` | 98.1 ms | 108.2 ms | -9.38% | | ❌ | WallTime | `test_color[graph0-3]` | 31 µs | 32.7 µs | -5.16% | | ⚡ | WallTime | `test_multiprocessing_map[1000]` | 66.4 ms | 60.9 ms | +9.08% | | ⚡ | WallTime | `test_threadpool_map[10]` | 2.5 ms | 2.3 ms | +5.54% | | ⚡ | WallTime | `test_sudoku[initial_grid0]` | 8.3 µs | 7.9 µs | +4.81% | _…31 more changed benchmarks omitted_ ``` Retrieve the full structured result of a single benchmark in a performance run: the instrument metrics (wall-time distribution, simulated time breakdown, or memory allocation breakdown), known issues, and whether a flamegraph is available. ```js title="Query" theme={null} get_benchmark_result({ repository: "CodSpeedHQ/codspeed", run_id: "6a315367cef14236ed18cd7b", benchmark_uri: "exec_harness::sleep 1", runner_mode: "WallTime", }); ``` ```json title="Response" theme={null} { "name": "sleep 1", "uri": "exec_harness::sleep 1", "runnerMode": "WallTime", "flamegraphAvailable": true, "issues": { "callgraphGenerationFailure": null }, "walltime": { "minSeconds": 1.001428826, "lowerFenceSeconds": 1.001428826, "q1Seconds": 1.0014400695, "medianSeconds": 1.001462736, "q3Seconds": 1.001541066, "upperFenceSeconds": 1.001572194, "maxSeconds": 1.001572194, "meanSeconds": 1.0014850014, "stdevSeconds": 0.00005703226150872855, "totalTimeSeconds": 5.007425007, "rounds": 5, "iqrOutlierRounds": 0, "stdevOutlierRounds": 0, "iterPerRound": 1, "warmupIters": 0 } } ``` Query and summarize a flame graph from a performance run. Returns hot spots (functions with highest self time), the call tree, the most expensive threads, and timing information for each function. Use the optional `filters` object to narrow the flame graph: `pid` / `tid` restrict it to specific processes/threads (their intersection — use `list_threads` to discover the available PIDs/TIDs), and `root_function_name` re-roots it at a specific function for deeper exploration. ```js title="Query" theme={null} query_flamegraph({ repository: "CodSpeedHQ/codspeed", run_id: "6a315367cef14236ed18cd7b", benchmark_uri: "crates/runner-shared/benches/memtrack_writer.rs::write_events[1000000]", runner_mode: "Simulation", filters: { tid: [12] }, }); ``` ```text title="Response" theme={null} # Flame Graph Summary --- benchmark_name: write_events[1000000] runner_mode: Simulation benchmark_total_time: 2.1 s run_id: 6a315367cef14236ed18cd7b filter_tid: 12 truncated: yes --- ## Selected Threads | PID | TID | Process | Thread | Total Time | | --: | --: | --------------- | ------ | -------------: | | 1 | 12 | codspeed-runner | main | 2.1 s (99.98%) | ## Top 10 expensive functions (by self time) | Function | Origin | Calls | Self Time | Total Time | | ----------------------------------------------------- | ---------- | ----: | ----------------: | ----------------: | | `ZSTD_compressBlock_fast_extDict_generic.constprop.0` | Unknown | 2 | 468.5 ms (22.05%) | 572.8 ms (26.96%) | | `Unknown symbol (0x188a80)` | Unknown | 12 | 244.8 ms (11.52%) | 244.8 ms (11.52%) | | `ZSTD_encodeSequences_bmi2` | Unknown | 2 | 223.6 ms (10.52%) | 223.6 ms (10.52%) | | `ZSTD_compressBlock_fast` | Unknown | 4 | 213.3 ms (10.04%) | 213.3 ms (10.04%) | | `ZSTD_compressStream` | Unknown | 2 | 97.9 ms (4.61%) | 1.3 s (60.07%) | | `rmp::encode::str::write_str` (src/encode/str.rs) | rmp@0.8.15 | 1 | 93.3 ms (4.39%) | 213 ms (10.03%) | ## Most expensive calls (by total time) | Function | Origin | Self Time | Total Time | | -------------------------------------------------------------------- | --------------- | --------------: | -------------: | | `runner_shared::artifacts::memtrack::MemtrackWriter::write_event` | User code | 32.5 ms (1.53%) | 2.1 s (99.80%) | | `rmp_serde::encode::MaybeUnknownLengthCompound::end` | rmp-serde@1.3.1 | 18.3 ms (0.86%) | 1.6 s (77.23%) | | `zstd::stream::zio::writer::Writer::write` | zstd@0.13.3 | 30.8 ms (1.45%) | 1.4 s (64.57%) | _…call tree, per-function details, and remaining hot spots omitted_ > Call tree filtered: showing up to 25 entries with ≥1.00% of root function > total time. Re-query with `filters.root_function_name` set to a function of > interest for deeper exploration. ``` List the processes and threads recorded for a benchmark, with each thread's total execution time and its start offset from the beginning of the benchmark. Use it to discover the PIDs/TIDs available before narrowing a flame graph with `query_flamegraph`'s `filters`. Returns an empty list for benchmarks recorded without per-thread data (e.g. single-threaded profiles). ```json title="Input" theme={null} { "repository": "CodSpeedHQ/codspeed", "run_id": "6a315367cef14236ed18cd7b", "benchmark_uri": "crates/runner-shared/benches/memtrack_writer.rs::write_events[1000000]", "runner_mode": "Simulation" } ``` ```json title="Output" theme={null} { "processes": [ { "id": 1, "name": "codspeed-runner", "totalTimeSeconds": 2.1, "startOffsetSeconds": 0, "threads": [ { "id": 12, "name": "main", "totalTimeSeconds": 2.0995, "startOffsetSeconds": 0 }, { "id": 42, "name": "writer", "totalTimeSeconds": 0.0005, "startOffsetSeconds": 0.13 } ] } ] } ``` ## Authentication The CodSpeed MCP server uses **OAuth** for authentication. When you first connect, your MCP client will open a browser window where you log in to CodSpeed and authorize access. The client stores the resulting token and refreshes it automatically — no API keys to manage. The MCP server has access to the same repositories and data as your CodSpeed account. # Agent Skills Source: https://codspeed.io/docs/ai/skills Instruction sets that teach AI coding assistants how to set up benchmarks and optimize performance with CodSpeed. Agent skills are instruction sets that teach AI coding assistants how to perform specific tasks with CodSpeed. They work alongside the [MCP server](/docs/ai/mcp) — the MCP server provides access to your performance data, while skills provide the know-how to act on it. Skills require the [CodSpeed MCP server](/docs/ai/mcp) to be configured. Set up the MCP server first if you have not already. ## Installation Install the plugin from the official Claude plugins marketplace: ```sh theme={null} /plugin install codspeed@claude-plugins-official ``` The plugin installs both the [MCP server](/docs/ai/mcp) and agent skills automatically. Skills are available as markdown files in the [CodSpeed repository](https://github.com/CodSpeedHQ/codspeed/tree/main/skills). Copy the skill files into your agent's skill directory. ### Alternative: auto-detect with `npx skills` If your agent supports it, you can use [`npx skills`](https://github.com/vercel-labs/skills) to automatically detect installed agents and configure them: ```sh theme={null} npx skills add CodSpeedHQ/codspeed ``` ## Available skills ### Optimize ([`codspeed-optimize`](https://github.com/CodSpeedHQ/codspeed/blob/main/skills/codspeed-optimize/SKILL.md)) Turns your AI assistant into an autonomous performance engineer. It works in a loop and keeps iterating until there is nothing left to gain. 1. **Measure** — run the benchmarks to establish a baseline. 2. **Analyze** — query flamegraphs to identify hot spots and bottlenecks. 3. **Optimize** — make a targeted change to the source code. 4. **Re-measure and compare** — run benchmarks again and compare against the baseline. Repeat until no further gains. **Example prompts:** > "Make my `parse_input` function faster." > "Find the bottleneck in `bench_serialize` and fix it." > "Optimize the hot path in this module." > "There's a regression on `feat/parser` — investigate." ### Setup Harness ([`codspeed-setup-harness`](https://github.com/CodSpeedHQ/codspeed/blob/main/skills/codspeed-setup-harness/SKILL.md)) Analyzes your project, picks the right benchmarking framework for your language, writes representative benchmarks, and verifies the setup works through the CodSpeed CLI. Supports Rust, Python, Node.js, Go, C/C++, and any language via the exec harness. The Optimize skill relies on benchmarks to measure performance. If your project does not have benchmarks yet, the Setup Harness skill will be triggered automatically to create them. **Example prompts:** > "Add benchmarks to this project." > "Set up CodSpeed for my Rust project." > "Benchmark this function." # Wizard (@codspeedbot) Source: https://codspeed.io/docs/ai/wizard CodSpeed's AI agent that sets up benchmarks, explains performance changes, and proposes fixes directly in GitHub. The CodSpeed Wizard is an AI agent that sets up benchmarks and CI configurations, explains regressions, and proposes code fixes. Use it from the **CodSpeed dashboard** to set up a repository, or mention `@codspeedbot` in any **pull request comment**, **issue comment**, or **review comment** for ongoing performance work. @codspeedbot responding to a GitHub pull request comment with a performance breakdown and proposed fix

@codspeedbot responding to a GitHub pull request comment with a performance breakdown and proposed fix

## Example usage * **Performance breakdown**: ask for a summary of how a branch or pull request affects performance. ``` @codspeedbot what's the performance impact of this PR? ``` * **Explain a regression**: ask the Wizard to investigate why a benchmark regressed and describe the root cause. ``` @codspeedbot explain the regression on bench_parse ``` * **Propose a fix**: ask the Wizard to investigate a regression and open a pull request with a targeted fix. ``` @codspeedbot fix the regression on bench_serialize ``` * **Add benchmarks**: request benchmarks for a specific function or module. ``` @codspeedbot add a benchmark for the parse_config function ``` `@codspeedbot` requires the CodSpeed GitHub App to be installed on your repository. If you used the dashboard Wizard for initial setup, the app is already installed. The Wizard is enabled by default on all accounts and organizations, but can be disabled by an admin in the settings. ## Next steps } href="/docs/ai/mcp"> Connect AI coding assistants to your CodSpeed performance data directly from your editor. Teach AI assistants to set up benchmarks and optimize performance autonomously. # Writing Benchmarks with CLI Commands Source: https://codspeed.io/docs/benchmarks/cli-commands Benchmark any language with the CodSpeed CLI and a configuration file CodSpeed provides [dedicated integrations](/docs/benchmarks/overview) for several languages, but you do not need one to start benchmarking. The [CodSpeed CLI](/docs/cli) can benchmark **any executable program**: define the commands to measure in a `codspeed.yml` configuration file, and run them locally or in CI. This approach works for any language or toolchain, even if it does not have a dedicated integration yet, e.g., Zig, OCaml, or Swift. It is also a good fit for benchmarking a program end to end, like a CLI tool or a compiler. Benchmarks defined this way measure a whole command, including process startup. If your language has a [dedicated integration](/docs/benchmarks/overview), prefer it to benchmark individual functions. Otherwise, you can also [build a custom harness](#benchmarking-a-subpart-of-your-program) to measure only a subpart of your program. If you would like a dedicated integration for your language, let us know on [Discord](https://discord.gg/MxpaCfKSqF) or [open a GitHub issue](https://github.com/CodSpeedHQ/codspeed/issues/new). ## Installing the CodSpeed CLI Install the CodSpeed CLI using the installation script: ```sh theme={null} curl -fsSL https://codspeed.io/install.sh | sh ``` After installation, authenticate with your CodSpeed account by running `codspeed auth login`. ## Defining the benchmarks Create a `codspeed.yml` file at the root of your repository. Each item in the `benchmarks` list describes a command to benchmark: ```yaml codspeed.yml theme={null} $schema: https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json benchmarks: - name: parse large file exec: ./my-tool parse fixtures/large.json - name: render template exec: ./my-tool render fixtures/template.html options: max-time: 2s ``` The `exec` field is the command to run. Build the executable beforehand with your usual toolchain, CodSpeed measures the command as is. See the [CLI configuration reference](/docs/cli#configuration) for all available fields and options. ## Running the benchmarks locally Build your program, then run all the benchmarks defined in the configuration file with the [walltime instrument](/docs/instruments/walltime): ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed run -m walltime ►►► Running the benchmarks Executing: parse large file Completed 13 warmup rounds Warmup done, now performing 20 rounds Executing: render template Completed 23 warmup rounds Warmup done, now performing 68 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results ┌──────────────────┬─────────────┐ │ Benchmark │ Measurement │ ├──────────────────┼─────────────┤ │ parse large file │ 14.10 ms │ ├──────────────────┼─────────────┤ │ render template │ 6.91 ms │ └──────────────────┴─────────────┘ To see the full report, visit: https://app.codspeed.io/[...] ``` ## Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: The walltime instrument runs on [CodSpeed Macro Runners](/docs/features/macro-runners), bare-metal machines that deliver low-variance measurements: Contrary to other CI usages, **the `run` input is intentionally omitted** here: this is what makes the action run the benchmarks defined in your configuration file. ## Advanced ### Benchmarking a subpart of your program Benchmarks defined with `exec` measure the whole command, including process startup. To measure only a specific section of your program, e.g., its core processing loop, you can build a custom harness with the [`instrument-hooks`](https://github.com/CodSpeedHQ/instrument-hooks) library. A custom harness instruments your program directly: it tells CodSpeed exactly when the measured section starts and stops. The library is a single C file that integrates with virtually any language through FFI. Follow the [custom harness guide](https://github.com/CodSpeedHQ/instrument-hooks/blob/main/CUSTOM_HARNESS.md) to build one. Since your program then embeds its own harness, declare it with `entrypoint` instead of `exec` in the configuration file: ```yaml codspeed.yml theme={null} benchmarks: - name: parse large file exec: ./my-tool parse fixtures/large.json # [!code --] entrypoint: ./my-tool parse fixtures/large.json # [!code ++] ``` Similarly, to benchmark a harness-equipped command without a configuration file, use `codspeed run` instead of `codspeed exec`: ```sh theme={null} codspeed run -m walltime -- ./my-tool parse fixtures/large.json ``` ## Next Steps Explore all the commands, instruments, and configuration options of the CodSpeed CLI. Learn more about the walltime instrument and how to use it. Learn how to use flamegraphs and profiling data to optimize your code. Catch regressions automatically on every pull request. # Writing Benchmarks in C++ Source: https://codspeed.io/docs/benchmarks/cpp Create benchmarks for your C++ codebase using `google_benchmark` To use CodSpeed in your C++ codebase, you can use [CodSpeed's `google_benchmark` library](https://github.com/CodSpeedHQ/codspeed-cpp/tree/main/google_benchmark), which is a compatibility layer to run both instrumented and walltime CodSpeed benchmarks. ## Writing benchmarks CodSpeed integrates with the `google_benchmark` library. Here is a small example on how to declare benchmarks. Otherwise, any existing benchmarks of your project can be reused. ```cpp main.cpp theme={null} // Define the function under test static void BM_StringCopy(benchmark::State &state) { std::string x = "hello"; // Google benchmark relies on state.begin() and state.end() to run the benchmark and count iterations for (auto _ : state) { std::string copy(x); // Use DoNotOptimize and ClobberMemory to prevent the compiler optimizing away your benchmark // See: https://google.github.io/benchmark/user_guide.html#preventing-optimization benchmark::DoNotOptimize(copy); benchmark::ClobberMemory(); } } // Register the benchmarked to be called by the executable BENCHMARK(BM_StringCopy); static void BM_memcpy(benchmark::State &state) { char *src = new char[state.range(0)]; char *dst = new char[state.range(0)]; memset(src, 'x', state.range(0)); for (auto _ : state) { memcpy(dst, src, state.range(0)); benchmark::DoNotOptimize(dst); benchmark::ClobberMemory(); } delete[] src; delete[] dst; } BENCHMARK(BM_memcpy)->Range(8, 8 << 10); // Entrypoint of the benchmark executable BENCHMARK_MAIN(); ``` ### Preventing compiler optimizations The compiler removes any computation whose result is never used, and CodSpeed reports the benchmark as [optimized out](/docs/troubleshooting#optimized-out-benchmarks). As in the example above, keep the result alive with `benchmark::DoNotOptimize`, and follow it with `benchmark::ClobberMemory` when the benchmarked code writes to memory, as described in the [Google Benchmark user guide](https://google.github.io/benchmark/user_guide.html#preventing-optimization). For a deeper dive into writing benchmarks with Google Benchmark, including [the optimization patterns to avoid](/docs/guides/how-to-benchmark-cpp-with-google-benchmark#prevent-compiler-optimizations), see: An in-depth guide to writing Google Benchmark benchmarks: fixtures, parameterized benchmarks, custom counters, and CodSpeed CI integration. ## Building & Running benchmarks To build and run benchmarks, CodSpeed officially support usage of the `google_benchmark` library using both [`CMake`](#cmake) and [`Bazel`](#bazel). If you are using another build system, you may find guidelines in the [custom build systems section](#custom-build-systems) ### CMake To use CodSpeed's `google_benchmark` integration using [`CMake`](https://cmake.org/documentation/), you can declare a benchmark executable as follows: ```cmake CMakeLists.txt theme={null} cmake_minimum_required(VERSION 3.12) include(FetchContent) project(my_codspeed_project VERSION 0.0.0 LANGUAGES CXX) # Enable release mode with debug symbols to display useful profiling data set(CMAKE_BUILD_TYPE RelWithDebInfo) set(BENCHMARK_DOWNLOAD_DEPENDENCIES ON) FetchContent_Declare( google_benchmark GIT_REPOSITORY https://github.com/CodSpeedHQ/codspeed-cpp # Target the codspeed cpp repository SOURCE_SUBDIR google_benchmark # Make sure to target the google_benchmark subdirectory GIT_TAG main # Or chose a specific version or git ref, check the releases page on the repository ) FetchContent_MakeAvailable(google_benchmark) # Declare your benchmark executable and its sources here add_executable(my_benchmark_executable benches/bench.cpp) # Link your executable against the `benchmark::benchmark`, the `google_benchmark` library # Note: the first argument must match the first argument of the `add_executable` call target_link_libraries(my_benchmark_executable benchmark::benchmark) ``` Checkout the [releases page](https://github.com/CodSpeedHQ/codspeed-cpp/releases) if you want to target a specific version of the library. This example is a dedicated `CMakeLists.txt` file for the benchmark executable. You can also add an executable target to your existing project's `CMakeLists.txt`. Make sure to link this target against the `benchmark::benchmark` library. #### Building benchmarks To build the benchmark executable, run: ```shellsession title=terminal icon="square-terminal" theme={null} $ mkdir build && cd build $ cmake -DCODSPEED_MODE=simulation .. -- The CXX compiler identification is GNU 14.2.1 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- ... -- Configuring done (8.6s) -- Generating done (0.1s) -- Build files have been written to: /home/user/project-benchmark/build $ make [ 1%] Building CXX object ... ... ... [100%] Built target my_benchmark_executable ``` #### The `CODSPEED_MODE` flag Please note the `-DCODSPEED_MODE=simulation` flag in the `cmake` command. This will enable the CodSpeed CPU simulation mode for the benchmark executable, where each benchmark is run only once on a simulated CPU. If you omit the `CODSPEED_MODE` cmake flag, CodSpeed will not be enabled in the benchmark executable, and it will run as a regular benchmark. The CODSPEED\_MODE cmake flag can take the following values: * `off`: defaulted to when the cmake flag is not provided, disables codspeed. * `simulation`: benchmarks are run only once on a simulated CPU. * `walltime`: used for walltime codspeed reports, see [dedicated documentation](/docs/instruments/walltime) * `memory`: benchmarks are run once using [memory profiling](/docs/instruments/memory) * `instrumentation`: (deprecated) alias of `simulation`. #### Debug symbols In order to get the most out of CodSpeed reports, debug symbols need to be enabled within your executable. In the [example](#cmake) above, this is done by setting `CMAKE_BUILD_TYPE` to `RelWithDebInfo`. #### Running the benchmarks locally Simply execute the compiled binary to run the benchmarks. ```shellsession title=terminal icon="square-terminal" theme={null} $ ./benchmark_example Codspeed mode: simulation 2025-02-27T16:15:03+01:00 Running ./benchmark_example Run on (12 X 4500 MHz CPUs) CPU Caches: L1 Data 48 KiB (x6) L1 Instruction 32 KiB (x6) L2 Unified 1280 KiB (x6) L3 Unified 12288 KiB (x1) Load Average: 1.73, 1.71, 1.52 NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/main.cpp::BM_rand_vector Checked: benches/main.cpp::BM_StringCopy Checked: benches/main.cpp::BM_memcpy[8] Checked: benches/main.cpp::BM_memcpy[64] Checked: benches/main.cpp::BM_memcpy[512] Checked: benches/main.cpp::BM_memcpy[4096] Checked: benches/main.cpp::BM_memcpy[8192] ``` Congratulations ! 🎉 You can now [run those benchmark in your CI](#running-the-benchmarks-in-your-ci) to get the actual performance measurements. #### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: #### Running benchmarks in parallel CI jobs If your benchmarks are taking too much time to run under the CodSpeed action, you can run them in parallel to speed up the execution. To parallelize your benchmarks, first split them in multiple executables that each run a subset of your benches. ```cmake CMakelists.txt theme={null} # Create individual benchmark executables set(BENCHMARKS first_bench second_bench third_bench) # Add `bench_name` target with `bench_name.cpp` source for each bench listed above foreach(benchmark IN LISTS BENCHMARKS) add_executable(${benchmark} benches/${benchmark}.cpp) target_link_libraries(${benchmark} benchmark::benchmark ) endforeach() # Create a custom target to run all benchmarks locally add_custom_target(run_all_benchmarks COMMAND ${CMAKE_COMMAND} -E echo "Running all benchmarks..." ) # Register each benchmark target as a dependency of foreach(benchmark IN LISTS BENCHMARKS) add_custom_command( TARGET run_all_benchmarks POST_BUILD COMMAND ${CMAKE_COMMAND} -E echo "Running ${benchmark}..." COMMAND $ WORKING_DIRECTORY ${CMAKE_BINARY_DIR} ) endforeach() ``` Then update your CI workflow to run benchmarks executable by executable To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). ### Bazel You can also use CodSpeed's `google_benchmark` integration with the [`Bazel`](https://bazel.build/reference) integration. #### Building benchmarks Import the library from the [Bazel Central Registry](https://registry.bazel.build/modules/codspeed_google_benchmark_compat/) in your `MODULE.bazel` file ```python MODULE.bazel theme={null} module(name = "my_module") # Starting from 2.0.0, codspeed_google_benchmark_compat is available from the Bazel central registry bazel_dep(name = "codspeed_google_benchmark_compat", version = "2.0.0") ``` Then, define your benchmark target in your packages's `BUILD.bazel` file: ```python path/to/bench/BUILD.bazel theme={null} cc_binary( name = "my_benchmark", # Name of your benchmark target srcs = glob(["*.cpp", "*.hpp"]), # Or define sources however you wish deps = ["@codspeed_google_benchmark_compat//:benchmark"], ) ``` Finally, you can build the benchmarks by running: ```shellsession title=terminal icon="square-terminal" theme={null} $ bazel build //path/to/bench:my_benchmark \ --@codspeed_google_benchmark_compat//:codspeed_mode=simulation --compilation_mode=dbg \ --copt=-O2 INFO: Analyzed target //examples/google_benchmark:my_benchmark (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //examples/google_benchmark:my_benchmark up-to-date: bazel-bin/examples/google_benchmark/my_benchmark INFO: Elapsed time: 0.138s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action ``` #### Build options As you may have noticed in the example, there are a few key build options essential for bazel to make full use of the CodSpeed library. * `--@codspeed_google_benchmark_compat//:codspeed_mode=simulation` enables the codspeed features of the library, which can take the following values here: * `off`: defaulted to when the cli flag is not provided, disables codspeed. * `simulation`: benchmarks are run only once on a simulated CPU. * `walltime`: used for walltime codspeed reports, see [dedicated documentation](/docs/instruments/walltime) * `memory`: benchmarks are run once using [memory profiling](/docs/instruments/memory) * `instrumentation`: (deprecated) alias of `simulation`. * `--compilation_mode=dbg`: enables debug symbols in the compiled binary, used to generate meaningful CodSpeed reports. * `--copt=-O2`: sets the desired level of compiler optimizations in the benchmarks binary. **Setting default build options** If you do not want to specify these flags every time, you can create a `.bazelrc` file at the root of the bazel workspace with the following content ```sh theme={null} build --@codspeed_cpp//core:codspeed_mode=simulation build --compilation_mode=dbg build --copt=-O2 ``` #### Running the benchmarks locally You can then run your benchmarks by running: ```shellsession title=terminal icon="square-terminal" theme={null} $ bazel run //path/to/bench:my_benchmark \ --@codspeed_cpp//core:codspeed_mode=simulation ... Cached build step ... Codspeed mode: simulation 2025-02-27T16:15:03+01:00 Running ./benchmark_example Run on (12 X 4500 MHz CPUs) CPU Caches: L1 Data 48 KiB (x6) L1 Instruction 32 KiB (x6) L2 Unified 1280 KiB (x6) L3 Unified 12288 KiB (x1) Load Average: 1.73, 1.71, 1.52 NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/main.cpp::BM_rand_vector Checked: benches/main.cpp::BM_StringCopy Checked: benches/main.cpp::BM_memcpy[8] Checked: benches/main.cpp::BM_memcpy[64] Checked: benches/main.cpp::BM_memcpy[512] Checked: benches/main.cpp::BM_memcpy[4096] Checked: benches/main.cpp::BM_memcpy[8192] ``` #### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: **Separated build and run steps** Note that we separated the build and run steps in the CI workflow. This is important to speed up the CI workflow and avoiding instrumenting the build step. ### Custom build systems If you need to have full control over your build system, here are guiding steps to take to use codspeed. #### Get the sources Sources are located in the [`codspeed-cpp`](https://github.com/CodSpeedHQ/codspeed-cpp) repository. You can either clone the repository, add it as a submodule or even download the sources as a zip file. #### Build the library Sources of the `google_benchmark` CodSpeed integration library are located in the [`google_benchmark` subdirectory](https://github.com/CodSpeedHQ/codspeed-cpp/tree/main/google_benchmark). 3. Make sure the following pre-processor variables are defined when you build the library When building the library, the tricky part is to make sure google\_benchmark's fork has access to the [`codspeed-core`](https://github.com/CodSpeedHQ/codspeed-cpp/tree/main/core) library. Additionally, the following pre-processor variables must be defined: * `CODPSEED_ENABLED`: if not defined, `google_benchmark` will the same as the upstream library, with no CodSpeed features. * `CODSPEED_SIMULATION`: if running in simulation mode * Note: For versions prior to v2.0.0, use `CODSPEED_INSTRUMENTATION` instead. * `CODSPEED_WALLTIME`: if running in walltime mode * `CODSPEED_ROOT_DIR`: absolute path to the root directory of your project. This is used in the report to display file path relative to your root project If you run into issues integrating CodSpeed's `google_benchmark` library with your project, please reach out and open an issue on the [codspeed-cpp](https://github.com/CodSpeedHQ/codspeed-cpp) repository. # Writing Benchmarks in Go Source: https://codspeed.io/docs/benchmarks/go Create benchmarks for your Go codebase using the `testing` package The Go integration is still in early development, and only some `go test` CLI flags are supported. See the [compatibility section](#compatibility) for more information on how to ensure your benchmarks work with CodSpeed. Additionally, **only the [walltime instrument](/docs/instruments/walltime) is currently supported** If you have any feedback, please reach out to us via [Discord](https://discord.gg/MxpaCfKSqF) or [email our support](mailto:contact@codspeed.io). Integrating CodSpeed into your Go codebase requires **no modification**. You can continue using `go test` and the `testing` package as you normally would. When running your benchmarks in CI with CodSpeed, your benchmarks will automatically be built and the reports will be sent to CodSpeed. ## Creating benchmarks You can just use the `testing` package to write benchmarks in Go. If the benchmarks are working with `go test`, then they will also automatically be detected by CodSpeed without any additional configuration. Here's an example of a simple Fibonacci function benchmark: ```go fib_test.go theme={null} package example import "testing" func BenchmarkFibonacci10(b *testing.B) { for b.Loop() { fib(10) } } ``` **We recommend using [`b.Loop()`](https://pkg.go.dev/testing#B.Loop) as it's more precise and efficient**. Still, `for i := 0; i < b.N; i++` is also supported for backward compatibility (before Go 1.24): ```go theme={null} func BenchmarkFibonacci20(b *testing.B) { for i := 0; i < b.N; i++ { fib(20) } } ``` For a deeper dive into Go benchmarking, see the dedicated guide: An in-depth guide to writing Go benchmarks: sub-benchmarks, parallel benchmarks, benchstat, pprof, and CodSpeed CI integration. ## Testing the benchmarks locally To run the benchmarks with CodSpeed locally, you need to install the `codspeed` runner: ```shellsession title=terminal icon="square-terminal" theme={null} curl -fsSL https://codspeed.io/install.sh | sh ``` You can then run your `go test` command with CodSpeed: ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed run --skip-upload -- go test -bench=. ►►► Running the benchmarks [INFO go_runner] Discovered 1 package [INFO go_runner] Total benchmarks discovered: 2 [INFO go_runner] Found BenchmarkFibonacci10 in "fib_test.go" [INFO go_runner] Found BenchmarkFibonacci20 in "fib_test.go" [INFO go_runner] Generating custom runner for package: example [INFO go_runner] Running benchmarks for package: example Running with CodSpeed instrumentation goos: linux goarch: amd64 cpu: 12th Gen Intel(R) Core(TM) i7-1260P @ 1215.790MHz BenchmarkFibonacci10-16 1348328 361.9 ns/op BenchmarkFibonacci20-16 106713 47947 ns/op PASS [INFO go_runner] Parsed 2 raw results [INFO go_runner] Results written to "/tmp/profile.qPQgi6h0iK.out/results/231603.json" ``` This will print all the benchmarks that can be run with CodSpeed and warnings if some benchmarks are not supported. ## Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: **Incomplete flamegraphs on ARM64?** If your flamegraphs appear incomplete on ARM64 (e.g., on [CodSpeed Macro runners](/docs/features/macro-runners)), try setting the `CODSPEED_PERF_UNWINDING_MODE` environment variable to `fp`: ```yaml {5-6} theme={null} - uses: CodSpeedHQ/action@v5 with: run: go test -bench=. instruments: walltime env: CODSPEED_PERF_UNWINDING_MODE: fp ``` This switches from DWARF-based unwinding to frame-pointer-based unwinding, which produces more reliable call stacks for Go on ARM64. CodSpeed tries to detect this automatically, but it cannot catch every case. ## Recipes ### Sharding benchmarks in parallel CI jobs If your benchmarks are taking too much time to run under the CodSpeed action, you can run them in parallel to speed up the execution. To parallelize your benchmarks, simply add filters to the `go test` command to only run a subset of benchmarks in each job. **Compatibility** We only support the following flags for `go test`: * `-bench` (required) If you run into issues or require certain features, please [open an issue](https://github.com/CodSpeedHQ/codspeed-go/issues) or [join our Discord](https://discord.com/invite/MxpaCfKSqF) to get help. ## Next steps Learn more about the Walltime instrument and how to use it. An in-depth guide to writing Go benchmarks with CodSpeed. Learn more about profiling and how to read flamegraphs. The example GitHub repository for this Gin Gonic API with benchmarks. # Writing Benchmarks in Java Source: https://codspeed.io/docs/benchmarks/java Create benchmarks for your Java codebase using Java Microbenchmark Harness The Java integration is still in early development, and **only the [walltime instrument](/docs/instruments/walltime) is currently supported**. If you have any feedback, please reach out to us via [Discord](https://discord.gg/MxpaCfKSqF) or [email our support](mailto:contact@codspeed.io). Integrating CodSpeed into your Java project works through a fork of [JMH](https://github.com/openjdk/jmh) (Java Microbenchmark Harness). You write standard JMH benchmarks and swap in the CodSpeed JMH fork as a dependency. When running in CI with CodSpeed, the results are automatically collected and reported. ## Installation CodSpeed provides a fork of JMH that collects walltime results and sends them to CodSpeed. Add the CodSpeed JMH fork as a Git submodule: ```bash theme={null} git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm ``` Then include it as a composite build in your `settings.gradle.kts`, with dependency substitution to redirect JMH dependencies to the CodSpeed fork: ```kotlin settings.gradle.kts theme={null} includeBuild("third-party/codspeed-jvm/jmh-fork") { // [!code ++] dependencySubstitution { // [!code ++] substitute(module("org.openjdk.jmh:jmh-core")) // [!code ++] .using(project(":jmh-core")) // [!code ++] substitute(module("org.openjdk.jmh:jmh-generator-annprocess")) // [!code ++] .using(project(":jmh-generator-annprocess")) // [!code ++] } // [!code ++] } // [!code ++] ``` We recommend using the [JMH Gradle Plugin](https://github.com/melix/jmh-gradle-plugin) as it handles benchmark compilation and provides the `jmh` task. Add it to your `build.gradle.kts`: ```kotlin build.gradle.kts theme={null} plugins { java id("me.champeau.jmh") version "0.7.2"// [!code ++] } ``` Add the CodSpeed JMH fork as a Git submodule and publish it to your local Maven repository: ```bash theme={null} git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm cd third-party/codspeed-jvm ./gradlew -p jmh-fork publishToMavenLocal ``` Then replace the JMH dependencies in your `pom.xml` with the CodSpeed fork. Maven resolves from `~/.m2/repository` by default. The repository entry below is only necessary if your local repository is at a non-default location: ```xml pom.xml theme={null} // [!code ++] // [!code ++] local // [!code ++] file://${user.home}/.m2/repository // [!code ++] // [!code ++] // [!code ++] org.openjdk.jmh// [!code --] 1.37// [!code --] io.codspeed.jmh// [!code ++] 0.1.0// [!code ++] jmh-core org.openjdk.jmh// [!code --] 1.37// [!code --] io.codspeed.jmh// [!code ++] 0.1.0// [!code ++] jmh-generator-annprocess ``` Your Maven project must be configured to produce an executable benchmark JAR. See the [official JMH setup guide](https://github.com/openjdk/jmh?tab=readme-ov-file#preferred-usage-command-line) for instructions. We're planning to publish to Maven Central in the future. If you need this, please reach out via [Discord](https://discord.gg/MxpaCfKSqF) or [email](mailto:contact@codspeed.io). ## Creating benchmarks Write your benchmarks using standard JMH annotations: ```java FibBenchmark.java theme={null} package bench; import org.openjdk.jmh.annotations.*; public class FibBenchmark { @Benchmark public long fib() { return fib(30); } private static long fib(int n) { if (n <= 1) return n; return fib(n - 1) + fib(n - 2); } } ``` JMH benchmarks must return their result or use `Blackhole.consume()` to prevent the JVM from eliminating dead code. All examples on this page return the computed value. For an in-depth tutorial on JMH, see the [How to Benchmark Java with JMH](/docs/guides/how-to-benchmark-java-with-jmh) guide. ## Testing the benchmarks locally To run the benchmarks with CodSpeed locally, first install the `codspeed` runner: ```bash theme={null} curl -fsSL https://codspeed.io/install.sh | sh ``` Then run your benchmarks with CodSpeed: ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed run --mode walltime -- ./gradlew jmh ►►► Running the benchmarks # JMH version: 0.1.0 # VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7 # Warmup: 5 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Benchmark: bench.FibBenchmark.fib # Run progress: 0.00% complete, ETA 00:01:40 # Warmup Iteration 1: 112.541 ns/op ... Benchmark Mode Cnt Score Error Units FibBenchmark.fib avgt 5 110.342 ± 2.153 ns/op ``` ```shellsession title=terminal icon="square-terminal" theme={null} $ mvn package -q $ codspeed run --mode walltime -- java -jar target/benchmarks.jar ►►► Running the benchmarks # JMH version: 0.1.0 # VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7 # Warmup: 5 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Benchmark: bench.FibBenchmark.fib # Run progress: 0.00% complete, ETA 00:01:40 # Warmup Iteration 1: 112.541 ns/op ... Benchmark Mode Cnt Score Error Units FibBenchmark.fib avgt 5 110.342 ± 2.153 ns/op ``` ## Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: ## Advanced usage JMH provides many features for writing expressive benchmarks. Below is a selection that can be useful in CodSpeed benchmarks. ### Parameterized benchmarks Use `@Param` to run the same benchmark with different input values: ```java ParamBenchmark.java theme={null} package bench; import org.openjdk.jmh.annotations.*; @State(Scope.Benchmark) public class ParamBenchmark { @Param({"10", "20", "30"}) int n; @Benchmark public long fib() { return fib(n); } private static long fib(int n) { if (n <= 1) return n; return fib(n - 1) + fib(n - 2); } } ``` ### Shared state Use `@State` to share setup logic across benchmarks and control the scope of the state object: ```java StateBenchmark.java theme={null} package bench; import org.openjdk.jmh.annotations.*; import java.util.ArrayList; import java.util.List; @State(Scope.Benchmark) public class StateBenchmark { List list; @Setup public void setup() { list = new ArrayList<>(); for (int i = 0; i < 1000; i++) { list.add(i); } } @Benchmark public int sum() { return list.stream().mapToInt(Integer::intValue).sum(); } } ``` For a deeper dive into JMH, see the dedicated guide: An in-depth guide to writing JMH benchmarks: project setup, annotations, parameters, common pitfalls, and CodSpeed CI integration. ## Compatibility * **JDK 21 or later** is required. * All standard JMH annotations are supported. * Only the [Walltime instrument](/docs/instruments/walltime) is supported * CodSpeed uses a custom mode for all benchmarks to collect statistically significant results. Any `@BenchmarkMode` annotations in your code are ignored. If you run into issues or require certain features, please [open an issue](https://github.com/CodSpeedHQ/codspeed-jvm/issues) or [join our Discord](https://discord.com/invite/MxpaCfKSqF) to get help. ## Next steps The CodSpeed JVM repository with example JMH benchmarks. An in-depth guide to writing JMH benchmarks with CodSpeed. Learn more about the Walltime instrument and how to use it. Learn more about profiling and how to read flame graphs. # Writing benchmarks with benchmark.js Source: https://codspeed.io/docs/benchmarks/nodejs/benchmarkjs ## Installation First install the plugin [`@codspeed/benchmark.js-plugin`](https://www.npmjs.com/package/@codspeed/benchmark.js-plugin) and `benchmark` (if not already installed): ## Usage ### Creating benchmarks Let's create a fibonacci function and benchmark it with benchmark.js and the CodSpeed plugin: ```js benches/bench.mjs {11} theme={null} import Benchmark from "benchmark"; import { withCodSpeed } from "@codspeed/benchmark.js-plugin"; function fibonacci(n) { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } const suite = withCodSpeed(new Benchmark.Suite()); suite .add("fibonacci10", () => { fibonacci(10); }) .add("fibonacci15", () => { fibonacci(15); }) .on("cycle", function (event: Benchmark.Event) { console.log(String(event.target)); }) .run(); ``` Noticed the `.mjs` extension? This is because we're using the ESM module format. Saving our file with the `.js` extension would have worked as well, but we would have needed to add `"type": "module"` to our `package.json` file to instruct Node.js to use the ESM module format. If you're working with CommonJS modules, you can totally use the `require` syntax for importing the libraries. ```typescript benches/bench.ts {11} theme={null} import { withCodSpeed } from "@codspeed/benchmark.js-plugin"; import Benchmark from "benchmark"; function fibonacci(n: number): number { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } const suite = withCodSpeed(new Benchmark.Suite()); suite .add("fibonacci10", () => { fibonacci(10); }) .add("fibonacci15", () => { fibonacci(15); }) .on("cycle", function (event: Benchmark.Event) { console.log(String(event.target)); }) .run(); ``` Here, a few things are happening: * We create a simple recursive fibonacci function. * We create a new `Benchmark.Suite` instance with CodSpeed support by using the **`withCodSpeed`** helper. This step is **critical** to enable CodSpeed on your benchmarks. * We add two benchmarks to the suite and launch it, benching our `fibonacci` function with 10 and 15. ### Testing the benchmarks locally Now, we can run our benchmarks locally to make sure everything is working as expected: **TypeScript runner** To run the `.ts` file directly, we recommend using [`esbuild-register`](https://github.com/egoist/esbuild-register). It allows running TypeScript & ESM files directly with Node.js. ```shellsession title=terminal icon="square-terminal" theme={null} $ node -r esbuild/register benches/bench.ts [CodSpeed] 2 benches detected but no instrumentation found [CodSpeed] falling back to benchmark.js fibonacci10 x 2,155,187 ops/sec ±0.50% (96 runs sampled) fibonacci15 x 194,742 ops/sec ±0.48% (95 runs sampled) ``` ```shellsession title=terminal icon="square-terminal" theme={null} $ node benches/bench.mjs [CodSpeed] 2 benches detected but no instrumentation found [CodSpeed] falling back to benchmark.js fibonacci10 x 2,155,187 ops/sec ±0.50% (96 runs sampled) fibonacci15 x 194,742 ops/sec ±0.48% (95 runs sampled) ``` And... Congrats🎉, CodSpeed is installed in your benchmarking suite! Locally, CodSpeed will fall back to benchmark.js since the CPU simulation is only available in the CI environment for now. You can now [run those benchmarks in your CI](#running-the-benchmarks-in-your-ci) to get consistent performance measurements. ### Integrating into a bigger project, multiple benchmark files Often time you will not be writing your benchmarks in a single file. Indeed, it can become quite difficult to maintain a single file with all your benchmarks as your project grows. You can find the source code for the following example in the [examples of the `codspeed-node` repository](https://github.com/CodSpeedHQ/codspeed-node/tree/main/examples). There are multiple examples available, for CJS, ESM, JavaScript, and TypeScript. For these kind of situations, we recommend the following approach. Let's say you have a file structure like this, in a project with **TypeScript**: ```text file-structure theme={null} . ├── bench │ ├── fibo.bench.ts │ ├── foobarbaz.bench.ts │ └── index.bench.ts ├── package.json ├── src │ ├── fibonacci.ts │ └── foobarbaz.ts └── tsconfig.json ``` * The `src` directory contains the source code of the project. Here we have two files, `fibonacci.ts` and `foobarbaz.ts`. * The `bench` directory contains the benchmarks for the project. There is a file for each source file that defines benchmarks for it. * The `bench/index.bench.ts` file is the entry point for the benchmarks. It imports all the other benchmark files and runs them. ```typescript bench/fibo.bench.ts theme={null} import type { WithCodSpeedSuite } from "@codspeed/benchmark.js-plugin"; import { iterativeFibonacci } from "../../src/fibonacci"; export function registerFiboBenchmarks(suite: WithCodSpeedSuite) { suite .add("test_iterative_fibo_10", () => { iterativeFibonacci(10); }) .add("test_iterative_fibo_100", () => { iterativeFibonacci(100); }); } ``` Here we define a function that takes an instance of `Bench` as a parameter and then adds some benchmarks to it. This will allow us to add benchmarks to the same suite from multiple files. ```typescript bench/index.bench.ts theme={null} import { withCodSpeed } from "@codspeed/benchmark.js-plugin"; import Benchmark from "benchmark"; import { registerFiboBenchmarks } from "./fibo.bench"; import { registerFoobarbazBenchmarks } from "./foobarbaz.bench"; export const suite = withCodSpeed(new Benchmark.Suite()); (async () => { registerFiboBenchmarks(suite); registerFoobarbazBenchmarks(suite); suite.on("cycle", function (event: Benchmark.Event) { console.log(String(event.target)); }); await suite.run({ async: true }); })(); ``` Here all the functions registering benchmarks are executed to import all the benchmarks from the different files. To run the benchmarks, use the following command: ```sh theme={null} node -r esbuild-register bench/index.bench.ts ``` Check out the full for this example: [`with-typescript-cjs` in the `codspeed-node` repository](https://github.com/CodSpeedHQ/codspeed-node/tree/main/examples/with-typescript-cjs). ### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). # Writing Benchmarks in JavaScript and TypeScript Source: https://codspeed.io/docs/benchmarks/nodejs/overview There are multiple ways to integrate CodSpeed with your JS/TS codebase: The most convenient way to run your Node.js benchmarks A lightweight benchmarking library with a simple API The legacy benchmarking library for JavaScript We recommend using the `vitest` plugin as it is the easiest to use and has the most features. If you are already using `benchmark.js` or `tinybench`, check out their respective plugins as they require only a small wrapping to work with CodSpeed. To benchmark a user flow end-to-end, use the Playwright integration: Drive an app through Playwright and measure real flows. ## Supported Node.js versions Each Node.js version is supported by a range of `@codspeed/*` packages versions, which are all released under the same version number: | Node.js version | `@codspeed/*` packages versions | | :-------------- | :------------------------------ | | 24 | `>=6.0.0` | | 22 | `>=6.0.0` | | 20 | `>=2.0.0 <6.0.0` | | 18 | `>=2.0.0 <6.0.0` | | 16 | `>=1.0.0 <5.0.0` | # Profiling apps with Playwright Source: https://codspeed.io/docs/benchmarks/nodejs/playwright Drive an app through Playwright and report measured user flows to CodSpeed. The `@codspeed/playwright-plugin` integration currently supports only the [walltime instrument](/docs/instruments/walltime). CPU Simulation is not available. [`@codspeed/playwright-plugin`](https://www.npmjs.com/package/@codspeed/playwright-plugin) is the CodSpeed integration for [Playwright](https://playwright.dev). It runs a user-defined flow against a target application, measures the time spent inside that flow, and reports it to CodSpeed. The flow itself is plain Playwright code, so anything Playwright can drive can be benchmarked. Today the plugin supports [Electron](https://www.electronjs.org) apps as a target. Browser-based targets (existing dev servers, static builds, hosted URLs) are on the roadmap and will be added under the same `bench` API. ## Installation Install the plugin alongside `playwright`: ## Example usage with Electron Build your Electron app first so the main entrypoint exists (e.g., `out/main/index.js`), then declare a benchmark with `target.kind` set to `"electron"`: ```ts bench/inbox.bench.ts theme={null} import { bench } from "@codspeed/playwright-plugin"; import path from "node:path"; bench( "inbox-search", async ({ page }) => { await page.fill("#search", "quarterly report"); await page.waitForSelector("#results"); }, { target: { kind: "electron", appPath: path.resolve("out/main/index.js"), }, beforeRound: async ({ page }) => { await page.waitForSelector("#main:not(.loading)"); }, rounds: 5, } ); ``` For each round, the plugin launches Electron with the provided main entrypoint, waits for the first window, runs `beforeRound`, measures `fn`, runs `afterRound`, then closes the app. For a more complete example, have a look at the [example benchmark](https://github.com/CodSpeedHQ/codspeed-node/tree/main/examples/with-electron-and-walltime) included in the codspeed-node repository. ## API The plugin exposes a single `bench` function. Its shape is target-agnostic: ```ts theme={null} import { bench } from "@codspeed/playwright-plugin"; bench(name, fn, options); ``` Identifier of the benchmark, used by CodSpeed to track it across runs. The function whose execution time is measured. Receives a Playwright [`Page`](https://playwright.dev/docs/api/class-page) bound to the target. Everything that runs inside `fn` counts toward the reported timing. Target configuration and benchmark settings, detailed below. ### Options Discriminated union describing what to drive. The `kind` field selects the target; the remaining fields are specific to that kind. Current variants: [`{ kind: "electron", ... }`](#electron-target-options). More will be added without breaking existing call sites. Number of measurement rounds. Can be overridden at runtime via the `CODSPEED_ROUNDS` environment variable. Runs before each round, after the target is ready. Use it to bring the app to a ready state. Not measured. Runs after each round, before the target is torn down. Not measured. ### Electron target options Selects the Electron target. Absolute path to the Electron main entrypoint, e.g., `out/main/index.js`. Extra CLI flags forwarded to the Electron process. Working directory for the Electron process. Also the directory `electron` is resolved from when `target.electronExecutablePath` is not set. Absolute path to the Electron binary. Only set this to override the default resolution. ## Running the benchmarks locally With node 24+, you can run typescript files directly: ```shellsession title=terminal icon="square-terminal" theme={null} $ node bench/inbox.bench.ts [CodSpeed] [round 1/5] 42.13 ms [CodSpeed] [round 2/5] 41.78 ms [CodSpeed] [round 3/5] 42.05 ms [CodSpeed] [round 4/5] 41.92 ms [CodSpeed] [round 5/5] 42.21 ms ``` Locally, `bench` runs the app and prints per-round timings to the terminal. Results are uploaded to CodSpeed only when running in the [CI environment](#running-the-benchmarks-in-your-ci) or when using the [CodSpeed CLI](/docs/cli#running-benchmarks). ## Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: Two pieces of this workflow are not optional: * The job must run on a [CodSpeed Macro runner](/docs/features/macro-runners) (`runs-on: codspeed-macro`). Walltime measurements are not stable on shared GitHub-hosted runners. * `CODSPEED_WALLTIME_PROFILER` must be set to `samply` on the benchmark step. The plugin currently only supports the [`samply`](https://github.com/mstange/samply) profiler; without it, the run will fail to produce [profiling information](/docs/features/profiling). Electron needs a display to render its window. On headless CI runners, wrap the benchmark command with [`xvfb-run`](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml) or install a virtual framebuffer, otherwise the app will fail to start. # Writing benchmarks with tinybench Source: https://codspeed.io/docs/benchmarks/nodejs/tinybench ## Installation First install the plugin [`@codspeed/tinybench-plugin`](https://www.npmjs.com/package/@codspeed/tinybench-plugin) and `tinybench` (if not already installed): The CodSpeed plugin now requires `tinybench` v4 and above. Note that tinybench v4 has dropped support for CommonJS modules, so ESM is now required. ## Usage ### Creating benchmarks Let's create a fibonacci function and benchmark it with tinybench and the CodSpeed plugin: ```typescript benches/bench.ts {11} theme={null} import { withCodSpeed } from "@codspeed/tinybench-plugin"; import { Bench } from "tinybench"; function fibonacci(n: number): number { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } const bench = withCodSpeed(new Bench()); bench .add("fibonacci10", () => { fibonacci(10); }) .add("fibonacci15", () => { fibonacci(15); }); await bench.run(); console.table(bench.table()); ``` ```js benches/bench.mjs {11} theme={null} import { Bench } from "tinybench"; import { withCodSpeed } from "@codspeed/tinybench-plugin"; function fibonacci(n) { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } const bench = withCodSpeed(new Bench()); bench .add("fibonacci10", () => { fibonacci(10); }) .add("fibonacci15", () => { fibonacci(15); }); await bench.run(); console.table(bench.table()); ``` Noticed the `.mjs` extension? This is because we're using the ESM module format. Saving our file with the `.js` extension would have worked as well, but we would have needed to add `"type": "module"` to our `package.json` file to instruct Node.js to use the ESM module format. Here, a few things are happening: * We create a simple recursive fibonacci function. * We create a new `Bench` instance with CodSpeed support by using the **`withCodSpeed`** helper. This step is **critical** to enable CodSpeed on your benchmarks. * We add two benchmarks to the suite and launch it, benching our `fibonacci` function for 10 and 15. ### Testing the benchmarks locally Now, we can run our benchmarks locally to make sure everything is working as expected: **TypeScript runner** To run the `.ts` file directly, we recommend using [`esbuild-register`](https://github.com/egoist/esbuild-register). It allows running TypeScript & ESM files directly with Node.js. ```shellsession title=terminal icon="square-terminal" theme={null} $ node -r esbuild/register benches/bench.ts ┌─────────┬───────────────┬───────────────────┬──────────┐ │ (index) │ Task Name │ Average Time (ns) │ Margin │ ├─────────┼───────────────┼───────────────────┼──────────┤ │ 0 │ 'fibonacci10' │ 552.4139857896414 │ '±0.18%' │ │ 1 │ 'fibonacci15' │ 5633.276191749634 │ '±0.14%' │ └─────────┴───────────────┴───────────────────┴──────────┘ ``` ```shellsession title=terminal icon="square-terminal" theme={null} $ node benches/bench.mjs ┌─────────┬───────────────┬───────────────────┬──────────┐ │ (index) │ Task Name │ Average Time (ns) │ Margin │ ├─────────┼───────────────┼───────────────────┼──────────┤ │ 0 │ 'fibonacci10' │ 552.4139857896414 │ '±0.18%' │ │ 1 │ 'fibonacci15' │ 5633.276191749634 │ '±0.14%' │ └─────────┴───────────────┴───────────────────┴──────────┘ ``` And... Congrats🎉, CodSpeed is installed in your benchmarking suite! When not used in the CI environment or with the [CLI](https://github.com/CodSpeedHQ/codspeed), CodSpeed will fallback to using the default `tinybench`. You can now [run those benchmarks in your CI](#running-the-benchmarks-in-your-ci) to get consistent performance measurements. ### Integrating into a bigger project, multiple benchmark files Often time you will not be writing your benchmarks in a single file. Indeed, it can become quite difficult to maintain a single file with all your benchmarks as your project grows. You can find the source code for the following example in the [examples of the `codspeed-node` repository](https://github.com/CodSpeedHQ/codspeed-node/tree/main/examples). There are multiple examples available, for CJS, ESM, JavaScript, and TypeScript. For these kind of situations, we recommend the following approach. Let's say you have a file structure like this, in a project with **TypeScript**: ```text file-structure theme={null} . ├── bench │ ├── fibo.bench.ts │ ├── foobarbaz.bench.ts │ └── index.bench.ts ├── package.json ├── src │ ├── fibonacci.ts │ └── foobarbaz.ts └── tsconfig.json ``` * The `src` directory contains the source code of the project. Here we have two files, `fibonacci.ts` and `foobarbaz.ts`. * The `bench` directory contains the benchmarks for the project. There is a file for each source file that defines benchmarks for it. * The `bench/index.bench.ts` file is the entry point for the benchmarks. It imports all the other benchmark files and runs them. ```typescript bench/fibo.bench.ts theme={null} import { Bench } from "tinybench"; import { iterativeFibonacci } from "../../src/fibonacci"; export function registerFiboBenchmarks(bench: Bench) { bench .add("test_iterative_fibo_10", () => { iterativeFibonacci(10); }) .add("test_iterative_fibo_100", () => { iterativeFibonacci(100); }); } ``` Here we define a function that takes an instance of `Bench` as a parameter and then adds some benchmarks to it. This will allow us to add benchmarks to the same suite from multiple files. ```typescript bench/index.bench.ts theme={null} import { withCodSpeed } from "@codspeed/tinybench-plugin"; import { Bench } from "tinybench"; import { registerFiboBenchmarks } from "./fibo.bench"; import { registerFoobarbazBenchmarks } from "./foobarbaz.bench"; export const bench = withCodSpeed(new Bench()); (async () => { registerFiboBenchmarks(bench); registerFoobarbazBenchmarks(bench); await bench.run(); console.table(bench.table()); })(); ``` Here all the functions registering benchmarks are executed to import all the benchmarks from the different files. To run the benchmarks, use the following command: ```sh theme={null} node -r esbuild-register bench/index.bench.ts ``` Check out the full for this example: [`with-typescript-cjs` in the `codspeed-node` repository](https://github.com/CodSpeedHQ/codspeed-node/tree/main/examples/with-typescript-cjs). ### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). ## Limitations ### Async code execution profiling [Execution profiles](/docs/features/profiling) for async code can sometimes be unreliable, as profiling tools may lose stack trace information due to the event loop. If your code is fully sync, consider using tinybench's `runSync` as an entrypoint to improve accuracy. # Writing benchmarks with vitest-bench Source: https://codspeed.io/docs/benchmarks/nodejs/vitest ## Installation First install the plugin [`@codspeed/vitest-plugin`](https://www.npmjs.com/package/@codspeed/vitest-plugin) and `vitest` (if not already installed): The CodSpeed plugin is only compatible with `vitest` [v3.2 and above](https://www.npmjs.com/package/vitest/v/3.2.4). ## Usage ### Creating benchmarks Let's create a fibonacci function and benchmark it with vitest and the CodSpeed plugin: ```typescript src/fibo.bench.ts theme={null} import { bench, describe } from "vitest"; function fibonacci(n: number): number { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } describe("fibo", () => { bench("fibo 10", () => { fibonacci(10); }); bench("fibo 15", () => { fibonacci(15); }); }); ``` ```js src/fibo.bench.js theme={null} import { bench, describe } from "vitest"; function fibonacci(n) { if (n < 2) { return n; } return fibonacci(n - 1) + fibonacci(n - 2); } describe("fibo", () => { bench("fibo 10", () => { fibonacci(10); }); bench("fibo 15", () => { fibonacci(15); }); }); ``` Here, a few things are happening: * We create a simple recursive fibonacci function. * We create a new `vitest` suite `"fibo"` with two benchmarks, benching our `fibonacci` function for 10 and 15. Create or update the `vitest.config.ts` file to use the CodSpeed runner: ```ts vitest.config.ts theme={null} import codspeedPlugin from "@codspeed/vitest-plugin"; import { defineConfig } from "vitest/config"; export default defineConfig({ plugins: [codspeedPlugin()], // ... }); ``` ### Testing the benchmarks locally Now, we can run our benchmarks locally to make sure everything is working as expected: ```sh npm theme={null} npx vitest bench --run ``` ```sh yarn theme={null} yarn vitest bench --run ``` ```sh pnpm theme={null} pnpm vitest bench --run ``` This will run the benchmarks and output the results in the terminal (for example, with `pnpm`): ```shellsession title=terminal icon="square-terminal" theme={null} $ pnpm vitest bench --run RUN v3.2.4 [CodSpeed] @codspeed/vitest-plugin v5.0.1 - setup ✓ benches/flat.bench.ts (2) 1521ms ✓ fibo (2) 1520ms name min max mean rme samples · fibo 10 0.0005 1.0279 0.0006 ±0.50% 880344 fastest · fibo 15 0.0052 0.0594 0.0057 ±0.08% 86967 BENCH Summary fibo 10 - benches/flat.bench.ts > fibo 10.12x faster than fibo 15 ``` **Troubleshooting errors** If you encounter an error like `Failed to resolve "@codspeed/vitest-plugin" from "vitest.config.ts"`, you might need to rename the `vitest.config.ts` file to `vitest.config.mts`. You can find more information in the [Vite documentation on importing ESM package](https://vitejs.dev/guide/troubleshooting.html#this-package-is-esm-only). And... Congrats🎉, CodSpeed is installed in your benchmarking suite! When not used in the CI environment or with the [CLI](https://github.com/CodSpeedHQ/codspeed), CodSpeed will fallback to using the default `vitest`. You can now [run those benchmarks in your CI](#running-the-benchmarks-in-your-ci) to get consistent performance measurements. ### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: ## Recipes ### Running benchmarks in parallel CI jobs To parallelize your benchmarks, you can use the [`shard`](https://vitest.dev/guide/cli.html#shard) options from vitest. For example with `pnpm` on Github Actions: **Same benchmark with different variations** For now, you cannot run the same benchmarks several times within the same run. If the same benchmark is run multiple times, you will receive the following comment on your pull request: Multiple Benchmark Variations Error Message

Multiple Benchmark Variations Error Message

Learn more about [benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks). #### Running benchmarks in parallel processes If you cannot split your benchmarks across multiple CI jobs, you can split them across multiple processes in the same job. We only recommend this as an alternative to the parallel CI jobs setup. `pytest-codspeed` is compatible with [`pytest-xdist`](https://pypi.org/project/pytest-xdist/), a `pytest` plugin allowing to distribute the execution across multiple processes. You can simply enable the `pytest-xdist` plugin on top of `pytest-codspeed`. This will allow you to run your benchmarks in parallel using multiple processes. First, install `pytest-xdist` as a development dependency: Then, you can run your benchmarks in parallel with the `pytest-xdist` flag: ```sh theme={null} pytest tests/ --codspeed -n auto ``` The change in the CI workflow would look like this: ```yaml .github/workflows/codspeed.yml {5} theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation run: pytest tests/ --codspeed -n auto ``` To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). ### Usage with Nox It's possible to use `pytest-codspeed` with [`Nox`](https://nox.thea.codes/en/stable/), a Python automation tool that allows you to automate the execution of Python code across multiple environments. Here is an example configuration file to run benchmarks with `pytest-codspeed` using `Nox`: ```python noxfile.py theme={null} import nox @nox.session def codspeed(session): session.install('pytest') session.install('pytest-codspeed') session.run('pytest', '--codspeed') ``` You can then run the benchmarks: ```sh theme={null} nox --sessions codspeed ``` To use it with Github Actions, you can use the following workflow: Splitting the virtualenv installation and the execution of the benchmarks is optional. Though this allows to speed up the execution of the benchmarks since the dependencies will be installed or compiled without the CPU simulation enabled. # Writing Benchmarks with bencher (libtest) Source: https://codspeed.io/docs/benchmarks/rust/bencher Using the bencher (libtest) compatibility layer for CodSpeed ## Installation For all Rust integrations, you will need [the `cargo-codspeed` command](/docs/benchmarks/rust/overview#cargo-codspeed) to build and run your CodSpeed benchmarks Install the [`bencher` compatibility layer](https://crates.io/crates/codspeed-bencher-compat): ```sh theme={null} cargo add --dev codspeed-bencher-compat --rename bencher ``` Or directly change your `Cargo.toml` if you already have `bencher` installed: ```toml theme={null} [dev-dependencies] bencher = { package = "codspeed-bencher-compat", version = "*" } ``` If you prefer, you can also install `codspeed-bencher-compat` as is and change your imports to use this new crate name. This will install the `codspeed-bencher-compat` crate and rename it to `bencher` in your `Cargo.toml`. This way, you can keep your existing imports and the compatibility layer will take care of the rest. The compatibility layer is a passthrough when not using `cargo codspeed`, running `cargo bench` will behave exactly as with the default `bencher` crate. ## Usage ### Creating benchmarks Let's start with the example from the [`bencher` documentation](https://docs.rs/bencher/latest/bencher/), creating a benchmark suite for 2 simple functions: ```rust benches/example.rs theme={null} use bencher::{benchmark_group, benchmark_main, Bencher}; fn a(bench: &mut Bencher) { bench.iter(|| { (0..1000).fold(0, |x, y| x + y) }) } fn b(bench: &mut Bencher) { const N: usize = 1024; bench.iter(|| { vec![0u8; N] }); bench.bytes = N as u64; } benchmark_group!(benches, a, b); benchmark_main!(benches); ``` The last step in creating the Bencher benchmark is to add the new benchmark target in your `Cargo.toml`: ```toml Cargo.toml theme={null} [[bench]] name = "example" harness = false ``` And that's it! You can now run your benchmark suite with CodSpeed ### Testing the benchmarks locally ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed build Finished release [optimized] target(s) in 0.12s Finished built 1 benchmark suite(s) $ cargo codspeed run Collected 1 benchmark suite(s) to run Running example Using codspeed-bencher-compat v1.0.0 compatibility layer NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/example.rs::a (group: benches) Checked: benches/example.rs::b (group: benches) Done running bencher_example Finished running 1 benchmark suite(s) ``` Congrats! 🎉 You can now [run those benchmark in your CI](#running-the-benchmarks-in-your-ci) to get the actual performance measurements 👇. Use `--measurement-mode` / `-m` to select the CodSpeed instrument: * **`simulation`** (default): Runs benchmarks once on a [simulated CPU](/docs/instruments/cpu) for consistent measurements. * **`walltime`**: Measures [wall-clock time](/docs/instruments/walltime) for real-world scenarios. * **`memory`**: Benchmarks are run once using [memory profiling](/docs/instruments/memory) to track heap allocations and memory usage. See the [`cargo-codspeed` reference](/docs/reference/codspeed-rust/cargo-codspeed#the-measurement-mode-flag) for more information. ### Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: ## Recipes ### Running benchmarks in parallel CI jobs With Rust, if you use multiple packages, a first sharding optimization is to split your benchmarks across these packages. For example, using Github Actions: It is not required to pass a `-p` flag as only the benchmarks built by `cargo codspeed build` will be run. For more information about multiple packages, check [the cargo-codspeed docs](/docs/benchmarks/rust/). **Same benchmark with different variations** For now, you cannot run the same benchmarks several times within the same run. If the same benchmark is run multiple times, you will receive the following comment on your pull request: Multiple Benchmark Variations Error Message

Learn more about [benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks). To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). # Writing Benchmarks in Rust Source: https://codspeed.io/docs/benchmarks/rust/index Learn how to write benchmarks and measure the performance of your Rust code. ## Supported Benchmarking Crates CodSpeed offers compatibility layers for several popular benchmarking crates: The most convenient way to run your Rust benchmarks A benchmarking crate inspired by the criterion haskell library The libtest (unstable) benchmark runner You should use the [`divan` benchmarking framework](/docs/benchmarks/rust/divan) due to its [extensive features](/docs/benchmarks/rust/divan#advanced-usage), such as running type-generic benchmarks. Its popularity is growing rapidly within the Rust ecosystem. If you're already using [`criterion.rs`](/docs/benchmarks/rust/criterion) or [`bencher`](/docs/benchmarks/rust/bencher), consider their respective plugins, as they require minimal adjustments to work with CodSpeed. ## How does CodSpeed work with Rust benchmarks? Rust, being a compiled language, has CodSpeed integrations that differ from those for interpreted languages. The CodSpeed benchmarking process for Rust occurs at both build time and runtime. To facilitate this, CodSpeed provides: 1. A [`cargo-codspeed` cargo subcommand](./#cargo-codspeed): used regardless of the benchmarking crate. 2. [Compatibility layers](./#benchmarking-crates) for popular benchmarking crates: chose the appropriate one based on your project's needs. ## `cargo-codspeed` To integrate CodSpeed with your Rust codebase, use the `cargo` subcommand: [cargo-codspeed](https://crates.io/crates/cargo-codspeed). This tool allows you to run CodSpeed benchmarks without modifying the behavior of the standard `cargo bench` command. Creating benchmarks with `cargo-codspeed` is the same as with the supported APIs. So if you already have benchmarks written with one of these, only a minor import change is required. Due to a [limitation of cargo](https://github.com/rust-lang/cargo/issues/5376), we currently do not support `build.rustflags` in `.cargo/config.toml` file when using `cargo-codspeed`. However, you can use `[target.'cfg(all())']` instead of `[build]` as a workaround which does the same and is compatible. ```toml theme={null} [build] // [!code --] [target.'cfg(all())'] // [!code ++] rustflags = ["--cfg", "your_feature_flag"] ``` ### Installation To check your benchmarks with CodSpeed, you first need to install the `cargo-codspeed` CLI tool: ```sh theme={null} cargo install cargo-codspeed --locked ``` This tool can then be used directly within cargo: ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed Cargo extension to build & run your codspeed benchmarks Usage: cargo codspeed Commands: build Build the benchmarks run Run the previously built benchmarks Options: -h, --help Print help information -V, --version Print version information ``` ### Usage No matter which benchmarking crate you're using, the `cargo-codspeed` command is used to help you build and run the benchmark in a CodSpeed environment. ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed build Finished release [optimized] target(s) in 0.12s Finished built 1 benchmark suite(s) $ cargo codspeed run Collected 1 benchmark suite(s) to run Running example Using codspeed-bencher-compat v1.0.0 compatibility layer NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/example.rs::a (group: benches) Checked: benches/example.rs::b (group: benches) Done running bencher_example Finished running 1 benchmark suite(s) ``` Use `--measurement-mode` / `-m` to select the CodSpeed instrument: * **`simulation`** (default): Runs benchmarks once on a [simulated CPU](/docs/instruments/cpu) for consistent measurements. * **`walltime`**: Measures [wall-clock time](/docs/instruments/walltime) for real-world scenarios. * **`memory`**: Benchmarks are run once using [memory profiling](/docs/instruments/memory) to track heap allocations and memory usage. See the [`cargo-codspeed` reference](/docs/reference/codspeed-rust/cargo-codspeed#the-measurement-mode-flag) for more information. #### Advanced build options By default, `cargo codspeed build` will build all the benchmark executables of your workspace. But you can also be more specific with the following options: **Cargo Workspaces** If you're using CodSpeed within a workspace you can use the `-p` flag to specify the crate to run the build command on: ```sh theme={null} cargo codspeed build -p my_package ``` **Build only specific benchmark executables** With the following folder structure: ```plaintext theme={null} benches/ ├── bench1.rs └── bench2.rs ``` To build only `bench1`, you can pass its name as the `--bench` flag: ```sh theme={null} cargo codspeed build --bench bench1 # Repeat this argument to build multiple benchmark executables cargo codspeed build --bench bench1 --bench bench2 ``` **Feature flags** If you're using feature flags in your benchmark suite, you can use the `--features` flag to specify the features to enable: ```sh theme={null} cargo codspeed build --features my_feature ``` #### Advanced run options By default, `cargo codspeed run` will run all the **built** benchmarks (of the latest `cargo codspeed build ...` command you ran). To run only a subset of the built benchmarks, you can do the following: ```sh theme={null} # Run all the benchmark executables of the `my_package` crate cargo codspeed run -p my_package # Run only the `bench1` benchmark executable cargo codspeed run --bench bench1 # Run only benches containing `foo` in their name cargo codspeed run foo # Run only benches matching the `foo.*bar` regex cargo codspeed run "foo.*bar" # You can combine these options cargo codspeed run -p my_package --bench bench1 foo ``` #### Building multiple instruments When using multiple measurement modes, you need to build your benchmarks with all the required instruments. Use the `-m` flag to specify each mode: ```sh theme={null} # Build with both simulation and memory instruments cargo codspeed build -m memory -m simulation ``` The resulting binaries are compatible across these modes, so you only need to build once. At runtime, `cargo codspeed run` will execute the benchmarks for each mode specified in your CI workflow. To combine measurement modes like simulation and memory, check out the documentation on [running multiple instruments serially](/docs/integrations/ci/github-actions/configuration#running-multiple-instruments-serially). ## Preventing compiler optimizations `cargo codspeed build` compiles in release mode, so `rustc` removes any computation whose result is never used, and CodSpeed reports the benchmark as [optimized out](/docs/troubleshooting#optimized-out-benchmarks). Return the measured value: it keeps the computation alive and leaves its `Drop` out of the measurement. Wrap constant inputs in `black_box`, re-exported by every compatibility layer, to stop compile-time evaluation. ```rust benches/my_benchmark.rs theme={null} #[divan::bench] fn fibonacci_bench() -> u64 { fibonacci(divan::black_box(30)) } ``` The divan guide covers the patterns to avoid in [Ensure code is not optimized out](/docs/guides/how-to-benchmark-rust-with-divan#ensure-code-is-not-optimized-out). ## Continue by choosing a benchmarking crate The most convenient way to run your Rust benchmarks A benchmarking crate inspired by the criterion haskell library The libtest (unstable) benchmark runner # CodSpeed CLI Source: https://codspeed.io/docs/cli Run performance tests from your terminal The CodSpeed CLI allows you to benchmark **any executable program** and collect performance data. It is open source, and its source code is available on [GitHub](https://github.com/CodSpeedHQ/codspeed). ## Getting Started ### Installation Install the CodSpeed CLI using the installation script: ```bash theme={null} curl -fsSL https://codspeed.io/install.sh | sh ``` After installation, authenticate with your CodSpeed account by running `codspeed auth login`. To install a specific version: ```bash theme={null} curl -fsSL https://codspeed.io/v5.1.0/install.sh | sh ``` Replace `v5.1.0` with the version you want to install. You can find all available versions on the [releases page](https://github.com/CodSpeedHQ/codspeed/releases?q=codspeed-runner). The CodSpeed CLI officially supports Ubuntu 20.04, 22.04, 24.04 and Debian 11, 12\. Other Linux distributions can be used as long as the required tools for your chosen instruments are [manually installed](#manual-tool-installation) and available in your `PATH`. ### Run your first command Let's run a simple benchmark using the walltime instrument: ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed exec -- sleep 1 ►►► Running the benchmarks Executing: sleep 1 Completed 2 warmup rounds Warmup done, now performing 3 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results sleep 1: 1.01 s To see the full report, visit: https://app.codspeed.io/[...] ``` ## Configuration You can define benchmarks and options in a `codspeed.yml` configuration file. Most importantly, items in the `benchmarks` list describe the commands that are run when using `codspeed run`. The CodSpeed CLI looks for configuration files with the following names, by order of priority: 1. `codspeed.yml` 2. `codspeed.yaml` 3. `.codspeed.yml` 4. `.codspeed.yaml` You can also specify a custom path with `--config `. ### Benchmark fields The command to execute. Can include arguments and flags. A descriptive name for the benchmark. This is used to identify the benchmark in reports and results. When omitted, the command is used as the benchmark name. Override global instrument options for this specific benchmark. See the [Instruments](#instruments) section for available options for each instrument and their effects. ### Running benchmarks To run all benchmarks defined in your configuration file, specify the instrument with the `-m` flag: ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed run -m walltime ►►► Running the benchmarks Executing: Completed 13 warmup rounds Warmup done, now performing 20 rounds Executing: A single warmup execution (517.ms) exceeded or met max_time (200ms). Executing: Completed 23 warmup rounds Warmup done, now performing 68 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results ┌──────────────┬─────────────┐ │ Benchmark │ Measurement │ ├──────────────┼─────────────┤ │ │ 517.23 ms │ ├──────────────┼─────────────┤ │ │ 14.10 ms │ ├──────────────┼─────────────┤ │ │ 6.91 ms │ └──────────────┴─────────────┘ To see the full report, visit: https://app.codspeed.io/[...] ``` You can also use `codspeed exec` to run a single command: ```shellsession wrap title=terminal icon="square-terminal" theme={null} $ codspeed exec -m walltime --warmup-time 1ms --max-time 2s -- ./my-binary --arg1 value --arg2 ►►► Running the benchmarks Executing: ./my-binary --arg1 value --arg2 Completed 1 warmup rounds Warmup done, now performing 115 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results ./my-binary --arg1 value --arg2: 14.01 ms To see the full report, visit: https://app.codspeed.io/[...] ``` ### Example configuration ```yaml codspeed.yml theme={null} $schema: https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json # Global options applied to all benchmarks options: warmup-time: "0.2s" max-time: 1s # List of benchmarks to run benchmarks: - name: "" exec: ./my_binary --arg1 value --arg2 # Override global options for this benchmark options: max-rounds: 20 - name: "" exec: ./my_binary2 options: max-time: 200ms - name: "" exec: ./my_binary3 --flag options: min-time: 200ms ``` As displayed in the above example, we provide a schema for the configuration file [here](https://github.com/CodSpeedHQ/codspeed/blob/main/schemas/codspeed.schema.json). If your editor supports it, you can use the `$schema` field to enable autocompletion and validation for the `codspeed.yml` file. Please note that by default, the VSCode yaml language server does not support declaring the schema through the `$schema` field. One of the ways to declare the schema as is to add ```yaml codspeed.yaml theme={null} # yaml-language-server: $schema=https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json ``` More information in the [yaml-language-server documentation](https://github.com/redhat-developer/yaml-language-server?tab=readme-ov-file#using-inlined-schema). ## Instruments CodSpeed supports three instruments for measuring different aspects of your program's performance. ### Walltime The walltime instrument measures **real-world execution time**. It runs your benchmark multiple times and collects timing data. See the [Walltime documentation](/docs/instruments/walltime) for more details. #### How it works 1. **Warmup phase**: The benchmark runs repeatedly until the warmup time is reached. This allows the system to reach a steady state. 2. **Measurement phase**: Based on the average time from warmup, the CodSpeed CLI calculates how many rounds to run within the configured time bounds, then collects timing data for each round. The walltime instrument requires `sudo` privileges to run because it needs to configure the kernel so that [linux perf](https://en.wikipedia.org/wiki/Perf_%28Linux%29) can collect all the necessary events. We plan to fine tune permissions in the future to reduce permissions to the strict minimum. Profiling for interpreted languages (e.g., Python, JavaScript) is not currently supported with the walltime instrument. We are currently working on it and it's coming soon! #### Walltime Options Time spent in warmup runs. Set to `0` to disable warmup. Minimum time spent running measurement rounds. Maximum time spent running measurement rounds. Minimum number of measurement rounds. Maximum number of measurement rounds. If there is a conflict between time and round constraints, the CodSpeed CLI prioritizes satisfying the maximum bounds first. #### Example ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed exec -m walltime --warmup-time 100ms --max-time 1s -- ►►► Running the benchmarks Executing: Completed 7 warmup rounds Warmup done, now performing 63 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results : 14.15 ms To see the full report, visit: https://app.codspeed.io/[...] ``` ```yaml codspeed.yml theme={null} benchmarks: - name: My command exec: options: warmup_time: 100ms max_time: 1s ``` ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed run ►►► Running the benchmarks Executing: My command Completed 7 warmup rounds Warmup done, now performing 63 rounds ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results : 14.15 ms To see the full report, visit: https://app.codspeed.io/[...] ``` ### Memory The memory instrument tracks **heap allocations** during benchmark execution. Unlike walltime, the benchmark runs **only once** while memory usage is recorded. See the [Memory documentation](/docs/instruments/memory) for more details. It measures: * Peak memory usage * Total allocated memory * Allocation count * Average allocation size See [Supported Allocators](/docs/instruments/memory#supported-allocators) for the list of allocators that can be tracked. The memory instrument requires `sudo` privileges to run because it uses [eBPF](https://en.wikipedia.org/wiki/EBPF) to track memory allocations. We plan to fine tune permissions in the future to reduce permissions to the strict minimum. #### Example ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed exec -m memory -- ►►► Running the benchmarks Executing: ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results : 7.8 MB To see the full report, visit: https://app.codspeed.io/[...] ``` ### Simulation The simulation instrument uses **CPU simulation** to measure performance. The benchmark runs **only once** and CPU behavior is simulated, providing consistent measurements independent of system load. See the [CPU Simulation documentation](/docs/instruments/cpu) for more details. This is the same instrument used by [CodSpeed integrations](/docs/benchmarks/overview) when running in CI. Profiling for interpreted languages (e.g., Python, JavaScript) is not currently supported with the simulation instrument. We are currently working on it and it's coming soon! #### Limitations Statically linked executables are not currently supported with the simulation instrument. If this is a limitation for your use case, please [open a GitHub issue](https://github.com/CodSpeedHQ/codspeed/issues/new). Additionally, the simulation instrument currently does not follow child processes. Support for this is planned for a future release. #### Example ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed exec -m simulation -- ►►► Running the benchmarks Executing: ►►► Uploading results Linked repository: [...] Performance data uploaded ►►► Benchmark results : 26.74 ms To see the full report, visit: https://app.codspeed.io/[...] ``` ### Setting instrument with `codspeed use` If you do not want to always explicitly specify the instrument with `-m` when running benchmarks, you can set an instrument for your whole shell session by using ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed use simulation ``` This will set the simulation instrument as the default for all subsequent `codspeed run` and `codspeed exec` commands in the current shell session. You can check the currently selected instrument with ```shellsession title=terminal icon="square-terminal" theme={null} $ codspeed show Simulation ``` Even if you ran `codspeed use `, you can still override the instrument for a specific command by using the `-m` flag with `codspeed run` or `codspeed exec`. ## Running the benchmarks in your CI To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to automatically run benchmarks and warn you about regressions during development. If you want more details on how to configure the CodSpeed action, you can check out the [Continuous Reporting section](/docs/integrations/ci). Here is an example of a GitHub Actions workflow that runs the benchmarks and reports the results to CodSpeed on every push to the `main` branch and every pull request: Contrary to other CI usages, **the `run` input is intentionally omitted** here. This is what defines whether the action will run benchmarks defined in your configuration file, or a benchmark that makes use of one of the [CodSpeed integrations](/docs/benchmarks/overview). ## Manual tool installation On Ubuntu and Debian, the CodSpeed CLI automatically installs the tools required by each instrument. On other distributions, you need to install them manually. #### Walltime The walltime instrument uses [linux perf](https://en.wikipedia.org/wiki/Perf_%28Linux%29) for profiling, which is enabled by default. Refer to your distribution's documentation for how to install the `perf` tool. If you don't need profiling data, you can disable it by setting the `CODSPEED_PERF_ENABLED` environment variable to `false`. On macOS, no additional tool installation is needed. Note that Apple system binaries cannot be profiled, see [Profiling System Processes on macOS](/docs/instruments/walltime/macos-profiling). #### Simulation The simulation instrument requires [CodSpeed's fork of Valgrind](https://github.com/CodSpeedHQ/valgrind-codspeed). You need to build and install it from source. **Prerequisites**: `git`, `autoconf`, `automake`, `make`, and a C compiler. ```bash theme={null} git clone https://github.com/CodSpeedHQ/valgrind-codspeed.git cd valgrind-codspeed ./autogen.sh ./configure make -j make install ``` After installation, verify that the CodSpeed version of Valgrind is in your `PATH`: ```shellsession theme={null} $ valgrind --version valgrind-3.26.0.codspeed ``` The `valgrind` binary in your `PATH` must be the CodSpeed version for the simulation instrument to work correctly. #### Memory The memory instrument requires a Linux kernel with eBPF support. No additional tool installation is needed, the CLI will download and load the necessary tools automatically. ## Next Steps Learn how to use flame graphs and profiling data to optimize your code Dive into instruments and learn how to measure different aspects of your program's performance # Data Deletion Source: https://codspeed.io/docs/data-deletion Learn how to delete repositories, accounts, and organizations in CodSpeed CodSpeed allows you to delete your data when you no longer need it. This page explains how to delete a repository and how to request deletion of your account or organization. ## Deleting a repository You can delete a repository directly from the CodSpeed dashboard. When you delete a repository, all associated data is permanently removed. This action is permanent and cannot be undone. All benchmarks, benchmark results, and runs associated with this repository will be permanently deleted. **Permissions** Only repository administrators can delete repositories. ### How to delete a repository 1. Navigate to your repository on CodSpeed 2. Go to **Settings > General** 3. Scroll to the **Delete Repository** section at the bottom of the page 4. Click **Delete repository** 5. Review the confirmation modal, which shows the number of benchmarks and runs that will be deleted 6. Type the full repository name (e.g., `owner/repository-name`) to confirm 7. Click **Delete repository** ## Deleting your account or organization Account and organization deletion is handled by our support team to ensure all associated data is properly removed. Deleting an account or organization will permanently remove all associated repositories, benchmarks, and performance data. ### How to request deletion To request deletion of your account or organization, contact us at [contact@codspeed.io](mailto:contact@codspeed.io). Please include in your request: * The account or organization name you want to delete * Confirmation that you understand this action is permanent ## Related Learn about permission levels and what each role can do in CodSpeed Learn about CodSpeed's security practices # FAQ Source: https://codspeed.io/docs/faq ### Is There a Light Mode? It's coming back soon! 🌑💡 ### Why are my benchmarks taking more time when run with the CodSpeed GitHub Action? Since we instrument the benchmarks for increased performance consistency, the benchmarks will take more time to run in the action than locally with the uninstrumented CodSpeed plugins. If you need to speed up your benchmark processes, check out [how to setup parallel benchmarks](/docs/features/sharded-benchmarks). ### Is it possible to make local CodSpeed runs? Yes, it's possible to upload runs from your local environment using the [CLI](https://github.com/CodSpeedHQ/codspeed) (aka runner), only available on Ubuntu and Debian for now. # Benchmark Archival Source: https://codspeed.io/docs/features/archiving-benchmarks/index Learn how to archive and restore outdated benchmarks in CodSpeed To keep your reports clean and relevant, you can **archive benchmarks that were removed from your codebase**. When benchmarks are **permanently removed** from your codebase, they will first appear as "skipped" in reports due to the [partial runs](/docs/features/partial-runs) feature. You can then **archive** these benchmarks to remove them from future reports. Archived benchmarks can be restored at any time if needed. ## Effects of archiving a benchmark An archived benchmark will be **completely removed from active reports**. Archived benchmarks can be found in the benchmarks page. You can filter the list by typing `is:archived` in the search bar or using the **Status** filter and select the "Archived" option. Their individual dashboard pages remain accessible with full historical data. They can be restored from there. Archiving a benchmark on a pull request (where the benchmark was skipped) **will remove the benchmark from the report**. ## How to archive a benchmark **Archiving Permissions** When working with an organization's repository, **only the admins** are allowed to archive or restore benchmarks. ### From the benchmark list The most common way to archive benchmarks is when they are removed from your codebase in a pull request. 1. Navigate to the branch page where the benchmarks were skipped 2. Filter the benchmark list by typing `is:skipped` in the search bar or using the **Status** filter and select the "Skipped" option 3. Select the benchmarks you want to archive using the checkboxes 4. Click the **Archive selected** button that appears in the list header ### From a benchmark's dashboard You can also archive individual benchmarks from their dashboard: 1. Navigate to the benchmark's dashboard by clicking on its name in any report 2. Click the *Archive* button on the benchmark's dashboard 3. Confirm the archival action ## How to restore an archived benchmark You can restore an archived benchmark at any time. ### From the benchmark list If you added back some archived benchmarks to your codebase, you can restore them from the corresponding branch page: 1. Navigate to the branch page and filter the benchmark list by typing `is:archived` in the search bar or using the **Status** filter and select the "Archived" option 2. Select the benchmarks you want to restore using the checkboxes 3. Click the **Restore selected** button that appears in the list header ### From a benchmark's dashboard 1. Navigate to the archived benchmark's dashboard 2. Click the *Restore* button 3. The benchmark will reappear in active reports ## Related Features Understand when benchmarks become skipped and may need archiving Ignore flaky benchmarks while keeping the code (different from archiving removed code) # Customization Source: https://codspeed.io/docs/features/customization Learn how to customize your integration with CodSpeed ## Regression threshold The regression threshold is the **percentage of performance degradation** relative to your default branch that is **considered acceptable**. CodSpeed provides fine-grained control over regression thresholds, allowing you to configure different thresholds for individual benchmarks. The status check passes only when **all benchmarks** remain within their respective regression thresholds. If any benchmark exceeds its threshold, the status check will fail. You can resolve this by either fixing the performance issue or [acknowledging the regression](./performance-checks#acknowledge-regressions-or-benchmark-drops). ### Global regression threshold By default, the global regression threshold is set to **10%** but you can change it in the **Settings** tab of your repository: Regression threshold input

The global regression threshold applies to all benchmarks that do not have a specific per-benchmark regression threshold set. ### Per-benchmark regression thresholds Individual benchmarks can have custom regression thresholds configured from the benchmark's dashboard page via the "Actions" menu:

Per-benchmark regression threshold in actions menu

The modal allows you to configure a custom regression threshold for the benchmark:

Per-benchmark regression threshold input

Per-benchmark regression thresholds override the global threshold when set, allowing fine-grained control over performance expectations for specific benchmarks. ## Informational Status Check on Failure Sometimes, you may not want a failure status to appear in your pull requests when the performance check fails. In this case, you can enable the **Informational Status Checks on Failure** option in the **Settings** tab of your repository. When this option is enabled, the status check will be sent as an informational status check instead of a failure. ## Pull Request comments You can customize when the Pull Request reports will be sent: * **Always**: performance will be reported **on every commit**. If a comment already exists, it will be updated. * **On Change**: performance will be reported **if there is a significant improvement**(more than the regression threshold) or **a failure**(a regression or a drop). If the comment is created after a change, it will be updated later on even if the performance is in standard bounds. * **Never**: the performance will **never be reported** in the comments of your Pull Requests. The status check will still be sent though. # Ignoring a Benchmark Source: https://codspeed.io/docs/features/ignoring-benchmarks/index Learn how to ignore a benchmark from a report ## Reasons to ignore a benchmark **Ignoring a benchmark should be a last resort** when the source of flakiness cannot be immediately addressed. Some reasons to ignore a benchmark include: * The benchmark is I/O bound (network access, file-system access, etc.), leading to inconsistent results * The benchmarked source code has a non-deterministic behavior (random number generation, memory allocation, etc.) * The benchmark is consistently flaky for no apparent reason Remember, the **goal of CodSpeed is to provide accurate and reliable measurements**. Ignoring a benchmark should be a temporary measure until the source of flakiness can be addressed. **Before ignoring a benchmark with expected variance**, consider using [per-benchmark regression thresholds](/docs/features/customization/#per-benchmark-regression-thresholds) instead. This is ideal when the benchmark has predictable variability due to its nature (e.g., I/O operations with consistent but higher variance) rather than random flakiness. This approach lets you maintain performance tracking while accommodating the benchmark's inherent characteristics. ## Effects of ignoring a benchmark An ignored benchmark will be **excluded from the overall measure** of a report. However, it will still be accessible at the bottom of the report, in the *Ignored* section. This allows you to keep track of ignored benchmarks, and keep access to their [execution profiles and flamegraphs](/docs/features/profiling) and re-enable them once the source of flakiness has been addressed. Ignoring or un-ignoring a benchmark **will regenerate and potentially modify the reports of the latest commits of the default branch and branches with opened pull requests**. ## How to ignore a benchmark **Ignoring Permissions**: When working with an organization's repository, **only the admins** are allowed to ignore or un-ignore benchmarks. To ignore a benchmark, follow these steps: 1. Navigate to the benchmark's dashboard. You can do this by clicking on the button in the report or directly through the benchmarks page 2. Once on the benchmark's dashboard, click on the *Ignore* button 3. Enter a reason for ignoring the benchmark. This information will be used by CodSpeed to prioritize areas where we can reduce the flakiness of benchmarks. 4. Click on the *Ignore* button to confirm the action ## How to un-ignore a benchmark You can reverse the effects of ignoring a benchmark by simply clicking on the *Un-ignore* button on an ignored benchmark's page. # Macro Runners Source: https://codspeed.io/docs/features/macro-runners Use CodSpeed's bare-metal runners for precise walltime measurements in your CI pipeline CodSpeed Macro Runners are **dedicated bare-metal machines** managed by CodSpeed that provide a stable, isolated environment for running your benchmarks. Unlike traditional CI runners, macro runners eliminate noise from virtualization and shared resources, enabling **precise [walltime](/docs/instruments/walltime) measurements** with low variance. Read our detailed post on [the consistency of CodSpeed Macro runners compared to traditional CI runners](https://codspeed.io/blog/benchmarks-in-ci-without-noise). ## What are Macro Runners? Macro runners are **16-core ARM64 bare-metal machines** with 32 GB RAM, specifically optimized for consistency in performance measurements. They complement CodSpeed's [CPU simulation](/docs/instruments/cpu) by providing an environment where walltime measurements are reliable and reproducible. ### When to Use Macro Runners Macro runners are ideal when you need to measure: * **System calls and I/O operations** that are excluded from the CPU Simulation instrument * **End-to-end performance** including network, disk, and system interactions * **Walltime benchmarks** where actual execution time matters * **Integration tests** that require a complete system environment **CPU Simulation vs Walltime with Macro Runners** Use the [CPU Simulation instrument](/docs/instruments/cpu) for pure algorithmic performance and the [Walltime instrument](/docs/instruments/walltime) with Macro Runners for system-level performance that includes I/O operations. ## Pricing and Usage ### Free Tier * **600 minutes per month** included with all plans * Available for both Free and Pro plans * Perfect for getting started with walltime measurements ### Additional Usage * **\$0.032 per minute** after the free 600 minutes * Volume discounts available for Enterprise plans * Transparent, pay-as-you-use pricing **Open Source Projects** We're happy to support open source projects with additional free minutes beyond the standard 600 minutes/month limit. [Contact us](mailto:contact@codspeed.io) with details about your project. ## Setup with GitHub Actions Ensure you have: * `CodSpeedHQ/action >= 3.1.0` * A GitHub organization (macro runners don't work with personal accounts) * CodSpeed enabled for your repository **Organization Required**: Macro runners are only available for GitHub organizations, not personal accounts. This is due to GitHub's permission requirements for self-hosted runners. Replace `runs-on: ubuntu-latest` with `runs-on: codspeed-macro` in your GitHub Actions workflow: The workflow setup is identical to [regular CodSpeed integration](/docs/integrations/ci/github-actions#2-create-the-benchmarks-workflow), just with a different runner. If you use caching, include `${{ runner.arch }}` in your cache keys to avoid cache misses: ```yaml {4} theme={null} - uses: actions/cache@v4 with: path: # insert your cache path here key: pip-${{ hashFiles('pyproject.toml') }} # [!code --] key: pip-${{ runner.arch }}-${{ hashFiles('pyproject.toml') }} # [!code ++] ``` ## Repository Access Configuration ### Private Repositories Macro runners work automatically with private repositories in your organization. ### Public Repositories For public repositories, you need to explicitly enable macro runner access: Go to your GitHub organization settings: **Organization Settings** → **Actions** → **Runner groups** → **Default** Allow the runner group to be used by public repositories: Enabling macro runners for public repositories

Enabling macro runners for public repositories

## Next Steps Learn how to use the walltime instrument with macro runners Interpret your walltime benchmark results Set up automated performance regression detection View detailed pricing and plan options # Partial Runs Source: https://codspeed.io/docs/features/partial-runs/index Learn how to leverage partial benchmark runs in CodSpeed to only run relevant benchmarks With your projects becoming larger, you might end up with long-running benchmark workflows, degrading the performance feedback loop and using a lot of resources in your CI. As a solution to these problems, when doing a CI run, you can run only a subset of the benchmarks that are defined in your codebase. For example: * only run benchmarks relevant to the code changes in a pull request * run a subset of long-running benchmarks on a schedule Partial runs allows you to **run incomplete benchmark suites** while still receiving performance reports containing all the benchmarks of your repository. This allow you to **reduce CI execution time** by only running benchmarks relevant to your code changes, while still maintaining a complete performance history. Performance Improvement with Partial Runs

Performance Improvement with Partial Runs

When benchmarks are missing from a run, CodSpeed automatically uses **baseline results** from previous runs to fill the gaps. ## How to make partial runs? There are several ways to implement partial runs in your CI workflow: * in addition to using [sharded benchmarks](/docs/features/sharded-benchmarks): * use your benchmarking framework's "affected" or "changed" feature if it exists * detect which package/library of your monorepo was affected by the code changes and run only its benchmarks * run a long-running benchmark suite on a schedule (e.g. nightly) in addition to running a smaller suite on each pull request ## Skipped Benchmarks When CodSpeed detects missing benchmarks in a run, it automatically: 1. **Identifies missing benchmarks** by comparing the current run with its [baseline run](/docs/features/understanding-the-metrics/#baseline-report-selection) 2. **Retrieves baseline results** from the baseline run 3. **Displays them as "skipped"** with in reports You will thus be able to track the complete performance history of your benchmarks and compare any two runs, even if they ran different subsets of benchmarks. ## Managing Removed Benchmarks When removing benchmarks from your codebase, they will first appear as "skipped" in reports. To remove them from future reports, use the [benchmark archival feature](/docs/features/archiving-benchmarks). # Setting up Continuous Performance Checks Source: https://codspeed.io/docs/features/performance-checks/index Learn how to enable performance safeguards in your project CodSpeed provides a performance **status check** to your branches and pull requests. The status of this check is based on the performance change between the current branch and your main branch (and the project's [regression threshold](/docs/features/customization#regression-threshold)). You can configure the performance check to fail if the performance metrics of a branch or pull request are overshooting your regression threshold. Thus, **blocking the pull request** from being merged if the performance issue is neither fixed nor [acknowledged](#acknowledge-regressions-or-benchmark-drops). ## Setup performance checks Once your project has run at least one benchmark, go to your repository's settings on GitHub. Under the **Code and automation** section, open the **Branches** tab. Add a new branch protection rule or update the one already protecting your main branch. Set the branch name pattern

Tick the **Require status checks to pass before merging** checkbox. Under this section, search for **CodSpeed Performance Analysis** and click on the item to enable the check: Configuration of the status check

**Save the changes** and you're done! 🎉 Your performance checks are now configured and will run on future pull requests, preventing you from merging pull requests that introduce performance regressions. Pull Request Checks failing because of the performance regression

Pull Request Checks failing because of the performance regression

## Acknowledge regressions or benchmark drops If you're aware of performance issues that are not yet fixed or were totally intentional, you can acknowledge them to **prevent the performance check from failing**. **Acknowledgement Permission**: When working with an organization's repository, **only admins** are allowed to acknowledge regressions. Once all the regressions are fixed or acknowledged, the performance check will pass and you can safely merge your pull request while being **aware of your performance issues**. ### Acknowledge a single benchmark First, head to the CodSpeed report (you can access it from the link in the inline performance report or directly from your [CodSpeed Dashboard](https://app.codspeed.io/dashboard)). Then, go to the regressed benchmark you want to acknowledge: failing benchmark

Click on the the benchmark to expand its content: Expanded benchmark

Click on the "Acknowledge regression" button and that's it 🎉 : Acknowledged benchmark

### Acknowledge multiple benchmarks To acknowledge multiple benchmarks at once: Filter the benchmark list to show only regressions using the **Status** filter or by typing `is:regression` in the search bar:

Select the benchmarks you want to acknowledge using the checkboxes: Acknowledge selected benchmarks

Click the **Acknowledge selected** button that appears in the list header and that's it 🎉 : Acknowledged multiple benchmarks

## Understanding Performance Regressions Learn how CodSpeed calculates performance impact and regression thresholds Use detailed profiling data to identify and fix performance bottlenecks ## Related Guides Customize when performance changes trigger failures Set up CodSpeed in your continuous integration pipeline # Performance Profiling and Flame Graphs Source: https://codspeed.io/docs/features/profiling Debug performance issues with detailed flame graphs and execution profiles generated by CodSpeed's instrumentation Cover Image

CodSpeed's profiling capabilities provide deep insights into your application's performance through detailed flame graphs and execution traces. Flame graphs are available for benchmarks using the [CPU Simulation instrument](/docs/instruments/cpu) and the [Walltime instrument](/docs/instruments/walltime). ## Reading Flame Graphs Flame graphs are a visualization tool for profiling software. They provide a graphical representation of your program's execution, making it easier to understand the runtime complexities involved. Let's start with an example: Flame graph Example

Here, each rectangle represents a function. The width of the rectangle is proportional to the amount of time spent in that function. The wider the rectangle, the more time was spent in that function. The vertical axis represents the call depth (a.k.a. stack depth), which represents the call hierarchy. For this example, here is the call hierarchy: 1. The root caller is the `app` function. 2. `app` calls the `init`, `handleRequest` and `terminate` functions. 3. `handleRequest` calls both `authenticateUser` and `processData`. 4. `processData` calls `foo` which in turn calls `bar`. ### Aggregated function calls Functions calls are aggregated, so if a function is called multiple times, the time spent in all calls is aggregated into a single block. Thus, the following code: ```python theme={null} def bar(): pass def foo(): bar() bar() def main(): foo() foo() ``` Will generate the following flame graph: Aggregated functions in a flame graph

Where the `foo` function is called twice and the `bar` function is called 4 times, but the time spent is aggregated into a single block for each function. ### Self-costs In the previous example, we could see the global cost of each function call quite clearly. However, it can be tricky to find out how much time was spent within the function itself. Self costs in flame graphs

In the above illustration, we can see two types of self-costs: * **(implicit) self-costs**: the time spent in the function itself is the whole width of the rectangle since it doesn't call any other functions. * **self-costs**: the self-cost here is visible as the space not occupied by the children of the block. **Self-costs in interpreted languages** In Python or Node.js, the self-cost is the time spent in the function itself, but also the time spent by the interpreter. This means that a function will always have a self-cost, even if the function does nothing. If we change a bit the example, adding a lot of computation directly in the main function: ```python {2-3} theme={null} def main(): for i in range(100): # do some computation foo() foo() ``` Then the flame graph would look like this, with the self-cost of `main` being much bigger than before: Bigger self costs in flame graphs

## Viewing Flame Graphs On the pull request page, you can access the flame graphs of a benchmark by expanding it. Example of flame graphs on a pull request page

Example of flame graphs on a pull request page

Three types of flame graphs are available: * **Base**: flame graph of the [benchmark base run](/docs/features/understanding-the-metrics/#baseline-report-selection) * **Head**: flame graph of the benchmark run from the latest commit of the pull request * **Diff**: difference between the head and the base flame graphs **FFI Support** If you're using [Foreign Function Interface](https://en.wikipedia.org/wiki/Foreign_function_interface), typically calling C/C++/Rust code from Python or Node.js, make sure to generate debug symbols for the foreign functions so that you can see them in the flame graph. ## Inspector Hover any bar to open the span details. This panel shows you what the function is, where it comes from, and how its time is spent. Flamegraph inspector

* **Metadata**: Function name, source file, code origin * **Self time**: Time spent in the function body only, excluding child calls. * **Total time**: Time spent in the function including all its children. The details of metrics displayed depend on the instrument used to collect the profiling data, head over to the respective instrument documentation for more information: * [CPU Simulation metrics](/docs/instruments/cpu#inspector-metrics) * [Walltime metrics](/docs/instruments/walltime#inspector-metrics) ## Color modes ### By Origin Colors by code origin: User, Library, System, or Unknown. Useful to separate your code from dependencies and the kernel. Flamegraph color mode example: By Origin

Flamegraph color mode example: By Origin

### Differential Compares Base vs Head and colors spans by change: slower, faster, added, or removed. Ideal for scanning regressions and wins after a commit. Flamegraph color mode example: Diff

Colors reflect the status of the span compared to the base run: * **Slower**: The span is slower than in the base run. * **Faster**: The span is faster than in the base run. * **Added**: The span has been added in the head run. * **Removed**: The span is removed in the head run. ### By Bottleneck Colors each span by the dominant bound on its self time: instruction-bound, cache-bound, memory-bound, or system-bound. Fast way to see what is blocking work. Flamegraph color mode example: By Bottleneck

Flamegraph color mode example: By Bottleneck

### By Function Colors spans by function symbol so identical functions share a color, no matter where they are called. Helps spot hot functions across call sites. Flamegraph color mode example: By Function

Flamegraph color mode example: By Function

Function colors will be the same across benchmarks, projects, and runs. So you can easily recognize the same function across different runs. ### System Calls toggle Include kernel and low-level runtime contributions. Off keeps focus on application and library code. ## Function list Upon expanding a flamegraph, you can access the function list. And dive in the details of each span.

## Next Steps Learn how to enable CPU Simulation to generate flame graphs Learn how to enable Walltime instrumentation to generate flame graphs Set up automated checks to catch performance issues early Create comprehensive benchmarks for your codebase # Roles & Permissions Source: https://codspeed.io/docs/features/roles-and-permissions/index Learn how permissions work in CodSpeed Permissions for an organization can be accessed by clicking on the `Permissions` tab on the organization's settings page. At the moment, CodSpeed only supports roles and permissions defined at the organization level. This means that all projects in an organization have the same permissions. We plan to add support for project-level permissions in the future. ## Joining an organization The members of a CodSpeed organization are the members of the corresponding organization on your repository provider, e.g., GitHub or GitLab. Adding someone there is enough to give them access to CodSpeed. A user appears in the CodSpeed organization when they: * Log into CodSpeed while being a member of the provider organization. * Trigger a CodSpeed run on one of the organization's repositories. Memberships are synced on every login. Removing a user from the provider organization also removes them from the CodSpeed organization. Once a user appears in the `Permissions` tab, an admin can promote them to admin. ## Roles Roles are a way to control what a user can do in CodSpeed. There are three levels of permissions, from highest to lowest: * **Provider admin**: Users who are admins in GitHub, GitLab, or Bitbucket are automatically provider admins in CodSpeed. They cannot be demoted to a lower level of permission. * **Admin**: Admins can promote and demote users to and from the admin level. They have full write access to CodSpeed. * **Member**: Members have basic read-only access to CodSpeed. ## Permissions The following table shows what each role can do in CodSpeed. | Permission | Provider admin | Admin | Member | | :------------------------- | :---------------: | :---------------: | :-----------------: | | View projects | | | | | Add projects | | | | | View project settings | | | | | Edit project settings | | | | | View project members | | | | | Edit project members roles | | | | | Manage billing | | | | | Acknowledge regressions | | | | For details on how seats and billing work, see [Seats & Billing](/docs/features/seats-and-billing). # Seats & Billing Source: https://codspeed.io/docs/features/seats-and-billing/index Learn how users, seats, and billing work in CodSpeed CodSpeed is free and unlimited on public repositories: there are no restrictions on the number of users. On private repositories, the Free plan includes up to **5 users**. Beyond that, you need a Pro plan, where each active user consumes one **seat**. ## Users A **user** is anyone who triggers a CodSpeed run on a private repository of an organization, or accesses the CodSpeed dashboard of that organization. Bot accounts (identified by the `[bot]` suffix, e.g., `dependabot[bot]`) are not counted as users and do not consume seats. ## Seats On the Pro plan, seats control how many users in your organization can use CodSpeed at the same time. A user without a seat is **disabled**: they lose access to CodSpeed for your organization, and their runs are blocked on private repositories. ## Managing seats ### Automatic seat allocation Automatic seat allocation controls whether CodSpeed adds seats on demand when new users appear. You can toggle this setting in the Billing tab of your organization's settings page. * **Enabled**: When a new user joins or triggers a run and no seats are available, CodSpeed automatically adds a seat and updates your subscription. * **Disabled**: New users are created as disabled if no seats are available. An admin must manually add seats and enable the user. ### Disabling a user There are two ways a user can become disabled: * **Manually**: An admin disables the user through the Members settings page. CodSpeed treats this as intentional. The user will **not** be automatically re-enabled, even if automatic seat allocation is on. * **Automatically**: A new user joins or triggers a run, but no seats are available and automatic seat allocation is off. Automatic seat allocation only applies to new users joining the organization. It does not re-enable users who were manually disabled by an admin. ### Re-enabling a user To re-enable a disabled user, go to the Members settings page and enable them. Re-enabling a user consumes a seat and triggers prorated billing. ## Billing Each seat on a Pro plan corresponds to one billing unit. ### Adding seats * **Monthly plan**: The cost of the new seat is prorated on the next invoice. * **Yearly plan**: The cost is prorated and invoiced immediately. ### Removing seats Removing a seat keeps it active and billed until the end of the current billing cycle. No proration is applied. ### Macro runner budget You can set a monthly budget for macro runner usage in the Billing tab. For more details, see [Macro Runners](/docs/features/macro-runners#pricing). # Sharded Benchmarks Source: https://codspeed.io/docs/features/sharded-benchmarks/index Learn how to improve your benchmarking process performance Performance Improvement with Parallel Benchmarks

Performance Improvement with Parallel Benchmarks

Running benchmarks can be quite long and slow down your CI process. With CodSpeed, you can run multiple benchmark commands in the same CI workflow. When running heavy benchmark suites, this can divide the total runtime by the number of jobs, dramatically speeding up your CI pipeline. We recommend limiting each shard to fewer than 1,000 benchmarks. Multiple Benchmark Overview

You can speed up your CI even further by combining sharded benchmarks with [partial runs](/docs/features/partial-runs). This way, you can run only the benchmark suites that are relevant to the code changes in your pull request. ## How to split the execution of benchmark suites? Sharding allows you to run your benchmarks in several commands, with each command running a subset of your benchmarks. There are two main ways of sharding your benchmarks: * splitting them in several files, and running each file or folder independently * using a sharding tool provided by your benchmarking framework Check the sharding docs corresponding to your benchmarking framework: * Python: * [`pytest`](/docs/benchmarks/python#running-benchmarks-in-parallel-ci-jobs) * NodeJS: * [`vitest`](/docs/benchmarks/nodejs/vitest#running-benchmarks-in-parallel-ci-jobs) * Rust: * [`divan`](/docs/benchmarks/rust/divan#running-benchmarks-in-parallel-ci-jobs) * [`criterion`](/docs/benchmarks/rust/criterion#running-benchmarks-in-parallel-ci-jobs) * [`bencher`](/docs/benchmarks/rust/bencher#running-benchmarks-in-parallel-ci-jobs) ## Configure your CI workflow to use sharded benchmarks CodSpeed only supports emitting results from your benchmarks if you split them within a single CI workflow. If you run benchmarks in multiple CI workflows, CodSpeed will not be able to aggregate the results correctly, and you may see incomplete or missing data in your CodSpeed reports. Once you've sharded your benchmarks, you can run using several CI jobs within the same workflow, depending on your CI provider: * [GitHub Actions](/docs/integrations/ci/github-actions/configuration#running-benchmarks-in-parallel-ci-jobs) * [GitLab CI](/docs/integrations/ci/gitlab-ci/configuration#running-benchmarks-in-parallel-ci-jobs) * [CircleCI](/docs/integrations/ci/circleci/configuration#running-benchmarks-in-parallel-ci-jobs) ## Multiple languages benchmarks With benchmarks written in several languages, it can be difficult to get a unified performance overview of your project. With CodSpeed, you can run benchmarks written in multiple languages. When run in the same CI workflow, CodSpeed will aggregate the results of these benchmarks into a single report. For example, using `pytest` and `vitest`: ```yaml .github/workflows/codspeed.yml icon="github" theme={null} jobs: python-benchmarks: name: "Run Python benchmarks" runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - name: Install required-version defined in uv.toml uses: astral-sh/setup-uv@v7 - uses: actions/setup-python@v6 with: python-version: 3.12.8 - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation run: uv run pytest tests/benchmarks/ --codspeed nodejs-benchmarks: name: "Run NodeJS benchmarks" runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - uses: "actions/setup-node@v6" - name: Install dependencies run: npm install - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation run: npm exec vitest bench ``` # Impact Metrics Source: https://codspeed.io/docs/features/understanding-the-metrics/index Learn how the CodSpeed metrics work ## Benchmark performance impact The performance impact denotes an improvement or regression in performance of a benchmark. It is calculated by comparing the benchmark time measurement of the head commit with the time measurement of the base commit. $$ impact = \frac{speed - baseSpeed}{baseSpeed} $$ A negative performance metric means that the benchmark is slower than the previous commit. The closer its value is to `-1`, the slower it is. $$ -1 \lt impact \lt 0 $$ A positive performance metric means that the benchmark is faster than the previous commit. Its value can go up to $+\infty$ to denote massive speed improvements. $$ 0 \lt impact \lt +\infty $$ Naturally, when the benchmark is as fast as the previous commit, the performance metric is $0$. ## Regression threshold On the settings page of a project, you can set a threshold for a regression to be considered a regression. By default, this value is set to `10%` (which is equivalent to `0.1`). The value can be set from `0%` to `50%` by an admin of the project. More information about setting the threshold can be found in the [customization documentation](/docs/features/customization#regression-threshold). Individual benchmarks can have their own custom regression thresholds that override the project-level setting. See the [customization documentation](/docs/features/customization#per-benchmark-regression-thresholds) for details on configuring per-benchmark thresholds. ## Commit performance impact To get the overall performance impact of a commit, we take the geometric mean of the benchmarks whose change exceeds their regression threshold (the "changed" benchmarks). In that case, benchmarks within the threshold are not taken into account. Let $k$ be the number of changed benchmarks and $changedImpact$ their list of impacts. $$ commitImpact = \left(\prod_{i=0}^{k-1} \left(1 + changedImpact_i\right)\right)^{1/k} - 1 $$ > *For example, with impacts `[0.3, 0.3, 0.3, -0.11]` and a threshold of `0.1`, > all four benchmarks exceed their threshold, so the overall commit impact is > approximately `+0.20`.* If no benchmark exceeds its threshold, we fall back to the geometric mean of **all** benchmarks. The overall commit impact reflects the aggregate story of the run. Regressions above threshold are surfaced **independently** of this number: they are listed in the report, trigger the performance status check to fail, and are flagged in the pull request comment — even when the overall impact is positive. We use the geometric mean rather than the arithmetic mean because it is less sensitive to outliers and composes the way relative performance changes naturally do. ## Baseline report selection To create a performance impact, we need to compare the execution speed of the benchmarks against a baseline of those benchmarks' execution speed. Depending on the context of the run, the baseline report can be different. A baseline only exists if the base branch (usually your default branch, e.g., `main`) already has CodSpeed runs. Run CodSpeed on `push` to your default branch so every commit there is benchmarked. If CodSpeed runs only on `pull_request`, there is nothing on the default branch to compare against, and pull requests show no performance impact. ### Pull Request When triggering a CodSpeed run on a pull request between a `head` branch and a `base` branch, the baseline report will be the report of the latest commit of the `base` branch with a CodSpeed run. **Checked-out commit of a Pull Request in GitHub Actions** By default, when using the [`action/checkout`](https://github.com/actions/checkout) in GitHub Actions, the checked-out commit of a pull request will be the [merge commit](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request) of the pull request. This means that GitHub will create a merge commit **M** between the `base` branch and the `head` branch, and use it as the checked-out commit. Pull Request Base

In this case, the report will be between the `HEAD` commit **C** of the `base` branch and the merge commit **M**. The report will thus only include the changes of commits **B1** and **B2**. Every time a new commit is pushed on the `feat-branch`, the baseline report will be updated to the latest commit of the `base` branch with a CodSpeed run. This pull request base selection algorithm will only work on `pull_request` event for GitHub Actions and `merge_request_event` for GitLab CI. If you run CodSpeed on a different event, the branch base selection algorithm defined below will be used instead. ### Branch When triggering a CodSpeed run following a push on a branch, the baseline report will be the report of the closest commit of the branch.

In this example, a report already exists for the commit **B** of the `main` branch. A new commit **C** is pushed on the `main` branch. The baseline report for **C** will be the report of the commit **B**. ## Performance impact Gauge The performance impact gauge is a visual representation of the performance impact, displayed in multiple places of the CodSpeed UI. Some examples of the gauge with their corresponding impact values:

-0.75

-0.2

0.3

1.5

To make it easier to spot regressions and improvements, the mapping between the actual performance impact value and the gauge is not linear. ## Next Steps Configure GitHub branch protection to automatically block performance regressions Use flame graphs and profiling data to identify and fix bottlenecks Fine-tune regression sensitivity for your project's requirements Build a robust benchmark suite to catch performance issues early # Benchmarking a Go Gin API Performance Source: https://codspeed.io/docs/guides/benchmarking-a-go-gin-api/index Build a Gin HTTP API, write Golang benchmarks, and run them with CodSpeed in consistent CI environments

This guide shows how to benchmark a Gin-based HTTP API using Go's `testing` package and CodSpeed. We'll create a minimal API, design clean benchmarks measuring what matters, and run them in CI with consistent results. **Prerequisites** * Basic knowledge of Gin and HTTP * Go 1.24+ (for using `b.Loop()` with `testing.B`) * A GitHub repository (to run CI examples) ## Creating a base Gin API to work with Let's start with an API from the [official Gin tutorial](https://go.dev/doc/tutorial/web-service-gin#completed_code). If you never have used Gin before, following this tutorial is a great way to get started before we start benchmarking. We'll organize the project so benchmarks target a library package while you still have a runnable server for manual testing. ```go api.go icon=golang expandable theme={null} package api import ( "net/http" "github.com/gin-gonic/gin" ) // album represents data about a record album. type album struct { ID string `json:"id"` Title string `json:"title"` Artist string `json:"artist"` Price float64 `json:"price"` } // albums slice to seed record album data. var albums = []album{ {ID: "1", Title: "Blue Train", Artist: "John Coltrane", Price: 56.99}, {ID: "2", Title: "Jeru", Artist: "Gerry Mulligan", Price: 17.99}, {ID: "3", Title: "Sarah Vaughan and Clifford Brown", Artist: "Sarah Vaughan", Price: 39.99}, } func main() { router := gin.Default() router.GET("/albums", getAlbums) router.GET("/albums/:id", getAlbumByID) router.POST("/albums", postAlbums) router.Run("localhost:8080") } // getAlbums responds with the list of all albums as JSON. func getAlbums(c *gin.Context) { c.IndentedJSON(http.StatusOK, albums) } // postAlbums adds an album from JSON received in the request body. func postAlbums(c *gin.Context) { var newAlbum album if err := c.BindJSON(&newAlbum); err != nil { return } albums = append(albums, newAlbum) c.IndentedJSON(http.StatusCreated, newAlbum) } // getAlbumByID returns the album matching the provided id. func getAlbumByID(c *gin.Context) { id := c.Param("id") for _, a := range albums { if a.ID == id { c.IndentedJSON(http.StatusOK, a) return } } c.IndentedJSON(http.StatusNotFound, gin.H{"message": "album not found"}) } ``` The only difference here with the original code is that we're now using the `api` package name instead of `main` for compatibility reasons. As a small recap, this small HTTP API handles music albums by storing them in memory and has three routes: * `GET /albums`: Returns all albums * `GET /albums/:id`: Returns a specific album by ID * `POST /albums`: Creates a new album Let's run it to make sure it works: ```shellsession focus={1-4, 11-15} title=terminal icon="square-terminal" theme={null} $ go mod init github.com/your/repo # initialize the module $ go get github.com/gin-gonic/gin@latest # get the latest version of Gin $ go mod tidy # tidy the module dependencies $ go run api.go # run the server [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] GET /albums --> go-gin-benchmarks-example/api.getAlbums (3 handlers) [GIN-debug] GET /albums/:id --> go-gin-benchmarks-example/api.getAlbumByID (3 handlers) [GIN-debug] POST /albums --> go-gin-benchmarks-example/api.postAlbums (3 handlers) [GIN-debug] Listening and serving HTTP on localhost:8080 ``` And now a few HTTP requests to make sure it works: ```shellsession Request title=terminal icon="square-terminal" theme={null} $ http :8080/albums ``` ```json Response theme={null} [ { "artist": "John Coltrane", "id": "1", "price": 56.99, "title": "Blue Train" }, { "artist": "Gerry Mulligan", "id": "2", "price": 17.99, "title": "Jeru" }, { "artist": "Sarah Vaughan", "id": "3", "price": 39.99, "title": "Sarah Vaughan and Clifford Brown" } ] ``` ```shellsession Request title=terminal icon="square-terminal" theme={null} $ http :8080/albums/1 ``` ```json Response theme={null} { "artist": "John Coltrane", "id": "1", "price": 56.99, "title": "Blue Train" } ``` ```shellsession Request title=terminal icon="square-terminal" theme={null} $ http POST :8080/albums \ id=4 \ title="Kind of Blue" \ artist="Miles Davis" \ price:=29.99 ``` ```json Response theme={null} { "artist": "Miles Davis", "id": "4", "price": 29.99, "title": "Kind of Blue" } ``` Use `:=` for numbers in HTTPie to send them as JSON numbers rather than strings. ## Adding benchmarks to the API Now, let's get started writing performance tests to actually measure the performance of each route of this API. First, we need to do a bit of refactoring to make it easier to write benchmarks. ### Isolating the router In the initial code, the router is created and configured in the `main` function. This is not ideal for any tests or benchmarks because it's impossible to reuse the router configuration. Let's isolate the router creation and configuration in a separate function: ```go api.go lines icon=golang theme={null} func SetupRouter() *gin.Engine { // [!code ++:7] router := gin.Default() router.GET("/albums", getAlbums) router.GET("/albums/:id", getAlbumByID) router.POST("/albums", postAlbums) return router } func main() { router := SetupRouter() // [!code ++] router := gin.Default() // [!code --:4] router.GET("/albums", getAlbums) router.GET("/albums/:id", getAlbumByID) router.POST("/albums", postAlbums) router.Run("localhost:8080") } ``` ### Writing the first benchmark Now, let's write the first benchmark for the `GET /albums` route, strongly inspired by [the Gin documentation on writing tests](https://gin-gonic.com/en/docs/testing/): ```go api/api_test.go icon=golang theme={null} package api import ( "net/http" "net/http/httptest" "testing" ) func BenchmarkGetAlbums(b *testing.B) { router := SetupRouter() req, _ := http.NewRequest("GET", "/albums", nil) w := httptest.NewRecorder() for b.Loop() { router.ServeHTTP(w, req) } } ``` This benchmark creates a router, creates a request and a response recorder, and then loops over the code to benchmark using `b.Loop()`, measuring the time it takes for each iteration. Let's run it: ```shellsession highlight={1, 27} lines wrap title=terminal icon="square-terminal" theme={null} $ go test -bench=. [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] GET /albums --> go-gin-benchmarks-example.getAlbums (3 handlers) [GIN-debug] GET /albums/:id --> go-gin-benchmarks-example.getAlbumByID (3 handlers) [GIN-debug] POST /albums --> go-gin-benchmarks-example.postAlbums (3 handlers) [GIN] 2025/09/19 - 17:56:36 | 200 | 1.959µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 2.125µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 2µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 2.666µs | GET "/albums" ... A LOT OF THOSE LINES ... [GIN] 2025/09/19 - 17:56:36 | 200 | 2.084µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 2µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 2.042µs | GET "/albums" [GIN] 2025/09/19 - 17:56:36 | 200 | 1.958µs | GET "/albums" goos: darwin goarch: arm64 pkg: go-gin-benchmarks-example cpu: Apple M1 Pro BenchmarkGetAlbums-10 54296 21328 ns/op PASS ok go-gin-benchmarks-example 1.460s ``` It works! The first benchmark is running, and the results are displayed. Let's dive in the numbers: * First we can see in the `[GIN]` logs that our request takes roughly 2µs on average. This is an interesting reference point but actually not the source of truth we'll use for our benchmark results. * The benchmark name `BenchmarkGetAlbums-10` has the `-10` suffix, which means it ran on 10 CPU cores. * It ran **54,296 times**, taking an average of **21,328 ns per operation** (which translates to **21.328 µs per request** in our case). * Overall, benchmarking this module took **1.460 seconds**. However, seeing the output of this first run, we can note a few things that are not ideal: * There is a significant overhead in our measurement: we end up measuring \~21µs per request, but the reported timing of a single request by the router is \~2 µs. That means 90% of what we measure in not what's happening in the router. * As mentioned in the logs, Gin is running in debug mode here: since we want to measure something as closely related to what happens in prod, we should run the router in release mode to measure realistic performance. * The output of the router is very verbose, while it's very convenient for integration or unit tests, it's harmful to have all those logs in the benchmark since we're also measuring STDOUT performance here. ### Configuring the router for benchmarking To fix those issues, we can create a helper function to set up the router specifically for benchmarking: ```go theme={null} func setupBenchmarkRouter() *gin.Engine { // Set Gin to release mode for benchmarks gin.SetMode(gin.ReleaseMode) // Discard all output during benchmarks to only preserve benchmark output gin.DefaultWriter = io.Discard return setupRouter() } ``` And here we go: ```shellsession title=terminal icon="square-terminal" theme={null} $ go test -bench=. goos: darwin goarch: arm64 pkg: go-gin-benchmarks-example cpu: Apple M1 Pro BenchmarkGetAlbums-10 381921 3060 ns/op PASS ok go-gin-benchmarks-example 1.495s ``` And now, we can first see that the output is way cleaner and simpler to understand what's going on during the benchmarking process! Also, we can see that the overhead is way lower, and the reported timing (3.060µs) is much closer to the actual time spent in the router, putting us in a much better position to make decisions about performance improvements. Still, there's one last thing we can do to improve the benchmark results. ### Optimizing the Response Writer Since the beginning, we reused the `httptest.NewRecorder()` inspired by the Gin testing example to get a response writer. This is necessary because Gin's `ServeHTTP` method requires an `http.ResponseWriter` to handle the HTTP response, and `httptest.NewRecorder()` provides a concrete implementation that captures the response for inspection. This is convenient but introduces some extra costs not really worth measuring in our case: * Extra useless allocations * JSON buffering to capture the responses * JSON processing to handle the responses Let's replace it with a dummy writer that discards all data: ```go theme={null} type DummyResponseWriter struct{} func (d *DummyResponseWriter) Header() http.Header { return http.Header{} } func (d *DummyResponseWriter) Write(data []byte) (int, error) { return len(data), nil } func (d *DummyResponseWriter) WriteHeader(statusCode int) { } ``` And now we can use it instead of the `httptest.NewRecorder()`: ```go theme={null} func BenchmarkGetAlbums(b *testing.B) { router := SetupRouter() req, _ := http.NewRequest("GET", "/albums", nil) w := httptest.NewRecorder() // [!code --] w := new(DummyResponseWriter) // [!code ++] for b.Loop() { router.ServeHTTP(w, req) } } ``` And voilà, **this implementation:** * Discards all data instead of storing it * No allocations - just returns success without buffering * Minimal CPU overhead - simple length calculation for Write() ### The benchmark factory Now, we can combine all those changes into a single helper function to create a benchmark for a given request: ```go theme={null} func benchmarkRequest(b *testing.B, req *http.Request) { router := setupBenchmarkRouter() w := new(DummyResponseWriter) for b.Loop() { router.ServeHTTP(w, req) } } ``` And then use it, making our benchmarks way cleaner and simpler to understand: ```go theme={null} func BenchmarkGetAlbums(b *testing.B) { req, _ := http.NewRequest("GET", "/albums", nil) benchmarkRequest(b, req) } ``` ### Scaling up the benchmarking suite Now, let's add some more benchmarks for the other routes and scenarios: ```go api_test.go icon=golang theme={null} package api import ( "bytes" "encoding/json" "net/http" "strings" "testing" ) func BenchmarkGetAlbums(b *testing.B) { req, _ := http.NewRequest("GET", "/albums", nil) benchmarkRequest(b, req) } func BenchmarkGetAlbumByIDExists(b *testing.B) { req, _ := http.NewRequest("GET", "/albums/1", nil) benchmarkRequest(b, req) } func BenchmarkGetAlbumByIDNotFound(b *testing.B) { req, _ := http.NewRequest("GET", "/albums/999", nil) benchmarkRequest(b, req) } func BenchmarkPostAlbumsValid(b *testing.B) { newAlbum := album{ ID: "4", Title: "Kind of Blue", Artist: "Miles Davis", Price: 29.99, } albumJSON, _ := json.Marshal(newAlbum) req, _ := http.NewRequest("POST", "/albums", bytes.NewBuffer(albumJSON)) req.Header.Set("Content-Type", "application/json") benchmarkRequest(b, req) } func BenchmarkPostAlbumsInvalidJSON(b *testing.B) { invalidJSON := `{"id": "5", "title": "Invalid Album", "artist": "Test Artist", "price": "invalid_price"}` req, _ := http.NewRequest("POST", "/albums", strings.NewReader(invalidJSON)) req.Header.Set("Content-Type", "application/json") benchmarkRequest(b, req) } func BenchmarkPostAlbumsEmptyBody(b *testing.B) { req, _ := http.NewRequest("POST", "/albums", strings.NewReader("")) req.Header.Set("Content-Type", "application/json") benchmarkRequest(b, req) } ``` And here's the output: ```shellsession title=terminal icon="square-terminal" theme={null} $ go test -bench=. -benchtime=5s goos: darwin goarch: arm64 pkg: go-gin-benchmarks-example cpu: Apple M1 Pro BenchmarkGetAlbums-10 1964038 3135 ns/op BenchmarkGetAlbumByIDExists-10 3295057 1804 ns/op BenchmarkGetAlbumByIDNotFound-10 3382840 1728 ns/op BenchmarkPostAlbumsValid-10 4789922 1257 ns/op BenchmarkPostAlbumsInvalidJSON-10 4762603 1337 ns/op BenchmarkPostAlbumsEmptyBody-10 4605133 1307 ns/op PASS ok go-gin-benchmarks-example 36.674s ``` We're using the `-benchtime=5s` flag to run the benchmarks for 5 seconds each, making sure we get enough samples to get a good estimate of the performance. And now, almost all branches of the API are covered by this set of benchmarks! ## Running the benchmarks in CI Local benchmarks are excellent for development iteration, but running benchmarks in CI provides consistency and automation that local benchmarking can't match: * **Consistent hardware**: CI runners eliminate the "works on my machine" problem. Your laptop's thermal throttling, background processes, and varying load create noise that masks real performance changes. * **Automated detection**: Catch performance regressions before they reach production. Every PR gets benchmarked automatically, making performance a first-class concern like tests. * **Historical tracking**: Build a performance timeline across commits. Spot trends, identify when regressions were introduced, and validate that optimizations actually worked. The CodSpeed GitHub Action will automatically run the benchmarks with instrumentation and upload the results to CodSpeed. It mostly boils down to this: Two important things to note: * We're using the `codspeed-macro` runners to run the benchmarks on an optimized and isolated CI machine, removing noise from virtualization and shared resources. Check out the [Macro Runners](/docs/features/macro-runners) page for more details. * We're again using the `-benchtime=5s` flag to run the benchmarks for 5 seconds each, making sure we get enough samples to get a good estimate of the performance. Feel free to change it to your needs. This example is for GitHub Actions, but you can use CodSpeed with any other CI providers. Check out the [CI integration docs](/docs/integrations/ci) for more details on the CI integration. And now each pull request will automatically run the benchmarks, and you'll be able to see the results in the CodSpeed dashboard. For example, let's use SQLite instead of an in-memory database to see the performance impact (check out the code [here](https://github.com/CodSpeedHQ/go-gin-benchmarks-example/pull/1)): Github Comment with the benchmark results

Github Comment with the benchmark results

This also emits a status check on the pull request: Status check on the PR which can be used to prevent merging regressions

Now, we can analyze the performance report to see the details of the performance regression:

Here, we clearly see the dramatic performance regression in `getAlbums` (in red) introduced by the new (in blue) `database/sql` usage. However, we can see that there is almost no difference in the `postAlbum` benchmark: Impact on the BenchmarkPostAlbumsValid benchmark: -2%

Impact on the BenchmarkPostAlbumsValid benchmark: -2%

Diving deeper, we can see that actually the changes on the `postAlbum` function are important but only impact 0.1% of the total time: Flamegraph inspector for the postAlbum function

Flamegraph inspector for the postAlbum function

In the end, using SQLite would impact primarily the read operations without any impact on the write operations. Check out [the pull request](https://github.com/CodSpeedHQ/go-gin-benchmarks-example/pull/1) and [the CodSpeed performance report](https://app.codspeed.io/CodSpeedHQ/go-gin-benchmarks-example/branches/feature%2Fsqlite-database) for more details. ## Next steps Now that you've seen how to benchmark a Go Gin API, you can start benchmarking your own code! Here are some useful resources: The example GitHub repository for this guide with all the code and Pull Requests. All the details about CodSpeed's Go integration. Learn more about the Walltime instrument and how to use it. Learn more about profiling and how to read flame graphs. ## Benchmarking Cookbook Gin benchmark utilities: ```go gin_benchmark_utils.go icon=golang theme={null} package api import ( "io" "net/http" "testing" "github.com/gin-gonic/gin" ) // DummyResponseWriter implements http.ResponseWriter but discards all data // This eliminates overhead from httptest.NewRecorder() in benchmarks type DummyResponseWriter struct{} func (d *DummyResponseWriter) Header() http.Header { return http.Header{} } func (d *DummyResponseWriter) Write(data []byte) (int, error) { // Discard all data - do nothing return len(data), nil } func (d *DummyResponseWriter) WriteHeader(statusCode int) { // Do nothing - discard status code } // setupBenchmarkRouter wraps the main setupRouter with benchmark mode configuration func setupBenchmarkRouter() *gin.Engine { // Set Gin to release mode for benchmarks gin.SetMode(gin.ReleaseMode) // Discard all output during benchmarks to only preserve benchmark output gin.DefaultWriter = io.Discard return setupRouter() } func benchmarkRequest(b *testing.B, req *http.Request) { router := setupBenchmarkRouter() w := new(DummyResponseWriter) for b.Loop() { router.ServeHTTP(w, req) } } ``` Sample usage: ```go api_test.go icon=golang theme={null} package api import ( "net/http" "testing" ) func BenchmarkGetAlbums(b *testing.B) { req, _ := http.NewRequest("GET", "/albums", nil) benchmarkRequest(b, req) } ``` See Go integration notes and compatibility in the [Go benchmarks guide](/docs/benchmarks/go). # Choosing the Correct Python Benchmarking Strategy Source: https://codspeed.io/docs/guides/choosing-the-correct-python-benchmarking-strategy Explore and compare different Python benchmarking approaches—from command-line tools to integrated test frameworks—to find the best fit for your workflow. Performance isn't just about making your code work—it's about making it work well. When you're building Python applications, understanding exactly how fast your code runs becomes the difference between software that feels responsive and software that makes users reach for the coffee while they wait. Benchmarking Python code isn't rocket science, but it does require the right tools and techniques. Let's explore four powerful approaches that will transform you from someone who hopes their code is fast to someone who knows exactly how fast it is! If you have a Python performance question, try asking it to [p99.chat](https://p99.chat), the assistant for code performance optimization. It can run, measure, and optimize any given code! ## Starting with the`time` command `time` is only available on UNIX-based systems, so if you're working with Windows, you can skip this first step. Sometimes the simplest tools are the most revealing. The Unix `time` command gives you a bird's-eye view of your script's performance, measuring everything from CPU usage to memory consumption. Let's start with a practical example. Create a script that demonstrates different algorithmic approaches: ```python bench.py theme={null} import sys import random def bubble_sort(arr): """Inefficient but educational sorting algorithm""" n = len(arr) for i in range(n): for j in range(0, n - i - 1): if arr[j] > arr[j + 1]: arr[j], arr[j + 1] = arr[j + 1], arr[j] return arr def quick_sort(arr): """More efficient divide-and-conquer approach""" if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) if __name__ == " __main__": # Generate test data size = int(sys.argv[1]) if len(sys.argv) > 1 else 1000 data = [random.randint(1, 1000) for _ in range(size)] # Pick the algorithm from command line arguments algorithm = sys.argv[2] if len(sys.argv) > 2 else "bubble" if algorithm == "bubble": result = bubble_sort(data.copy()) else: result = quick_sort(data.copy()) print(f"Sorted {len(result)} elements using {algorithm} sort") ``` Now let's see the power of the `time` command in action: ```shellsession title=terminal icon="square-terminal" theme={null} $ time python3 bench.py 5000 bubble Sorted 5000 elements using bubble sort python3 bench.py 5000 bubble 0.89s user 0.01s system 99% cpu 0.910 total $ time python3 bench.py 5000 quick Sorted 5000 elements using quick sort python3 bench.py 5000 quick 0.03s user 0.01s system 75% cpu 0.049 total ``` Look at that dramatic difference! Bubble sort consumed 0.89 seconds of CPU time while quicksort finished in just 0.03 seconds—nearly 30x faster. The 99% CPU utilization for bubble sort shows it's working hard but inefficiently, while quicksort's lower CPU percentage reflects its brief execution time. Different systems format the output slightly differently. On Linux systems for example, you might see: ```shellsession title=terminal icon="square-terminal" theme={null} $ time python bench.py 5000 bubble Sorted 5000 elements using bubble sort real 0m0.825s user 0m0.806s sys 0m0.022s $ time python bench.py 5000 quick Sorted 5000 elements using quick sort real 0m0.091s user 0m0.054s sys 0m0.040s ``` The `time` command reveals three crucial metrics: * **Real time** (`real` or `total`): Wall-clock time from start to finish * **User time** (`user`): CPU time spent in user mode (your Python code executing—loops, calculations, memory operations) * **System time** (`sys` or `system`): CPU time spent in kernel mode (system calls, file I/O, memory allocation from the OS) This approach is perfect when you want to understand your script's overall resource consumption, including startup overhead and system interactions. ## Precision Benchmarking with `hyperfine` While `time` gives you the basics, hyperfine transforms benchmarking into a science. It runs multiple iterations, provides statistical analysis, and even generates beautiful comparison charts. After having [installed hyperfine](https://github.com/sharkdp/hyperfine?tab=readme-ov-file#installation), you can get started pretty quickly: ```shellsession title=terminal icon="square-terminal" theme={null} $ hyperfine python bench.py 5000 quick Benchmark 1: python Time (mean ± σ): 19.9 ms ± 2.4 ms [User: 14.0 ms, System: 4.0 ms] Range (min … max): 17.8 ms … 36.5 ms 74 runs ``` Instead of running only once, hyperfine automatically ran your code 74 times and calculated meaningful statistics. That ± 2.4 ms standard deviation tells you how consistent your performance is, a crucial information that a single `time` run can't provide. You can also compare commands with hyperfine: ```shellsession title=terminal icon="square-terminal" theme={null} $ hyperfine "python bench.py 5000 quick" "python bench.py 5000 bubble" Benchmark 1: python bench.py 5000 quick Time (mean ± σ): 28.1 ms ± 2.0 ms [User: 21.6 ms, System: 4.5 ms] Range (min … max): 26.5 ms … 40.4 ms 68 runs Benchmark 2: python bench.py 5000 bubble Time (mean ± σ): 917.0 ms ± 24.6 ms [User: 895.4 ms, System: 8.3 ms] Range (min … max): 899.3 ms … 969.3 ms 10 runs Summary python bench.py 5000 quick ran 32.59 ± 2.52 times faster than python bench.py 5000 bubble ``` Now we're talking! Hyperfine not only confirms our 30x performance difference but quantifies the uncertainty in that measurement. The "± 2.52" tells us the speedup could range from about 30x to 35x. ## Function-Level Precision with `timeit` When you need to focus on specific functions rather than entire scripts, Python's built-in `timeit` module becomes your microscope. It's designed to minimize timing overhead and provide accurate measurements of small code snippets. Here is an example measuring the functions we previously created: ```python time.py theme={null} import timeit from bench import bubble_sort, quick_sort # Generate test data data = [random.randint(1, 1000) for _ in range(5000)] bubble_time = timeit.timeit( lambda: bubble_sort(data.copy()), number=10 ) quick_time = timeit.timeit( lambda: quick_sort(data.copy()), number=10 ) ``` ```shellsession title=terminal icon="square-terminal" theme={null} $ python time.py Bubble sort: 0.6165 seconds Quick sort: 0.0100 seconds Speedup: 31.78x ``` The conclusion remains the same—quicksort dramatically outperforms bubble sort—but notice something interesting about these numbers. Both measurements are significantly smaller than our earlier script-level benchmarks. We've eliminated the noise of Python interpreter startup, module imports, and command-line argument parsing. Now we're measuring pure algorithmic performance, which gives us a clearer picture of what's happening inside our functions. Notice how we use `lambda` functions to wrap our calls—this approach is cleaner than string-based timing and provides better IDE support. The `data.copy()` call ensures each iteration works with fresh data, preventing any side effects from skewing our results. The beauty of `timeit` lies in its surgical precision. While our previous tools measured entire script execution, `timeit` isolates the exact performance characteristics of individual functions. This granular approach becomes invaluable when you're optimizing specific bottlenecks rather than entire applications. ## Create Benchmarks from Existing Test Suites First, let's create proper tests for our sorting functions. The first step is to install the testing library: `pytest`: ```shellsession title=terminal icon="square-terminal" theme={null} $ uv add --dev pytest ``` We recommend using [uv](https://docs.astral.sh/uv/getting-started/installation/) to create a project, you can simply run `uv init` and it will turn your directory into a Python project. Then we can create some tests: ```python test_sort.py theme={null} import random from bench import bubble_sort, quick_sort data = [random.randint(1, 1000) for _ in range(5000)] def test_bubble_sort_performance(): """Benchmark bubble sort with 5000 elements""" result = bubble_sort(data) assert result == sorted(data) def test_quick_sort_performance(): """Benchmark quick sort with 5000 elements""" result = quick_sort(data) assert result == sorted(data) ``` Now let's run those tests: ```shellsession title=terminal icon="square-terminal" theme={null} $ uv run pytest test_sort.py ============================== test session starts =============================== platform darwin -- Python 3.13.0, pytest-8.4.0, pluggy-1.6.0 rootdir: /private/tmp/bench configfile: pyproject.toml collected 2 items test_sort.py .. [100%] =============================== 2 passed in 0.78s ================================ ``` Great! Your tests validate that both sorting algorithms produce correct results. ### Turning the test cases into benchmarks Now comes the real magic. Install `pytest-codspeed` and transform these correctness tests into performance benchmarks with minimal changes: ```shellsession title=terminal icon="square-terminal" theme={null} $ uv add --dev pytest-codspeed ``` Update your tests, adding the `benchmark` fixture as a parameter and using it to wrap the execution of the sort algorithm: ```python test_sort.py {6,8,12,14} theme={null} import random from bench import bubble_sort, quick_sort data = [random.randint(1, 1000) for _ in range(5000)] def test_bubble_sort_performance(benchmark): """Benchmark bubble sort with 5000 elements""" result = benchmark(lambda: bubble_sort(data)) assert result == sorted(data) def test_quick_sort_performance(benchmark): """Benchmark quick sort with 5000 elements""" result = benchmark(lambda: quick_sort(data)) assert result == sorted(data) ``` Finally, let's burn the CPU for a bit: ```shellsession title=terminal icon="square-terminal" theme={null} $ uv run pytest --codspeed test_sort.py ============================== test session starts =============================== platform darwin -- Python 3.13.0, pytest-8.4.0, pluggy-1.6.0 codspeed: 3.2.0 (enabled, mode: walltime, timer_resolution: 41.7ns) rootdir: /private/tmp/bench configfile: pyproject.toml plugins: codspeed-3.2.0 collected 2 items test_sort.py .. [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━╋━━━━━━━━━━╋━━━━━━━┫ ┃ test_bubble_sort_performance ┃ 448,626,916ns ┃ 3.1% ┃ 2.74s ┃ 6 ┃ ┃ test_quick_sort_performance ┃ 194,546ns ┃ 9.7% ┃ 3.04s ┃ 1,005 ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━┻━━━━━━━━━━┻━━━━━━━┛ ================================= 2 benchmarked ================================== =============================== 2 passed in 8.94s ================================ ``` Now we're seeing the true power of statistical benchmarking! Look at those numbers—bubble sort clocked in at 448 milliseconds while quicksort blazed through in just 194 microseconds. That's a staggering 2,300x performance difference. Notice how `pytest-codspeed` automatically determined the optimal number of iterations: 6 runs for the slow bubble sort versus 1,005 runs for the lightning-fast quicksort. This intelligent adaptation ensures statistical significance regardless of your algorithm's performance characteristics. Learn more about the plugin in [the `pytest-codspeed` reference](/docs/reference/pytest-codspeed). ### The Foundation for Continuous Performance Testing What makes this approach transformative isn't just the numbers—it's how easily it integrates into your existing workflow. You've just created the foundation for a performance monitoring system that can run locally during development and automatically in CI/CD pipelines. This is the first step toward performance-conscious development. While you can now validate performance locally, the real power emerges when you integrate these benchmarks into your continuous integration pipeline. Every pull request becomes a performance checkpoint, every deployment includes performance validation, and performance regressions are caught before they reach production. The CodSpeed ecosystem makes this transition seamless—from local development to continuous testing in just a few configuration steps. Check out this guide: ## Choosing Your Benchmarking Strategy Each tool serves a specific purpose in your performance toolkit: * **Use the** `time` **command when** you need a quick sanity check of overall script performance or want to understand system resource usage. It's perfect for comparing different implementations at the application level. * **Choose** `hyperfine` **when** you need statistical rigor for command-line tools or want to track performance across different input parameters. Its warmup runs and statistical analysis make it ideal for detecting small performance changes. * **Reach for** `timeit` **when** you're optimizing specific functions or comparing different algorithmic approaches. Its focus on eliminating timing overhead makes it perfect for micro-benchmarks. * **Implement** `pytest-codspeed` **when** performance becomes a first-class concern in your development process. It transforms performance testing from an afterthought into an integral part of your test suite. ## Suggested Reading A more advanced resource on writing benchmarks with pytest-codspeed. A more advanced resource on continuous performance testing in Python ## Resources * [`pytest-codspeed — a pytest benchmarking plugin`](/docs/reference/pytest-codspeed) * [`hyperfine — a CLI benchmarking tool`](https://github.com/sharkdp/hyperfine) * [`timeit — Measure execution time of small code snippets`](https://docs.python.org/3.13/library/timeit.html) * [`time (Unix) — Wikipedia`](https://en.wikipedia.org/wiki/Time_$Unix$) # How to Benchmark C++ with Google Benchmark? Source: https://codspeed.io/docs/guides/how-to-benchmark-cpp-with-google-benchmark Learn how to measure the performance of your C++ code by writing and running benchmarks locally and continuously in CI to catch regressions. ## Choosing our Benchmarking Strategy We are going to use [`google_benchmark`](https://github.com/google/benchmark), the standard C++ benchmarking library maintained by Google. It's widely adopted across the C++ ecosystem, supports fixtures and parameterized benchmarks with statistical analysis, and works with CMake, Bazel, and other build systems. This guide uses [CMake](https://cmake.org/) as the build system. If you're using [Bazel](https://bazel.build/), check out the [Bazel integration documentation](/docs/benchmarks/cpp#bazel) for build instructions. ## Your First Benchmark Let's start by creating a benchmark for a recursive Fibonacci function to see how we can measure computational performance. ### Project Setup First, create a basic project structure: ```bash icon="square-terminal" theme={null} mkdir my_project && cd my_project mkdir benchmarks ``` ### Writing the Benchmark Create a new file `benchmarks/main.cpp`: ```cpp benchmarks/main.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} #include // Recursive Fibonacci function to benchmark static long long fibonacci(int n) { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); } // Define the benchmark static void BM_Fibonacci(benchmark::State &state) { // Use a volatile variable to prevent compile-time optimization volatile int n = 30; // This loop runs multiple times to get accurate measurements for (auto _ : state) { // Prevent compiler from optimizing away the computation auto result = fibonacci(n); benchmark::DoNotOptimize(result); } } // Register the benchmark, specifying the time unit as milliseconds for better // readability BENCHMARK(BM_Fibonacci)->Unit(benchmark::kMillisecond); // Entrypoint that runs all registered benchmarks BENCHMARK_MAIN(); ``` A few things to note: * `volatile int n = 30` prevents the compiler from computing the result at compile time * `benchmark::State& state` provides the benchmark loop that runs your code multiple times * `for (auto _ : state)` is where your actual benchmark code goes - this loop is timed * `benchmark::DoNotOptimize()` prevents the compiler from optimizing away the result * `BENCHMARK()` registers your function as a benchmark * `->Unit(benchmark::kMillisecond)` displays results in milliseconds for better readability as by default it's in nanoseconds * `BENCHMARK_MAIN()` provides the entry point that discovers and runs all benchmarks To learn more about preventing compiler optimizations, check out the [Prevent Compiler Optimizations](#prevent-compiler-optimizations) section below. ### Configuration with CMake Create a `CMakeLists.txt` file in the `benchmarks/` folder: ```cmake benchmarks/CMakeLists.txt theme={null} cmake_minimum_required(VERSION 3.14) project(my_benchmarks VERSION 0.1.0 LANGUAGES CXX) # Use C++17 (or your preferred version) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) # Enable optimizations with debug symbols for profiling set(CMAKE_BUILD_TYPE RelWithDebInfo) # Fetch google_benchmark from CodSpeed's repository include(FetchContent) FetchContent_Declare( google_benchmark GIT_REPOSITORY https://github.com/CodSpeedHQ/codspeed-cpp SOURCE_SUBDIR google_benchmark GIT_TAG main ) set(BENCHMARK_DOWNLOAD_DEPENDENCIES ON) FetchContent_MakeAvailable(google_benchmark) # Create the benchmark executable add_executable(bench main.cpp) # Link against google_benchmark target_link_libraries(bench benchmark::benchmark) ``` Key configuration points: * `CMAKE_BUILD_TYPE RelWithDebInfo` enables optimizations with debug symbols for accurate profiling * We use CodSpeed's fork of `google_benchmark` which adds performance measurement capabilities and CI integration * `BENCHMARK_DOWNLOAD_DEPENDENCIES ON` allows google\_benchmark to download its dependencies ### Building and Running the Benchmark Build your benchmark: ```bash icon="square-terminal" theme={null} cd benchmarks mkdir build && cd build cmake .. make ``` You should see output like: ```shellsession title=terminal icon="square-terminal" theme={null} -- The CXX compiler identification is GNU 14.2.1 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Configuring done (8.6s) -- Generating done (0.1s) -- Build files have been written to: /home/user/my_project/benchmarks/build [ 1%] Building CXX object ... ... [100%] Built target bench ``` Now run your benchmark: ```bash icon="square-terminal" theme={null} ./bench ``` You should see output like this: ```shellsession title=terminal icon="square-terminal" theme={null} 2025-12-01T17:24:27+01:00 Running ./bench Run on (8 X 24 MHz CPU s) CPU Caches: L1 Data 64 KiB L1 Instruction 128 KiB L2 Unified 4096 KiB (x8) Load Average: 8.47, 7.96, 7.04 ------------------------------------------------------- Benchmark Time CPU Iterations ------------------------------------------------------- BM_Fibonacci 2.74 ms 2.65 ms 271 ``` Congratulations! You've created your first C++ benchmark. The output shows that computing `fibonacci(30)` takes about 2.74 milliseconds on average. **Understanding the results:** * **Time**: Wall-clock time per iteration (lower is better) * **CPU**: CPU time per iteration (accounts for multi-threading) * **Iterations**: How many times the benchmark ran to get reliable measurements ## Benchmarking with Parameters So far, we've only tested our function with a single input (n=30). But what if we want to see how performance changes with different input sizes? This is where `DenseRange` comes in. Let's add a parameterized benchmark to test Fibonacci with various input sizes. Update your `main.cpp` to include: ```cpp benchmarks/main.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // Define the benchmark with a parameter static void BM_Fibonacci_DenseRange(benchmark::State &state) { // Get the input value from the benchmark parameter volatile int n = state.range(0); for (auto _ : state) { auto result = fibonacci(n); benchmark::DoNotOptimize(result); } } // Test Fibonacci with inputs from 15 to 35 in steps of 5 BENCHMARK(BM_Fibonacci_DenseRange) ->DenseRange(15, 35, 5) // Test inputs 15, 20, 25, 30, 35 ->Unit(benchmark::kMillisecond); ``` Now `state.range(0)` gives us the input parameter, and `DenseRange(15, 35, 5)` tells the benchmark to run with inputs 15, 20, 25, 30, and 35. Rebuild and run: ```bash icon="square-terminal" theme={null} make ./bench --benchmark_filter=Fibonacci_DenseRange ``` We used the `--benchmark_filter` flag to only run benchmarks matching `Fibonacci_DenseRange`. This is useful when you have many benchmarks and want to focus on a subset. Learn more about [benchmark a subset of benchmarks](https://google.github.io/benchmark/user_guide.html#running-a-subset-of-benchmarks). You should see output like: ```shellsession title=terminal icon="square-terminal" theme={null} --------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------- BM_Fibonacci_DenseRange/15 0.002 ms 0.002 ms 380948 BM_Fibonacci_DenseRange/20 0.022 ms 0.021 ms 33413 BM_Fibonacci_DenseRange/25 0.276 ms 0.234 ms 3050 BM_Fibonacci_DenseRange/30 2.62 ms 2.59 ms 278 BM_Fibonacci_DenseRange/35 28.1 ms 28.0 ms 25 ``` Notice how the execution time grows exponentially with the input size, clearly demonstrating the O(2^n) complexity of the recursive Fibonacci algorithm. This is the power of parameterized benchmarks – they help you understand how your code scales with different inputs. #### Multiple Arguments What if your function takes multiple parameters? For example, let's benchmark the performance of `std::string::find()` with varying text and pattern sizes. Let's add a new benchmark to `main.cpp`: ```cpp benchmarks/main.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // ... (previous code) ... #include static void BM_StringFind(benchmark::State& state) { size_t string_size = state.range(0); size_t pattern_size = state.range(1); // Setup std::string text(string_size, 'a'); std::string pattern(pattern_size, 'b'); // Place pattern near the end for worst-case scenario text.replace(string_size - pattern_size, pattern_size, pattern); // Benchmark for (auto _ : state) { auto pos = text.find(pattern); benchmark::DoNotOptimize(pos); } } // Benchmark different combinations of text and pattern sizes using ArgsProduct BENCHMARK(BM_StringFind) ->ArgsProduct({ {1000, 10000, 100000}, // Text sizes {50, 500} // Pattern sizes }); ``` The `ArgsProduct()` function creates benchmarks for all combinations of the provided argument lists. In this case, it generates 6 benchmarks (3 text sizes × 2 pattern sizes), letting you analyze how both parameters affect performance. Here is the output when you run this benchmark: ```shellsession title=terminal icon="square-terminal" theme={null} ./bench --benchmark_filter=StringFind ... ------------------------------------------------------------------- Benchmark Time CPU Iterations ------------------------------------------------------------------- BM_StringFind/1000/50 28.7 ns 28.0 ns 25077651 BM_StringFind/10000/50 337 ns 237 ns 3123341 BM_StringFind/100000/50 2157 ns 2066 ns 287731 BM_StringFind/1000/500 30.3 ns 28.6 ns 24820407 BM_StringFind/10000/500 248 ns 243 ns 2987100 BM_StringFind/100000/500 2075 ns 2031 ns 348384 ``` There are more ways to define parameterized benchmarks, check out the [`google_benchmark` documentation on parameterized benchmarks](https://google.github.io/benchmark/user_guide.html#passing-arguments). ## Benchmarking Only What Matters Sometimes you have expensive setup that shouldn't be included in your benchmark measurements. For example, loading data from a file or creating large data structures. Google Benchmark provides several ways to handle this. ### Fresh Setup per Iteration Let's benchmark a sorting algorithm where we need fresh data for each iteration. We do not want the data generation time to be included in the benchmark. We can exclude it using `PauseTiming()` and `ResumeTiming()`: ```cpp benchmarks/main.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // ... (previous code) ... #include #include #include static void BM_SortVector(benchmark::State &state) { size_t size = state.range(0); std::mt19937 gen(42); // Fixed seed for reproducibility for (auto _ : state) { // Pause timing during setup state.PauseTiming(); // Generate random data (NOT measured) std::vector data(size); std::uniform_int_distribution<> dis(1, 10000); for (size_t i = 0; i < size; ++i) { data[i] = dis(gen); } // Resume timing for the actual work state.ResumeTiming(); // Sort the vector (MEASURED) std::sort(data.begin(), data.end()); benchmark::DoNotOptimize(data.data()); benchmark::ClobberMemory(); } } BENCHMARK(BM_SortVector)->Range(100, 100000)->Unit(benchmark::kMicrosecond); ``` The setup code (generating random data) runs before each iteration but isn't included in the timing. Only the `std::sort()` call is measured. **Use PauseTiming/ResumeTiming sparingly** While `PauseTiming()` and `ResumeTiming()` are useful, they add overhead to your benchmarks. If your setup can be done once before all iterations (like loading a file), use fixtures instead (see next section) for better performance and cleaner code. ### Shared Setup for All Iterations When you can reuse the same data across iterations, fixtures are more efficient. They are a class that defines a setup and teardown process that runs once for all iterations. Both of these methods are not included in the timing. Here is an example where we set up a sorted vector once for all iterations and benchmark binary search on it: ```cpp benchmarks/main.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // Define a fixture class that sets up a random vector for searching class VectorFixture : public benchmark::Fixture { public: std::vector data; // Setup runs once before all iterations void SetUp(const ::benchmark::State &state) { size_t size = state.range(0); std::mt19937 gen(42); // Fixed seed for reproducibility std::uniform_int_distribution<> dis(1, size); data.resize(size); for (size_t i = 0; i < size; ++i) { data[i] = dis(gen); } std::sort(data.begin(), data.end()); } // TearDown runs once after all iterations void TearDown(const ::benchmark::State &) { data.clear(); } }; // Define the BinarySearch benchmark using VectorFixture BENCHMARK_DEFINE_F(VectorFixture, BinarySearch)(benchmark::State &state) { int target = data.size() / 2; for (auto _ : state) { // Only this is measured bool found = std::binary_search(data.begin(), data.end(), target); benchmark::DoNotOptimize(found); } } // Register the fixture benchmark with different vector sizes BENCHMARK_REGISTER_F(VectorFixture, BinarySearch)->Range(1000, 100000); ``` In this example, the `SetUp()` method initializes a sorted vector once before all iterations, and `TearDown()` cleans up afterward. The benchmark only measures the `std::binary_search()` calls. Fixtures use different macros: `BENCHMARK_DEFINE_F` to define and `BENCHMARK_REGISTER_F` to register with parameters. ## Best Practices ### Prevent Compiler Optimizations The C++ compiler is extremely aggressive with optimizations. Always protect your benchmarks: ```cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // ❌ BAD: Compiler might optimize everything away static void BM_Bad(benchmark::State& state) { for (auto _ : state) { int x = 42; int y = x * 2; // Compiler knows this is 84 at compile time } } // ✅ GOOD: Use DoNotOptimize for values static void BM_Good(benchmark::State& state) { for (auto _ : state) { int x = 42; benchmark::DoNotOptimize(x); int y = x * 2; benchmark::DoNotOptimize(y); } } // ✅ BETTER: Use DoNotOptimize and ClobberMemory static void BM_Better(benchmark::State& state) { for (auto _ : state) { int x = 42; benchmark::DoNotOptimize(x); int y = x * 2; benchmark::DoNotOptimize(y); benchmark::ClobberMemory(); } } ``` **Important**: Always use `benchmark::DoNotOptimize()` to prevent the compiler from optimizing away your benchmarks. Without it, the compiler might eliminate the code you're trying to measure, giving you inaccurate results. **Understanding DoNotOptimize vs ClobberMemory:** * `DoNotOptimize(value)` forces the result of a computation to be stored in memory or a register, preventing the compiler from eliminating the computation entirely * `ClobberMemory()` forces the compiler to flush all pending writes to memory, preventing operations with memory side effects from being optimized away * Use `DoNotOptimize()` for return values and computed results * Add `ClobberMemory()` when benchmarking operations that modify memory (like filling vectors or copying data) Learn more in the [Google Benchmark guide on preventing optimization](https://google.github.io/benchmark/user_guide.html#preventing-optimization). ### Keep Benchmarks Deterministic Use fixed seeds for random number generators: ```cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} // ❌ BAD: Non-deterministic results static void BM_NonDeterministic(benchmark::State& state) { std::random_device rd; std::mt19937 gen(rd()); // Different every run! for (auto _ : state) { // ... } } // ✅ GOOD: Deterministic with fixed seed static void BM_Deterministic(benchmark::State& state) { std::mt19937 gen(42); // Fixed seed for (auto _ : state) { // ... } } ``` ### Benchmark Real-World Code In real projects, you'll benchmark functions from your library. Here's a typical structure for a C++ project with benchmarks: ```shellsession title=terminal icon="square-terminal" theme={null} my_project/ ├── CMakeLists.txt ├── include/ │ └── mylib/ │ └── algorithms.hpp ├── src/ │ └── algorithms.cpp └── benchmarks/ └── bench_algorithms.cpp ``` The header `include/mylib/algorithms.hpp` defines your library's API: ```cpp include/mylib/algorithms.hpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} #pragma once #include namespace mylib { std::vector bubble_sort(std::vector arr); } // namespace mylib ``` The implementation `src/algorithms.cpp` contains the actual algorithm: ```cpp src/algorithms.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} #include "mylib/algorithms.hpp" namespace mylib { std::vector bubble_sort(std::vector arr) { size_t n = arr.size(); for (size_t i = 0; i < n; ++i) { for (size_t j = 0; j < n - 1 - i; ++j) { if (arr[j] > arr[j + 1]) { std::swap(arr[j], arr[j + 1]); } } } return arr; } } // namespace mylib ``` The benchmark `benchmarks/bench_algorithms.cpp` tests the bubble sort function: ```cpp benchmarks/bench_algorithms.cpp icon="https://mintcdn.com/codspeed/GDLcp8Ny8u4pFbNX/assets/icons/cpp.svg?fit=max&auto=format&n=GDLcp8Ny8u4pFbNX&q=85&s=420e72f7613b61e7f1961ccdd2e4b9bb" theme={null} #include "mylib/algorithms.hpp" #include #include // Define a fixture class that sets up random data for sorting class SortFixture : public benchmark::Fixture { public: std::vector original_data; // Setup runs once before all iterations void SetUp(const ::benchmark::State &state) { size_t size = state.range(0); std::mt19937 gen(42); // Fixed seed for reproducibility std::uniform_int_distribution<> dis(1, size); original_data.resize(size); for (size_t i = 0; i < size; ++i) { original_data[i] = dis(gen); } } // TearDown runs once after all iterations void TearDown(const ::benchmark::State &) { original_data.clear(); } }; // Define the BubbleSort benchmark using SortFixture BENCHMARK_DEFINE_F(SortFixture, BubbleSort)(benchmark::State &state) { for (auto _ : state) { // Make a copy of the original data for each iteration // Only the sorting is measured, not the copy state.PauseTiming(); std::vector data = original_data; state.ResumeTiming(); auto sorted = mylib::bubble_sort(data); benchmark::DoNotOptimize(sorted.data()); benchmark::ClobberMemory(); } } // Register the fixture benchmark with different data sizes BENCHMARK_REGISTER_F(SortFixture, BubbleSort) ->Range(1000, 100000) ->Unit(benchmark::kMillisecond); BENCHMARK_MAIN(); ``` Update your `CMakeLists.txt` to build both your library and benchmarks: ```cmake CMakeLists.txt theme={null} cmake_minimum_required(VERSION 3.14) project(mylib VERSION 0.1.0 LANGUAGES CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) # Enable optimizations with debug symbols for profiling set(CMAKE_BUILD_TYPE RelWithDebInfo) # Your library add_library(mylib src/algorithms.cpp) target_include_directories(mylib PUBLIC include) # Fetch google_benchmark include(FetchContent) FetchContent_Declare( google_benchmark GIT_REPOSITORY https://github.com/CodSpeedHQ/codspeed-cpp SOURCE_SUBDIR google_benchmark GIT_TAG main ) set(BENCHMARK_DOWNLOAD_DEPENDENCIES ON) FetchContent_MakeAvailable(google_benchmark) # Benchmark executable add_executable(bench_algorithms benchmarks/bench_algorithms.cpp) target_link_libraries(bench_algorithms mylib benchmark::benchmark) ``` You can now build and run your benchmarks with the following commands: ```bash icon="square-terminal" theme={null} mkdir build && cd build cmake .. make ./bench_algorithms ``` This will yield an output similar to: ```shellsession title=terminal icon="square-terminal" theme={null} 2025-12-02T16:50:44+01:00 Running ./bench_algorithms Run on (8 X 24 MHz CPU s) CPU Caches: L1 Data 64 KiB L1 Instruction 128 KiB L2 Unified 4096 KiB (x8) Load Average: 9.83, 10.83, 8.99 ------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------ SortFixture/BubbleSort/1000 0.381 ms 0.321 ms 2219 SortFixture/BubbleSort/4096 5.80 ms 4.97 ms 136 SortFixture/BubbleSort/32768 732 ms 718 ms 1 SortFixture/BubbleSort/100000 10848 ms 9529 ms 1 ``` ## Running Benchmarks Continuously with CodSpeed So far, you've been running benchmarks locally. But local benchmarking has limitations: * **Inconsistent hardware**: Different developers get different results * **Manual process**: Easy to forget to run benchmarks before merging * **No historical tracking**: Hard to spot gradual performance degradation * **No PR context**: Can't see performance impact during code review This is where **CodSpeed** comes in. It runs your benchmarks automatically in CI and provides: * Automated performance regression detection in PRs * Consistent metrics with reliable measurements across all runs * Historical tracking to see performance over time with detailed charts * Flamegraph profiles to see exactly what changed in your code's execution For the full CodSpeed integration reference, see [Writing Benchmarks in C++](/docs/benchmarks/cpp). ### How to set up CodSpeed with google\_benchmark Here's how to integrate CodSpeed with your `google_benchmark` benchmarks using CMake: CodSpeed provides a special build mode that instruments your benchmarks for performance tracking. This is controlled with the `CODSPEED_MODE` CMake flag, which can be set to: * `off`: (default) Regular benchmarking without CodSpeed * `simulation`: CodSpeed CPU simulation mode for CI * `walltime`: Walltime measurements (see [walltime docs](/docs/instruments/walltime)) Build your benchmarks with CodSpeed mode enabled: ```bash icon="square-terminal" theme={null} cd benchmarks mkdir build && cd build cmake -DCODSPEED_MODE=simulation .. make ``` Run the benchmarks to verify everything works: ```bash icon="square-terminal" theme={null} ./bench_algorithms ``` You should see output indicating CodSpeed is enabled: ```shellsession title=terminal icon="square-terminal" theme={null} Codspeed mode: simulation 2025-12-02T17:21:57+01:00 Running ./bench_algorithms Run on (8 X 24 MHz CPU s) CPU Caches: L1 Data 64 KiB L1 Instruction 128 KiB L2 Unified 4096 KiB (x8) Load Average: 9.22, 7.26, 6.71 NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: cpp/benchmarks/bench_algorithms.cpp::BubbleSort[SortFixture][1000] Checked: cpp/benchmarks/bench_algorithms.cpp::BubbleSort[SortFixture][4096] Checked: cpp/benchmarks/bench_algorithms.cpp::BubbleSort[SortFixture][32768] Checked: cpp/benchmarks/bench_algorithms.cpp::BubbleSort[SortFixture][100000] ``` Notice there are no timing measurements in the local output. CodSpeed only captures actual performance data when running in CI. Create a workflow file to run benchmarks on every push and pull request: Once the workflow runs, your pull requests will receive a performance report comment: Pull Request Result

After your benchmarks run in CI, head over to your CodSpeed dashboard to see detailed performance reports, historical trends, and flamegraph profiles for deeper analysis. Profiling Report on CodSpeed

Profiling works out of the box, no extra configuration needed! [Learn more about flamegraphs and how to use them to optimize your code](/docs/features/profiling). **Using Bazel?** If you're using Bazel as your build system, check out the [Bazel integration documentation](/docs/benchmarks/cpp#bazel) for detailed setup instructions with CodSpeed. ## Next Steps Check out these resources to continue your C++ benchmarking journey: Sign up and start tracking your C++ performance in CI }> Explore the full google\_benchmark API reference Learn how to use flamegraphs to optimize your code Explore all of google\_benchmark's features in depth # How to Benchmark Go with the testing Package? Source: https://codspeed.io/docs/guides/how-to-benchmark-go-with-testing Learn how to measure the performance of your Go code using the standard testing package by writing and running benchmarks locally and continuously in CI to catch regressions. ## Why the `testing` Package? Go has first-class benchmarking built into its standard library — no external framework needed. The `testing` package provides `testing.B`, which handles iteration control, timing, memory tracking, sub-benchmarks, and parallel execution out of the box. Every Go developer already has the tools installed. Benchmarks live in `_test.go` files alongside your code, run with `go test`, and integrate with the Go ecosystem's profiling tools (`pprof`, `benchstat`). ## Your First Benchmark Let's start with the simplest possible Go benchmark: measuring a recursive Fibonacci function. ### Setting Up Create a module and two files — the function and its benchmark: ```sh title=terminal icon="square-terminal" theme={null} mkdir my-benchmarks && cd my-benchmarks go mod init example.com/bench ``` ```go title=fib.go icon="golang" theme={null} package bench func Fibonacci(n int) int { if n <= 1 { return n } return Fibonacci(n-1) + Fibonacci(n-2) } ``` ```go title=fib_test.go icon="golang" theme={null} package bench import "testing" func BenchmarkFibonacci(b *testing.B) { for b.Loop() { Fibonacci(20) } } ``` A few things to note: * Benchmark functions must start with `Benchmark` and accept `*testing.B`. * `b.Loop()` (Go 1.24+) controls iteration. Iteration count and timing are handled automatically. * The function lives in a `_test.go` file, just like unit tests. `b.Loop()` was introduced in Go 1.24. It replaces the older `for i := 0; i < b.N; i++` pattern and is more precise — it automatically resets the timer, and the compiler is prevented from optimizing away the loop body. If you are on an older Go version, use the `b.N` pattern instead: ```go title="Legacy pattern (before Go 1.24)" icon="golang" theme={null} func BenchmarkFibonacci(b *testing.B) { for i := 0; i < b.N; i++ { Fibonacci(20) } } ``` ### Running the benchmarks ```shellsession title=terminal icon="square-terminal" highlight={6} theme={null} $ go test -bench=. goos: linux goarch: amd64 pkg: example.com/bench cpu: Intel(R) Xeon(R) Platinum 8488C BenchmarkFibonacci-8 38594 31076 ns/op PASS ok example.com/bench 1.207s ``` The highlighted line breaks down as follows: ```text theme={null} BenchmarkFibonacci-8 38594 31076 ns/op ^ ^ ^ | | | | | time per iteration | number of iterations benchmark name ``` The framework automatically adjusts the iteration count to run for at least 1 second by default. The `-8` suffix on `BenchmarkFibonacci-8` is the [`GOMAXPROCS` value](https://pkg.go.dev/runtime#GOMAXPROCS), defaulting to the number of available CPUs. ## Configuring Your Benchmarks ### Benchmark Duration Control how long each benchmark runs with `-benchtime`: ```sh title=terminal icon="square-terminal" theme={null} go test -bench=. -benchtime=5s ``` You can also specify an exact iteration count: ```sh title=terminal icon="square-terminal" theme={null} go test -bench=. -benchtime=1000x ``` ### Memory Allocation Tracking Add `-benchmem` to report allocation stats, or call `b.ReportAllocs()` inside the benchmark: ```sh title=terminal icon="square-terminal" theme={null} go test -bench=. -benchmem ``` ```shellsession title=terminal icon="square-terminal" theme={null} BenchmarkFibonacci-8 38594 31076 ns/op 0 B/op 0 allocs/op ``` The two extra columns show bytes allocated per operation and number of allocations per operation. These are essential for catching allocation regressions — even if latency stays flat, increased allocations put pressure on the garbage collector. ### Filtering and Skipping Tests Run only benchmarks (skip unit tests) with a regex: ```sh title=terminal icon="square-terminal" theme={null} go test -run='^$' -bench=. ``` Filter to specific benchmarks: ```sh title=terminal icon="square-terminal" theme={null} go test -run='^$' -bench=BenchmarkFibonacci ``` Run benchmarks in a specific package, or recursively across all packages: ```sh title=terminal icon="square-terminal" theme={null} go test -bench=. ./pkg/foo go test -bench=. ./... ``` ### Key CLI Flags Run benchmarks matching the regular expression. Use `-bench=.` for all. Minimum time per benchmark. Accepts a duration (`5s`, `100ms`) or an exact iteration count (`1000x`). Report memory allocation statistics (`B/op`, `allocs/op`). Run each benchmark n times. Use `-count=10` or higher for statistical analysis with `benchstat`. Comma-separated `GOMAXPROCS` values to test with (e.g., `-cpu=1,2,4,8`). Filter tests. Use `-run='^$'` to skip unit tests when benchmarking. Maximum total time for all tests and benchmarks. ## Sub-benchmarks and Table-Driven Patterns ### Sub-benchmarks with `b.Run` Use `b.Run()` to create sub-benchmarks — the standard way to test different inputs or configurations: ```go title=fib_test.go icon="golang" theme={null} func BenchmarkFibonacciSizes(b *testing.B) { sizes := []int{5, 10, 15, 20, 30} for _, n := range sizes { b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) { for b.Loop() { Fibonacci(n) } }) } } ``` ```shellsession title=terminal icon="square-terminal" theme={null} BenchmarkFibonacciSizes/n=5-8 58634750 21.27 ns/op BenchmarkFibonacciSizes/n=10-8 4858176 248.2 ns/op BenchmarkFibonacciSizes/n=15-8 431430 2781 ns/op BenchmarkFibonacciSizes/n=20-8 38892 30986 ns/op BenchmarkFibonacciSizes/n=30-8 312 3823289 ns/op ``` The exponential growth of recursive Fibonacci is clearly visible: n=5 takes 21ns, n=30 takes 3.8ms — a factor of 180,000x. You can filter sub-benchmarks from the command line: ```sh title=terminal icon="square-terminal" theme={null} go test -bench=BenchmarkFibonacciSizes/n=20 ``` ### Comparing Algorithms Sub-benchmarks make algorithm comparison straightforward: ```go title=fib_test.go icon="golang" theme={null} func FibonacciIterative(n int) int { if n <= 1 { return n } a, b := 0, 1 for i := 2; i <= n; i++ { a, b = b, a+b } return b } func BenchmarkAlgorithms(b *testing.B) { for _, n := range []int{10, 20, 30} { b.Run(fmt.Sprintf("recursive/n=%d", n), func(b *testing.B) { for b.Loop() { Fibonacci(n) } }) b.Run(fmt.Sprintf("iterative/n=%d", n), func(b *testing.B) { for b.Loop() { FibonacciIterative(n) } }) } } ``` ```shellsession title=terminal icon="square-terminal" theme={null} BenchmarkAlgorithms/recursive/n=10-8 4782566 251.9 ns/op BenchmarkAlgorithms/iterative/n=10-8 226538564 5.375 ns/op BenchmarkAlgorithms/recursive/n=20-8 38359 31287 ns/op BenchmarkAlgorithms/iterative/n=20-8 159598522 7.423 ns/op BenchmarkAlgorithms/recursive/n=30-8 313 3814719 ns/op BenchmarkAlgorithms/iterative/n=30-8 100000000 10.10 ns/op ``` At n=30, the iterative version is **377,000x faster** than the recursive one (10ns vs 3.8ms). The hierarchical sub-benchmark naming makes it easy to compare across both dimensions. ## Benchmarking Only What Matters ### Excluding Setup with `b.ResetTimer` When your benchmark has expensive one-time setup, use `b.ResetTimer()` to exclude it from measurements: ```go title="Excluding setup" icon="golang" theme={null} func BenchmarkProcess(b *testing.B) { data := expensiveSetup() // NOT measured b.ResetTimer() for i := 0; i < b.N; i++ { process(data) // MEASURED } } ``` With `b.Loop()` (Go 1.24+), the timer is automatically reset on the first iteration, so `b.ResetTimer()` is no longer required unless the setup happens inside the loop body. ### Per-Iteration Setup with Timer Control When each iteration needs fresh data (e.g., sorting an unsorted slice), use `b.StopTimer()` and `b.StartTimer()`: ```go title=sort_test.go icon="golang" theme={null} func BenchmarkSort(b *testing.B) { for _, size := range []int{100, 1000, 10000} { b.Run(fmt.Sprintf("size=%d", size), func(b *testing.B) { original := make([]int, size) for i := range original { original[i] = size - i // reverse-sorted = worst case } b.ResetTimer() for i := 0; i < b.N; i++ { b.StopTimer() data := make([]int, len(original)) copy(data, original) b.StartTimer() sort.Ints(data) } }) } } ``` ```shellsession title=terminal icon="square-terminal" theme={null} BenchmarkSort/size=100-8 5558869 214.7 ns/op BenchmarkSort/size=1000-8 1000000 1096 ns/op BenchmarkSort/size=10000-8 124218 9888 ns/op ``` `b.StopTimer()` and `b.StartTimer()` have overhead. If the per-iteration setup is extremely cheap relative to what you are measuring, the timer overhead may distort results. Use this pattern only when the setup cost is significant. ### Custom Metrics Report domain-specific metrics with `b.ReportMetric()`: ```go title="Custom metrics" icon="golang" theme={null} func BenchmarkCustomMetrics(b *testing.B) { var compares int64 for b.Loop() { s := []int{5, 4, 3, 2, 1} slices.SortFunc(s, func(a, b int) int { compares++ return cmp.Compare(a, b) }) } b.ReportMetric(float64(compares)/float64(b.N), "compares/op") } ``` Use `b.SetBytes(n)` to report throughput in MB/s for I/O-bound benchmarks: ```go title="Throughput reporting" icon="golang" theme={null} func BenchmarkRead(b *testing.B) { b.SetBytes(1024) // 1KB per operation for b.Loop() { readData(buf) } } // Output includes: 3125.00 MB/s ``` ## Parallel Benchmarks Use `b.RunParallel()` to benchmark code under concurrent load. This creates `GOMAXPROCS` goroutines and distributes iterations among them: ```go title=fib_test.go icon="golang" theme={null} func BenchmarkFibonacciParallel(b *testing.B) { b.RunParallel(func(pb *testing.PB) { for pb.Next() { Fibonacci(20) } }) } ``` ```shellsession title=terminal icon="square-terminal" theme={null} BenchmarkFibonacciParallel-8 211784 5754 ns/op ``` Compare this with the sequential result (31076 ns/op) — the parallel version runs \~5x faster on 8 cores, showing that this CPU-bound workload scales well across threads. Do not call `b.StopTimer()`, `b.StartTimer()`, or `b.ResetTimer()` inside `b.RunParallel`. They have global effect and will corrupt measurements. Each goroutine must maintain its own local state. Use `b.SetParallelism(n)` to increase concurrency beyond `GOMAXPROCS` for I/O-bound workloads: ```go title="High concurrency" icon="golang" theme={null} func BenchmarkHTTPConcurrent(b *testing.B) { b.SetParallelism(4) // 4 * GOMAXPROCS goroutines b.RunParallel(func(pb *testing.PB) { for pb.Next() { makeHTTPRequest() } }) } ``` ## Avoiding Common Pitfalls ### Compiler Dead Code Elimination If a computation's result is unused, the Go compiler may eliminate it entirely. This is the single most common source of misleading benchmark results. ```go title="Dead code elimination" icon="golang" theme={null} // BAD: result is unused — compiler may eliminate the call entirely func BenchmarkBroken(b *testing.B) { for i := 0; i < b.N; i++ { Fibonacci(20) // may report ~0 ns/op } } // GOOD: b.Loop() prevents the compiler from optimizing away the body func BenchmarkCorrect(b *testing.B) { for b.Loop() { Fibonacci(20) } } ``` If you must use the `b.N` pattern (Go \< 1.24), pass the result to `runtime.KeepAlive` so the compiler treats it as observed: ```go title="runtime.KeepAlive pattern" icon="golang" theme={null} func BenchmarkCorrect(b *testing.B) { var r int for i := 0; i < b.N; i++ { r = Fibonacci(20) } runtime.KeepAlive(r) } ``` ### Do Not Use `b.N` as Input Using `b.N` as a function parameter means the workload grows with the iteration count. The benchmark never converges and reports meaningless numbers: ```go title="b.N misuse" icon="golang" theme={null} // BAD: workload grows with b.N — benchmark never converges func BenchmarkBad(b *testing.B) { for i := 0; i < b.N; i++ { Fibonacci(i) // i grows, each iteration is slower } } // GOOD: fixed input func BenchmarkGood(b *testing.B) { for b.Loop() { Fibonacci(20) } } ``` ### Keep Benchmarks Deterministic Use fixed seeds for random data: ```go title="Deterministic setup" icon="golang" theme={null} func BenchmarkSort(b *testing.B) { rng := rand.New(rand.NewSource(42)) // fixed seed data := make([]int, 1000) for i := range data { data[i] = rng.Intn(1000) } b.ResetTimer() // ... } ``` ## Comparing Results with `benchstat` Raw benchmark numbers are noisy. Use [`benchstat`](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) to compare results with statistical rigor. ### Installation ```sh title=terminal icon="square-terminal" theme={null} go install golang.org/x/perf/cmd/benchstat@latest ``` ### Workflow Run benchmarks multiple times (at least 10) to collect enough samples: ```sh title=terminal icon="square-terminal" theme={null} go test -run='^$' -bench=. -count=10 > old.txt ``` Make your changes, then run again: ```sh title=terminal icon="square-terminal" theme={null} go test -run='^$' -bench=. -count=10 > new.txt ``` Compare with `benchstat`: ```sh title=terminal icon="square-terminal" theme={null} $ benchstat old.txt new.txt │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ Fibonacci-8 30.96µ ± 0% 30.99µ ± 0% ~ (p=0.841 n=10) ``` The columns report: * **± 0%**: the 95% confidence interval. Lower means more stable results. * **\~ (p=0.841)**: no statistically significant difference. The p-value is from a [Mann-Whitney U-test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test); values below 0.05 indicate a real change. * **n=10**: the sample count. Use `-count=10` or higher for reliable results. ## Profiling with `pprof` Go benchmarks integrate directly with the `pprof` profiler. Generate profiles while benchmarking: Write CPU profile. Shows where time is spent. Write memory allocation profile. Shows where allocations happen. Write goroutine blocking profile. Shows where goroutines wait. Write mutex contention profile. Shows lock contention hotspots. Generate and analyze a CPU profile: ```sh title=terminal icon="square-terminal" theme={null} go test -run='^$' -bench=BenchmarkFibonacci -cpuprofile=cpu.prof go tool pprof -http=:8080 cpu.prof ``` This opens an interactive web UI with flamegraphs, call graphs, and source-level annotations. Collect only one profile type at a time for accuracy — profiling itself has overhead that can distort other measurements. ## Best Practices #### Run on Idle Machines Close background processes, avoid running on battery, and disable CPU throttling when collecting benchmark data. Noise from other processes can mask real performance changes. #### Use `-count` and `benchstat` for Decisions Never eyeball raw `ns/op` numbers to decide if a change helped. Run with `-count=10` and use `benchstat` to test for statistical significance. With \~20 benchmarks at alpha=0.05, expect \~1 false positive. #### Track Memory Alongside Latency Always use `-benchmem` or `b.ReportAllocs()`. Even if latency stays flat, increased allocations put pressure on the garbage collector and cause latency spikes under production load. #### Use Sub-Benchmarks for Input Variation Table-driven sub-benchmarks let you test across input sizes, data shapes, and configurations in a single benchmark function. They also enable filtering from the command line. ## Running Benchmarks Continuously with CodSpeed So far, you've been running benchmarks locally. But local benchmarking has limitations: * **Inconsistent hardware**: Different developers get different results * **Manual process**: Easy to forget to run benchmarks before merging * **No historical tracking**: Hard to spot gradual performance degradation * **No PR context**: Can't see performance impact during code review This is where **CodSpeed** comes in. It runs your benchmarks automatically in CI and provides: * Automated performance regression detection in PRs * Consistent metrics with reliable measurements across all runs * Historical tracking to see performance over time with detailed charts * Flamegraph profiles to see exactly what changed in your code's execution For the full CodSpeed integration reference, see [Writing Benchmarks in Go](/docs/benchmarks/go). ### How to Set Up CodSpeed Here's how to integrate CodSpeed with your Go benchmarks: Create a workflow file to run benchmarks on every push and pull request. Once the workflow runs, your pull requests will receive a performance report comment: Pull Request Result

After your benchmarks run in CI, head over to your CodSpeed dashboard to see detailed performance reports, historical trends, and flamegraph profiles for deeper analysis. Profiling Report on CodSpeed

Profiling works out of the box, no extra configuration needed! [Learn more about flamegraphs and how to use them to optimize your code](/docs/features/profiling). ## Next Steps Check out these resources to continue your Go benchmarking journey: Sign up and start tracking your Go performance in CI CodSpeed's Go integration reference and compatibility notes A hands-on guide to benchmarking a real HTTP API with Gin Learn how to use flamegraphs to optimize your code # How to Benchmark Java with JMH? Source: https://codspeed.io/docs/guides/how-to-benchmark-java-with-jmh Learn how to measure the performance of your Java code using JMH (Java Microbenchmark Harness) by writing and running benchmarks locally and continuously in CI to catch regressions. ## Why JMH? This guide uses [JMH (Java Microbenchmark Harness)](https://github.com/openjdk/jmh), the standard benchmarking framework for the JVM. JMH is developed as part of the OpenJDK project by the same engineers who build the JVM itself, so it understands JVM internals like JIT compilation, dead code elimination, and constant folding that can silently invalidate naive benchmarks. It handles warmup, fork isolation, and statistical analysis out of the box so you can focus on writing the code you want to measure. This guide covers [Maven](https://maven.apache.org/) and [Gradle](https://github.com/melix/jmh-gradle-plugin). JMH also works with [SBT](https://github.com/ktoso/sbt-jmh). ## Your First Benchmark Let's start with the simplest possible JMH benchmark: a single method that measures how fast a recursive Fibonacci function runs. ### Project Setup The recommended way to use JMH with Maven is through its archetype, which generates a project pre-configured with the annotation processor and uber-JAR packaging: ```sh title=terminal icon="square-terminal" theme={null} mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=com.example \ -DartifactId=my-benchmarks \ -Dversion=1.0 ``` This creates a `my-benchmarks/` directory with the following structure: The generated `pom.xml` includes `jmh-core` (the runtime library), `jmh-generator-annprocess` (the annotation processor that generates benchmark harness code at compile time), and `maven-shade-plugin` (packages everything into a single executable `benchmarks.jar`). Create a new project directory and add the [`jmh-gradle-plugin`](https://github.com/melix/jmh-gradle-plugin): ```groovy build.gradle icon="java" theme={null} plugins { id 'java' id 'me.champeau.jmh' version '0.7.3' } repositories { mavenCentral() } jmh { jmhVersion = '1.37' } ``` Then create the benchmark source directory: ```sh title=terminal icon="square-terminal" theme={null} mkdir -p src/jmh/java/com/example ``` The plugin handles the annotation processor and uber-JAR generation automatically. Do not add `jmh-core` to an existing project without the annotation processor. JMH needs to generate synthetic benchmark code at compile time. The archetype (Maven) and plugin (Gradle) handle this correctly. ### Writing the Benchmark The archetype generates a stub `MyBenchmark.java` with an empty `@Benchmark` method. Open `src/main/java/com/example/MyBenchmark.java` and replace its contents with: ```java src/main/java/com/example/MyBenchmark.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.Benchmark; public class MyBenchmark { @Benchmark public long fibonacci() { return fibonacci(30); } static long fibonacci(int n) { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); } } ``` That's it. `@Benchmark` is the only annotation you need. JMH generates the measurement harness around it. The method **returns** its result, which prevents the JVM from eliminating the computation as dead code (more on this in [avoiding common pitfalls](#dead-code-elimination)). ### Building and Running Build the uber-JAR and run the benchmark: ```sh title=terminal icon="square-terminal" theme={null} cd my-benchmarks mvn clean verify java -jar target/benchmarks.jar ``` ```sh title=terminal icon="square-terminal" theme={null} cd my-benchmarks ./gradlew jmh ``` **This will take about 8 minutes.** JMH defaults are thorough: 5 forked JVMs, each running 5 warmup + 5 measurement iterations of 10 seconds. For a faster first run, add flags to reduce the iteration count: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar -f 1 -wi 3 -i 5 -w 1 -r 1 ``` ```sh title=terminal icon="square-terminal" theme={null} ./gradlew jmh -Pjmh.fork=1 -Pjmh.warmupIterations=3 -Pjmh.iterations=5 -Pjmh.warmup='1s' -Pjmh.timeOnIteration='1s' ``` These flags and annotations are explained in [Configuring Your Benchmark](#configuring-your-benchmark). You should see output like this: ```shellsession title=terminal icon="square-terminal" theme={null} # JMH version: 1.37 # VM version: JDK 17.0.18, OpenJDK 64-Bit Server VM, 17.0.18+8-Debian-1deb12u1 # Warmup: 5 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: com.example.MyBenchmark.fibonacci # Run progress: 0.00% complete, ETA 00:08:20 # Fork: 1 of 5 # Warmup Iteration 1: 320.348 ops/s # Warmup Iteration 2: 321.605 ops/s # Warmup Iteration 3: 323.393 ops/s # Warmup Iteration 4: 323.038 ops/s # Warmup Iteration 5: 321.964 ops/s Iteration 1: 320.996 ops/s Iteration 2: 320.143 ops/s Iteration 3: 323.586 ops/s Iteration 4: 322.946 ops/s Iteration 5: 321.108 ops/s # Run progress: 20.00% complete, ETA 00:06:40 # Fork: 2 of 5 ... Benchmark Mode Cnt Score Error Units MyBenchmark.fibonacci thrpt 25 320.479 ± 1.013 ops/s ``` Without any configuration, JMH automatically warmed up the JIT compiler across 5 separate JVM processes, collected 25 measurement iterations (5 per fork), and computed a tight 99.9% confidence interval. The default mode is **Throughput** (`thrpt`), measured in operations per second. **Understanding the results:** * **Mode**: The benchmark mode (`thrpt` = throughput, operations per second). * **Cnt**: Total measurement iterations across all forks (5 forks x 5 iterations \= 25). * **Score**: The measured value (higher is better for `thrpt`). * **Error**: The 99.9% confidence interval margin. The true value lies within `Score ± Error` with 99.9% confidence. * **Units**: `ops/s` = operations per second. ## Configuring Your Benchmark The previous benchmark used all JMH defaults. In practice, you want to embed settings into your benchmark class using annotations. This makes benchmarks self-describing and reproducible regardless of how they are invoked. Update `MyBenchmark.java`: ```java src/main/java/com/example/MyBenchmark.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.*; import java.util.concurrent.TimeUnit; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MILLISECONDS) @State(Scope.Thread) @Fork(1) @Warmup(iterations = 3, time = 1) @Measurement(iterations = 5, time = 1) public class MyBenchmark { private int n = 30; @Benchmark public long fibonacci() { return fibonacci(n); } static long fibonacci(int n) { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); } } ``` Rebuild and run. No flags needed, everything is in the annotations: ```sh title=terminal icon="square-terminal" theme={null} mvn clean verify java -jar target/benchmarks.jar ``` ```sh title=terminal icon="square-terminal" theme={null} ./gradlew jmh ``` ```shellsession title=terminal icon="square-terminal" theme={null} # Benchmark mode: Average time, time/op # Benchmark: com.example.MyBenchmark.fibonacci # Fork: 1 of 1 # Warmup Iteration 1: 3.117 ms/op # Warmup Iteration 2: 3.131 ms/op # Warmup Iteration 3: 3.088 ms/op Iteration 1: 3.091 ms/op Iteration 2: 3.097 ms/op Iteration 3: 3.100 ms/op Iteration 4: 3.096 ms/op Iteration 5: 3.097 ms/op Benchmark Mode Cnt Score Error Units MyBenchmark.fibonacci avgt 5 3.096 ± 0.012 ms/op ``` The output now shows `avgt` (average time) in `ms/op`. A single fork completed in seconds instead of minutes. Computing `fibonacci(30)` takes about 3.1 milliseconds. The following sections break down each annotation. ### Benchmark Mode `@BenchmarkMode` controls what JMH measures. It can be placed on a class (applies to all methods) or on individual methods. Measures operations per second. Use this to quantify system capacity and compare throughput across implementations. Measures average time per operation. The general-purpose choice for latency benchmarking when you care about typical performance. Samples individual operation times and reports percentiles (p50, p90, p99, p99.9). Use this to understand tail latency, not just the average. Particularly useful because it reports percentiles directly: ```shellsession title=terminal icon="square-terminal" theme={null} MyBenchmark.fibonacci sample 177816 41.340 ± 0.936 ns/op MyBenchmark.fibonacci:p0.50 sample 38.000 ns/op MyBenchmark.fibonacci:p0.90 sample 44.000 ns/op MyBenchmark.fibonacci:p0.99 sample 58.000 ns/op MyBenchmark.fibonacci:p0.999 sample 279.183 ns/op MyBenchmark.fibonacci:p0.9999 sample 3199.859 ns/op ``` This reveals that while the median latency is 38ns, the p99.99 is 3.2 microseconds, an 84x spike. Percentile data like this is invaluable for understanding real-world latency characteristics. Measures the time for a single invocation with no warmup. Use this to benchmark cold-start performance and one-shot initialization costs. You can pass an array to run multiple modes in one benchmark run, e.g., `@BenchmarkMode({Mode.Throughput, Mode.AverageTime})`. Use `Mode.All` to run every mode at once, which is useful for exploratory benchmarking. ### State and Scope `@State` marks a class as a holder for benchmark data. Without it, you cannot use instance fields in benchmark methods. The `Scope` parameter controls how state is shared: Creates one state instance per thread with no sharing between threads. The default choice for most benchmarks. Shares one state instance across all threads. Use this when measuring contention and thread-safety overhead. Shares one state instance per thread group. Use this for asymmetric benchmarks (e.g., producer/consumer patterns). The benchmark class itself can be the state (as in our example), or you can define separate state classes: ```java title="Separate state class" icon="java" theme={null} @State(Scope.Benchmark) public static class SharedState { ConcurrentHashMap map = new ConcurrentHashMap<>(); } @Benchmark public void concurrentPut(SharedState state) { state.map.put("key", "value"); } ``` ### Fork, Warmup, Measurement, and Output Unit These annotations control the execution strategy and output formatting: Controls how many separate JVM processes to run. Forks run **sequentially**, not in parallel. Each fork starts a fresh JVM, isolating profile-guided optimizations and JIT compilation state. Use `jvmArgs` to control heap size, GC settings, and other JVM flags. Use `jvmArgsPrepend` or `jvmArgsAppend` to add flags without replacing defaults. ```java theme={null} @Fork(value = 3, jvmArgs = {"-Xms2G", "-Xmx2G"}) @Fork(value = 1, jvmArgsPrepend = {"-XX:+UseG1GC"}) ``` Controls how many iterations run before measurement begins, giving the JIT compiler time to optimize your code to steady state. Parameters: `iterations`, `time`, `timeUnit`. ```java theme={null} @Warmup(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS) ``` Controls how many iterations are recorded and included in the results. Accepts the same parameters as `@Warmup`: `iterations`, `time`, `timeUnit`. ```java theme={null} @Measurement(iterations = 10, time = 5, timeUnit = TimeUnit.SECONDS) ``` Controls the time unit displayed in results. Accepts any `java.util.concurrent.TimeUnit` value (e.g., `TimeUnit.NANOSECONDS`, `TimeUnit.MILLISECONDS`). ```java theme={null} @OutputTimeUnit(TimeUnit.MICROSECONDS) ``` ```java title="Configuration examples" icon="java" theme={null} // Quick feedback during development @Fork(1) @Warmup(iterations = 3, time = 1, timeUnit = TimeUnit.SECONDS) @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) // Reliable measurements for CI @Fork(value = 3, jvmArgs = {"-Xms2G", "-Xmx2G"}) @Warmup(iterations = 10, time = 5, timeUnit = TimeUnit.SECONDS) @Measurement(iterations = 10, time = 5, timeUnit = TimeUnit.SECONDS) ``` ### Threads `@Threads` controls how many threads run the benchmark **concurrently**. The default is 1. Combined with [`Scope.Benchmark`](#param-scope-benchmark), this is how you measure contention: ```java title="Multi-threaded benchmark" icon="java" theme={null} @Threads(4) @State(Scope.Benchmark) public class ConcurrencyBenchmark { private ConcurrentHashMap map = new ConcurrentHashMap<>(); @Benchmark public Integer concurrentPut() { return map.put(Thread.currentThread().hashCode(), 42); } } ``` Use `@Threads(Threads.MAX)` to use all available processors. ## Benchmarking with Parameters The previous examples all used a single input value (30). But what if you want to see how performance changes with different input sizes? This is where `@Param` comes in. ### Single Parameter Add a new benchmark class to test multiple input sizes: ```java src/main/java/com/example/FibonacciParameterized.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.*; import java.util.concurrent.TimeUnit; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS) @State(Scope.Thread) @Fork(1) @Warmup(iterations = 3, time = 1) @Measurement(iterations = 5, time = 1) public class FibonacciParameterized { @Param({"5", "10", "15", "20", "30"}) private int n; static long fibonacci(int n) { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); } @Benchmark public long fibRecursive() { return fibonacci(n); } } ``` `@Param` tells JMH to run the benchmark once for each value. Rebuild and run: ```sh title=terminal icon="square-terminal" theme={null} mvn clean verify java -jar target/benchmarks.jar FibonacciParameterized ``` ```sh title=terminal icon="square-terminal" theme={null} ./gradlew jmh -Pjmh.includes='FibonacciParameterized' ``` ```shellsession title=terminal icon="square-terminal" theme={null} Benchmark (n) Mode Cnt Score Error Units FibonacciParameterized.fibRecursive 5 avgt 5 0.013 ± 0.001 us/op FibonacciParameterized.fibRecursive 10 avgt 5 0.202 ± 0.001 us/op FibonacciParameterized.fibRecursive 15 avgt 5 2.277 ± 0.030 us/op FibonacciParameterized.fibRecursive 20 avgt 5 25.213 ± 0.295 us/op FibonacciParameterized.fibRecursive 30 avgt 5 3122.539 ± 55.028 us/op ``` The results clearly show the exponential O(2^n) growth of recursive Fibonacci: going from n=5 (13 nanoseconds) to n=30 (3.1 milliseconds), a factor of 240,000x. You can override `@Param` values from the command line without recompiling: ```sh theme={null} java -jar target/benchmarks.jar -p n=25,35 ``` ### Multiple Parameters Each `@Param` annotation applies to a single field, but you can use multiple `@Param` fields to benchmark across several dimensions. JMH runs all combinations automatically: ```java title="Multiple @Param fields" icon="java" theme={null} @Param({"1000", "10000"}) private int size; @Param({"ArrayList", "LinkedList"}) private String listType; ``` This produces four benchmark runs: `1000/ArrayList`, `1000/LinkedList`, `10000/ArrayList`, `10000/LinkedList`. ### Comparing Algorithms Parameters are powerful for comparing different implementations side-by-side. Let's benchmark recursive vs. iterative Fibonacci: ```java src/main/java/com/example/AlgorithmComparison.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.*; import java.util.concurrent.TimeUnit; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS) @State(Scope.Thread) @Fork(1) @Warmup(iterations = 3, time = 1) @Measurement(iterations = 5, time = 1) public class AlgorithmComparison { @Param({"10", "20", "30"}) private int n; static long fibRecursive(int n) { if (n <= 1) return n; return fibRecursive(n - 1) + fibRecursive(n - 2); } static long fibIterative(int n) { if (n <= 1) return n; long a = 0, b = 1; for (int i = 2; i <= n; i++) { long temp = a + b; a = b; b = temp; } return b; } @Benchmark public long recursive() { return fibRecursive(n); } @Benchmark public long iterative() { return fibIterative(n); } } ``` ```shellsession title=terminal icon="square-terminal" theme={null} Benchmark (n) Mode Cnt Score Error Units AlgorithmComparison.iterative 10 avgt 5 0.003 ± 0.001 us/op AlgorithmComparison.iterative 20 avgt 5 0.004 ± 0.001 us/op AlgorithmComparison.iterative 30 avgt 5 0.006 ± 0.001 us/op AlgorithmComparison.recursive 10 avgt 5 0.203 ± 0.001 us/op AlgorithmComparison.recursive 20 avgt 5 25.596 ± 0.795 us/op AlgorithmComparison.recursive 30 avgt 5 3122.110 ± 33.573 us/op ``` The iterative version computes `fibonacci(30)` in 6 nanoseconds while the recursive version takes 3.1 milliseconds: over **500,000x faster**. This is the power of parameterized benchmarks: they make algorithmic trade-offs visible at a glance. ## Benchmarking Only What Matters Sometimes you have expensive setup that should not be included in your benchmark measurements. For example, generating test data or loading files. JMH provides `@Setup` and `@TearDown` annotations with different `Level` options to control when fixture methods run. ### Setup and Teardown Let's benchmark an outlier detection algorithm where the dataset generation is expensive but should not be measured: ```java src/main/java/com/example/OutlierDetection.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.*; import java.util.concurrent.TimeUnit; import java.util.ArrayList; import java.util.List; import java.util.Random; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS) @State(Scope.Thread) @Fork(1) @Warmup(iterations = 3, time = 1) @Measurement(iterations = 5, time = 1) public class OutlierDetection { @Param({"10000", "100000", "1000000"}) private int size; private double[] data; @Setup(Level.Trial) public void setUp() { // NOT MEASURED: expensive data generation runs once before all iterations Random random = new Random(42); data = new double[size]; for (int i = 0; i < size; i++) { if (random.nextDouble() < 0.95) { data[i] = 100.0 + random.nextGaussian() * 15.0; } else { data[i] = 200.0 + random.nextDouble() * 100.0; } } } public static List detectOutliers(double[] data, double threshold) { double sum = 0; for (double v : data) sum += v; double mean = sum / data.length; double variance = 0; for (double v : data) variance += (v - mean) * (v - mean); variance /= data.length; double stdDev = Math.sqrt(variance); List outliers = new ArrayList<>(); for (int i = 0; i < data.length; i++) { double zScore = stdDev > 0 ? Math.abs((data[i] - mean) / stdDev) : 0; if (zScore > threshold) { outliers.add(i); } } return outliers; } @Benchmark public List findOutliers() { // MEASURED: only the outlier detection algorithm return detectOutliers(data, 2.0); } } ``` The `@Setup(Level.Trial)` method runs once before all measurement iterations. Only the `findOutliers()` method is timed: ```shellsession title=terminal icon="square-terminal" theme={null} Benchmark (size) Mode Cnt Score Error Units OutlierDetection.findOutliers 10000 avgt 5 22.659 ± 0.322 us/op OutlierDetection.findOutliers 100000 avgt 5 282.683 ± 2.461 us/op OutlierDetection.findOutliers 1000000 avgt 5 3079.753 ± 49.886 us/op ``` ### Fixture Levels JMH offers three levels for `@Setup` and `@TearDown`: Runs once per benchmark fork. Use this for loading files and building large datasets that are reused across all iterations. Runs before and after each measurement iteration. Use this to reset mutable state between iterations. Runs before and after each individual method call. Use sparingly - this adds overhead on every invocation. `Level.Invocation` adds timing overhead on every call. Only use it when the benchmark method is slow enough (milliseconds or more) that the fixture cost is negligible in comparison. Here is an example using `Level.Iteration` to provide fresh unsorted data for each iteration of a sorting benchmark: ```java src/main/java/com/example/SortBenchmark.java icon="java" theme={null} package com.example; import org.openjdk.jmh.annotations.*; import java.util.concurrent.TimeUnit; import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.Random; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS) @State(Scope.Thread) @Fork(1) @Warmup(iterations = 3, time = 1) @Measurement(iterations = 5, time = 1) public class SortBenchmark { @Param({"1000", "10000", "100000"}) private int size; private List data; @Setup(Level.Iteration) public void setUp() { // Regenerate unsorted data before each iteration Random random = new Random(42); data = new ArrayList<>(size); for (int i = 0; i < size; i++) { data.add(random.nextInt(size)); } } @Benchmark public List sortList() { List copy = new ArrayList<>(data); Collections.sort(copy); return copy; } } ``` ## Running Benchmarks from the Command Line The `benchmarks.jar` supports a rich set of command-line options. Here are the most useful ones: ### Filtering Benchmarks Run only benchmarks matching a regex: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar "FibonacciParameterized" ``` Exclude benchmarks matching a pattern: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar -e ".*Slow.*" ``` ### Overriding Parameters Override [`@Param`](#single-parameter), [`@Fork`](#param-fork), [`@Warmup`](#param-warmup), and [`@Measurement`](#param-measurement) from the command line: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar -p n=25,35 -f 3 -wi 5 -i 10 ``` Number of forks. Number of threads. Warmup iterations. Measurement iterations. Warmup iteration time (e.g., `2s`). Measurement iteration time. Override `@Param` values. Override benchmark mode (`thrpt`, `avgt`, `sample`, `ss`). Override time unit (`ns`, `us`, `ms`, `s`). ### Exporting Results JMH can export results in various formats for further analysis or visualization: Result format. One of `text`, `csv`, `scsv`, `json`, `latex`. Result file path. Where to write the output (e.g., `results.json`). For example, to export JSON results: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar -rf json -rff results.json ``` ### Using Profilers JMH ships with built-in profilers. List them with: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar -lprof ``` The most useful profilers: Samples hot methods and thread states to show where time is being spent. Reports allocation rate, GC pressure, and bytes allocated per operation. Reports JIT compilation activity during the measurement window. Reports per-operation hardware counters: cache misses, branch mispredictions, and CPI. Linux only. Generates CPU flamegraphs using [async-profiler](https://github.com/async-profiler/async-profiler). For example, to measure allocation pressure: ```sh title=terminal icon="square-terminal" theme={null} java -jar target/benchmarks.jar OutlierDetection -prof gc ``` This adds GC metrics to the output, showing bytes allocated per operation (`gc.alloc.rate.norm`) and GC event counts, essential for understanding allocation-heavy code. ## Avoiding Common Pitfalls The JVM is a sophisticated optimizing runtime. Without care, it can silently eliminate or transform the code you are trying to measure, producing misleading results. JMH is designed to help, but you still need to follow certain patterns. ### Dead Code Elimination If a computation's result is never used, the JIT compiler may eliminate it entirely: ```java title="Dead code elimination" icon="java" theme={null} // BAD: result is discarded, JVM may eliminate the entire computation @Benchmark public void measureWrong() { Math.log(x); } // GOOD: returning the result prevents dead code elimination @Benchmark public double measureRight() { return Math.log(x); } ``` JMH automatically consumes the return value of `@Benchmark` methods through an internal `Blackhole`, preventing elimination. Always return your computed result. ### Blackholes for Multiple Results When you produce multiple results, you can only return one. Use `Blackhole.consume()` for the rest: ```java title="Blackhole usage" icon="java" theme={null} @Benchmark public void computeMultiple(Blackhole bh) { bh.consume(Math.log(x)); bh.consume(Math.sqrt(x)); } ``` Import `Blackhole` from `org.openjdk.jmh.infra.Blackhole`. JMH injects it automatically as a method parameter. ### Constant Folding If the JVM can determine a computation's inputs at compile time, it folds the entire computation into a constant: ```java title="Constant folding" icon="java" theme={null} // BAD: the JVM knows wrongX is always Math.PI, result is precomputed private final double wrongX = Math.PI; @Benchmark public double measureWrong() { return Math.log(wrongX); } // GOOD: non-final field prevents constant folding private double x = Math.PI; @Benchmark public double measureRight() { return Math.log(x); } ``` Your IDE may suggest making `x` final. Do not. Non-final `@State` fields are essential for preventing constant folding in benchmarks. ### Do Not Loop Manually Never write manual loops inside benchmark methods. The JVM aggressively optimizes loops. It unrolls, pipelines, and hoists invariant computations out of them, producing unrealistically low per-operation numbers: ```java title="Manual loops" icon="java" theme={null} // BAD: JVM optimizes the loop, results are misleading @Benchmark public int measureWrong() { int sum = 0; for (int i = 0; i < 1000; i++) { sum += compute(i); } return sum; } // GOOD: let JMH control the iteration @Benchmark public int measureRight() { return compute(x); } ``` JMH handles iteration internally with proper safeguards. Trust the framework. ## Best Practices ### Use Multiple Forks The JVM is non-deterministic. Profile-guided optimizations, garbage collection, and thread scheduling vary between runs. A single fork can give misleading results. Use multiple forks (see [`@Fork`](#param-fork)) to capture this variance: ```java title="Fork configuration" icon="java" theme={null} // For development, 1 fork is fine for fast feedback @Fork(1) // For reliable measurements, use 3-5 forks @Fork(5) ``` Each fork starts a fresh JVM, isolating profile-guided optimizations and giving JMH enough data points to compute meaningful confidence intervals. ### Keep Benchmarks Deterministic Use fixed seeds in your [`@Setup`](#param-level-trial) methods for random number generators: ```java title="Deterministic setup" icon="java" theme={null} // BAD: different data every run, results are not reproducible @Setup(Level.Trial) public void setUp() { Random rng = new Random(); // non-deterministic seed // ... } // GOOD: fixed seed, results are reproducible @Setup(Level.Trial) public void setUp() { Random rng = new Random(42); // deterministic seed // ... } ``` ### Verify Correctness Alongside Performance Include assertions in your setup or dedicated test methods to ensure you are benchmarking correct code, not broken code that happens to be fast: ```java title="Correctness check" icon="java" theme={null} @Setup(Level.Trial) public void setUp() { // Verify the algorithm is correct before measuring it if (fibonacci(10) != 55) { throw new IllegalStateException("fibonacci(10) should be 55"); } } ``` ### Use Realistic Data Sorted or regular data can exploit hardware optimizations like branch prediction and cache prefetching, giving misleadingly good results. Use representative data that matches your production workload. ### Benchmark Your Own Code In real projects, organize your benchmarks alongside your source code: The benchmark submodule depends on your library and uses the JMH archetype setup. This keeps benchmark infrastructure separate from production code. ## Running Benchmarks Continuously with CodSpeed So far, you've been running benchmarks locally. But local benchmarking has limitations: * **Inconsistent hardware**: Different developers get different results * **Manual process**: Easy to forget to run benchmarks before merging * **No historical tracking**: Hard to spot gradual performance degradation * **No PR context**: Can't see performance impact during code review This is where **CodSpeed** comes in. It runs your benchmarks automatically in CI and provides: * Automated performance regression detection in PRs * Consistent metrics with reliable measurements across all runs * Historical tracking to see performance over time with detailed charts * Flamegraph profiles to see exactly what changed in your code's execution ### How to Set Up CodSpeed Here's how to integrate CodSpeed with your JMH benchmarks: CodSpeed integrates with JMH through a custom fork. Before configuring CI, follow the [Java integration reference](/docs/benchmarks/java) to add the fork as a Maven or Gradle dependency. Create a workflow file to run benchmarks on every push and pull request. Once the workflow runs, your pull requests will receive a performance report comment: Pull Request Result

After your benchmarks run in CI, head over to your CodSpeed dashboard to see detailed performance reports, historical trends, and flamegraph profiles for deeper analysis. Profiling Report on CodSpeed

Profiling works out of the box, no extra configuration needed! [Learn more about flamegraphs and how to use them to optimize your code](/docs/features/profiling). ## Next Steps Check out these resources to continue your Java benchmarking journey: Sign up and start tracking your Java performance in CI Set up the CodSpeed JMH fork in your Maven or Gradle project Learn how to use flamegraphs to optimize your code Dive into the JMH source and annotation reference # How to Benchmark Python with pytest? Source: https://codspeed.io/docs/guides/how-to-benchmark-python-with-pytest Learn how to measure the performance of your Python code by writing and running benchmarks locally and continuously in CI to catch regressions. ## Why pytest-codspeed? This guide uses [`pytest-codspeed`](https://github.com/CodSpeedHQ/pytest-codspeed) because it integrates seamlessly with [`pytest`](https://docs.pytest.org/), the most popular Python testing framework. Your benchmarks live right alongside your tests using the same familiar syntax, no separate infrastructure to maintain. Plus, all of `pytest`'s ecosystem (parametrization, fixtures, plugins) works seamlessly with your benchmarks. You can even turn existing tests into benchmarks by adding a single decorator. If you're wondering whether to use command-line tools like `time` or `hyperfine` versus integrated frameworks like `pytest-codspeed`, check out our [Choosing the Right Python Benchmarking Strategy guide](/docs/guides/choosing-the-correct-python-benchmarking-strategy) for a detailed comparison. ## Your First Benchmark Let's start by creating a simple benchmark for a recursive Fibonacci function. ### Installation First, add `pytest-codspeed` to your project's dependencies using [`uv`](https://docs.astral.sh/uv/): ```bash icon="square-terminal" theme={null} uv add --dev pytest-codspeed ``` **Don't have `uv`?** You can use `pip install pytest-codspeed` instead. `uv` is a modern, fast Python package manager that we recommend for new projects, but any package manager works fine. ### Writing the Benchmark Create a new file `tests/test_benchmarks.py`: ```python tests/test_benchmarks.py icon="python" theme={null} import pytest # Define the function we want to benchmark def fibonacci(n: int) -> int: if n <= 1: return n else: return fibonacci(n - 2) + fibonacci(n - 1) # Register a simple benchmark using the pytest marker @pytest.mark.benchmark def test_fib_bench(): result = fibonacci(30) assert result == 832040 ``` A few things to note: `@pytest.mark.benchmark` is a standard [`pytest` marker](https://docs.pytest.org/en/stable/how-to/mark.html) that marks this test as a benchmark. The entire test function is measured, including both the computation and the assertion. It's a regular `pytest` test, so you can run it with `pytest` as usual. The test validates correctness (via assertions) and tracks performance at the same time. ### Running the Benchmark Now run your benchmark: ```bash icon="square-terminal" theme={null} uv run pytest tests/ --codspeed ``` **What does `--codspeed` do?** This flag activates CodSpeed's benchmarking engine to collect performance measurements. Without it, pytest runs your tests normally without gathering performance data. If you're not using `uv`, run `pytest tests/ --codspeed` instead. You should see output like this: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 1 item tests/test_benchmarks.py . [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩ │ test_fib_bench │ 73.1ms │ 2.1% │ 2.96s │ 40 │ └────────────────┴─────────────┴─────────────┴──────────┴───────┘ ================================== 1 benchmarked ================================== ================================ 1 passed in 4.09s ================================ ``` The output shows that `test_fib_bench` takes about 73 milliseconds to compute `fibonacci(30)`. It ran 40 times in 2.96 seconds to get a reliable measurement. **Understanding the results:** * **Time (best)**: The fastest single iteration - this is your function's performance (lower is better). * **Rel. StdDev**: Relative standard deviation - measures consistency between runs (lower means more reliable results). * **Run time**: Total time spent running the benchmark. * **Iters**: How many times your code ran - automatically adjusted based on speed (fast code runs more times for accuracy). ## Benchmarking with Arguments So far, we've only tested our function with a single input value (30). But what if we want to see how performance changes with different input sizes? This is where `pytest`'s [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/how-to/parametrize.html) comes in, and it works seamlessly with benchmarks. Let's update our benchmark to test multiple input sizes: ```python tests/test_benchmarks.py icon="python" theme={null} @pytest.mark.benchmark @pytest.mark.parametrize("n", [5, 10, 15, 20, 30]) def test_fib_parametrized(n): result = fibonacci(n) assert result > 0 ``` When you run this benchmark, pytest will create separate test instances for each parameter value, allowing you to compare performance across different inputs: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 5 items tests/test_benchmarks.py ..... [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩ │ test_fib_parametrized[5] │ 0ns │ 1.7% │ 2.92s │ 1,026,802 │ │ test_fib_parametrized[10] │ 1ns │ 1.7% │ 2.89s │ 395,754 │ │ test_fib_parametrized[15] │ 76ns │ 0.8% │ 2.94s │ 52,256 │ │ test_fib_parametrized[20] │ 8.49µs │ 3.6% │ 3.00s │ 4,970 │ │ test_fib_parametrized[30] │ 72.9ms │ 0.7% │ 2.94s │ 40 │ └───────────────────────────┴─────────────┴─────────────┴──────────┴───────────┘ ================================== 5 benchmarked ================================== =============================== 5 passed in 19.88s ================================ ``` Notice how parametrization creates five separate benchmarks, one for each input value. The results reveal the exponential time complexity of our recursive Fibonacci implementation: `fibonacci(5)` takes virtually no time (0ns) and runs over 1 million iterations, while `fibonacci(30)` takes 72.9ms and runs only 40 times. This dramatic difference (from nanoseconds to milliseconds) demonstrates how quickly recursive Fibonacci becomes expensive as the input grows. ### Multiple Parameters You can also benchmark across multiple dimensions: ```python tests/test_benchmarks.py icon="python" theme={null} def fibonacci_iterative(n: int) -> int: if n <= 1: return 1 a, b = 1, 1 for _ in range(n - 1): a, b = b, a + b return b @pytest.mark.benchmark @pytest.mark.parametrize("algorithm, n", [ ("recursive", 10), ("recursive", 20), ("iterative", 100), ("iterative", 200), ]) def test_fib_algorithms(algorithm, n): if algorithm == "recursive": result = fibonacci(n) else: result = fibonacci_iterative(n) assert result > 0 ``` Then run it: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 4 items tests/test_benchmarks.py .... [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓ ┃ ┃ Time ┃ Rel. ┃ ┃ ┃ ┃ Benchmark ┃ (best) ┃ StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩ │ test_fib_algorithms[recursive-10] │ 1ns │ 1.0% │ 2.93s │ 614,789 │ │ test_fib_algorithms[recursive-20] │ 8.49µs │ 26.9% │ 3.01s │ 4,970 │ │ test_fib_algorithms[iterative-100] │ 0ns │ 42.1% │ 3.04s │ 1,474,1… │ │ test_fib_algorithms[iterative-200] │ 0ns │ 1.3% │ 2.29s │ 587,099 │ └────────────────────────────────────┴──────────┴───────────┴──────────┴──────────┘ ================================== 4 benchmarked ================================== =============================== 4 passed in 15.40s ================================ ``` This benchmark creates four separate test cases, one for each combination of algorithm and input size. The output clearly shows the dramatic performance difference between the two implementations: the iterative version handles much larger inputs (100, 200) in virtually no time, while the recursive version takes 8.49µs for `n=20`. Notice how `fibonacci_iterative(200)` runs over 500,000 iterations in the same time budget that `fibonacci(20)` only manages about 5,000. Parametrization makes algorithmic trade-offs visible at a glance, helping you choose the most efficient implementation for your use case. ### Naming Parametrized Cases By default, `pytest` generates benchmark names from the parameter values. That works well for primitives like numbers or short strings, e.g., `test_fib_parametrized[5]`. With richer parameters such as dictionaries, lists, or `callable` objects, the auto-generated names degrade into opaque labels like `test_my_bench[param0-param1]`. They are hard to read, and if the underlying values change between runs, CodSpeed treats each case as a new benchmark and loses the historical comparison. Use the `ids` argument to attach a stable, descriptive label to each case: ```python tests/test_benchmarks.py icon="python" theme={null} @pytest.mark.benchmark @pytest.mark.parametrize( "n", [5, 10, 15, 20, 30], ids=["tiny", "small", "medium", "large", "huge"], ) def test_fib_named(n): result = fibonacci(n) assert result > 0 ``` The benchmark output now reads `test_fib_named[tiny]` through `test_fib_named[huge]`, which is easier to scan and stays stable even if you tweak the parameter values later. For finer-grained control, wrap individual cases in `pytest.param` to attach an id one at a time: ```python tests/test_benchmarks.py icon="python" theme={null} @pytest.mark.benchmark @pytest.mark.parametrize("payload", [ pytest.param({"users": 100}, id="small-payload"), pytest.param({"users": 10_000}, id="large-payload"), ]) def test_serialize(payload): serialize(payload) ``` This form is especially useful when parameters are dictionaries, `dataclass` instances, or other non-trivial objects that `pytest` cannot turn into readable ids on its own. `pytest.param` also works with multiple parameters. Pass one positional value per name in the `parametrize` declaration, then attach a single `id` that describes the whole case: ```python tests/test_benchmarks.py icon="python" theme={null} @pytest.mark.benchmark @pytest.mark.parametrize( "algorithm, n", [ pytest.param("recursive", 10, id="recursive-small"), pytest.param("recursive", 20, id="recursive-large"), pytest.param("iterative", 100, id="iterative-small"), pytest.param("iterative", 200, id="iterative-large"), ], ) def test_fib_algorithms(algorithm, n): if algorithm == "recursive": result = fibonacci(n) else: result = fibonacci_iterative(n) assert result > 0 ``` The output now reads `test_fib_algorithms[recursive-small]` instead of the default `test_fib_algorithms[recursive-10]`, so the benchmark name stays stable even if you later change `10` to `15` to keep run times in a useful range. Pick ids that describe the scenario, not the raw value. `["cold-cache", "warm-cache"]` tells you more about what is being measured than `[0, 1]`, and it stays meaningful when the underlying values change. ## Benchmarking Only What Matters Sometimes, you have expensive setup that shouldn't be included in your benchmark measurements. For example, generating large datasets, creating complex data structures, or preparing test data. This is where the `benchmark` [`fixture`](https://docs.pytest.org/en/stable/how-to/fixtures.html) comes in. The `benchmark` fixture gives you precise control over what gets measured. Let's benchmark a data analysis function that identifies outliers in numerical data. The expensive part is generating the test dataset, but we only want to measure the outlier detection algorithm: ```python tests/test_outlier_detection.py icon="python" theme={null} import pytest import random def generate_dataset(size: int) -> list[float]: """Generate a large dataset with some outliers (expensive operation).""" random.seed(42) # Fixed seed for reproducibility data = [] for _ in range(size): # 95% normal values from a normal distribution if random.random() < 0.95: data.append(random.gauss(100.0, 15.0)) else: # 5% outliers data.append(random.uniform(200.0, 300.0)) return data def detect_outliers(data: list[float], threshold: float = 2.0) -> list[int]: """Detect outliers using z-score method (what we want to benchmark).""" # Calculate mean mean = sum(data) / len(data) # Calculate standard deviation variance = sum((x - mean) ** 2 for x in data) / len(data) std_dev = variance ** 0.5 # Find outliers outliers = [] for i, value in enumerate(data): z_score = abs((value - mean) / std_dev) if std_dev > 0 else 0 if z_score > threshold: outliers.append(i) return outliers # Benchmark for dataset generation @pytest.mark.benchmark @pytest.mark.parametrize("size", [10_000, 100_000, 1_000_000]) def test_generate_dataset(size): generate_dataset(size) # Benchmark for outlier detection only @pytest.mark.parametrize("size", [10_000, 100_000, 1_000_000]) def test_outlier_detection(benchmark, size): # NOT MEASURED: Expensive setup - generate large dataset dataset = generate_dataset(size) # MEASURED: Only the outlier detection algorithm result = benchmark(detect_outliers, dataset) # NOT MEASURED: Assertions assert len(result) > 0 # We should find some outliers assert all(isinstance(idx, int) for idx in result) ``` The setup code (generating the dataset) runs once, and **only** the `detect_outliers()` call inside `benchmark()` is measured. This gives you accurate performance data without the noise of test setup. Run this benchmark by filtering the `pytest` command to this file: ```bash icon="square-terminal" theme={null} uv run pytest tests/test_outlier_detection.py --codspeed ``` You should see output like this: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 6 items tests/test_outlier_detection.py ...... [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩ │ test_generate_dataset[10000] │ 124.29µs │ 4.4% │ 2.92s │ 1,278 │ │ test_generate_dataset[100000] │ 11.0ms │ 6.2% │ 2.96s │ 130 │ │ test_generate_dataset[1000000] │ 225.5ms │ 29.6% │ 3.04s │ 13 │ │ test_outlier_detection[10000] │ 46.01µs │ 18.4% │ 2.89s │ 2,059 │ │ test_outlier_detection[100000] │ 3.3ms │ 6.1% │ 3.04s │ 220 │ │ test_outlier_detection[1000000] │ 132.4ms │ 12.6% │ 3.04s │ 22 │ └─────────────────────────────────┴─────────────┴─────────────┴──────────┴───────┘ ================================== 6 benchmarked ================================== =============================== 6 passed in 24.84s ================================ ``` The results reveal a crucial insight about what we're actually measuring. Notice the dramatic difference between the two benchmark groups: **Dataset generation** (`test_generate_dataset`): * 10k elements: 124.29µs * 100k elements: 11.0ms (88x slower) * 1M elements: 225.5ms (1,814x slower than 10k) **Outlier detection** (`test_outlier_detection`): * 10k elements: 46.01µs * 100k elements: 3.3ms (72x slower) * 1M elements: 132.4ms (2,878x slower than 10k) This comparison shows that for the 1M element dataset, **dataset generation takes 225.5ms while outlier detection takes 132.4ms**, the setup is actually slower than the algorithm we want to measure. Without using the `benchmark` fixture to exclude the setup, our measurements would include both operations, making it impossible to understand the true performance of the outlier detection algorithm. The `benchmark` fixture ensures we **measure only what matters: the algorithm itself**, not the test infrastructure around it. ## Additional Techniques ### Marking an Entire Module If you have a dedicated benchmarks file, you can mark all tests as benchmarks at once using `pytest`'s module-level marking: ```python tests/benchmarks/test_math_operations.py icon="python" theme={null} import pytest # Mark all tests in this module as benchmarks pytestmark = pytest.mark.benchmark def test_sum_squares(): # MEASURED: Everything in this test result = sum(i**2 for i in range(1000)) assert result > 0 def test_sum_cubes(): # MEASURED: Everything in this test result = sum(i**3 for i in range(1000)) assert result > 0 ``` Now all tests in this file are automatically benchmarked without individual decorators. This is useful for benchmark-specific test files. ### Fine-Grained Control with Pedantic For maximum control over your benchmarks, use [`benchmark.pedantic()`](https://pytest-benchmark.readthedocs.io/en/latest/pedantic.html). This allows you to specify custom setup and teardown functions, control the number of rounds and iterations, configure warmup behavior, and more: ```python tests/test_advanced.py icon="python" theme={null} import json import pytest def parse_json_data(json_string: str) -> dict: """Parse JSON string into a dictionary.""" return json.loads(json_string) @pytest.mark.parametrize("size", [10_000, 30_000]) def test_json_parsing(benchmark, size): # NOT MEASURED: Setup to create test data items = [{"id": i, "name": f"item_{i}", "value": i * 10} for i in range(size)] json_string = json.dumps(items) # MEASURED: Only the parse_json_data() function result = benchmark.pedantic( parse_json_data, # Function to benchmark args=(json_string,), # Arguments to the function rounds=100, # Number of benchmark rounds iterations=10, # Iterations per round warmup_rounds=2 # Warmup rounds before measuring ) # NOT MEASURED: The assertion assert len(result) == size ``` Here is the output when you run this benchmark: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 2 items tests/test_advanced.py .. [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩ │ test_json_parsing[10000] │ 294.85µs │ 0.9% │ 2.99s │ 1,000 │ │ test_json_parsing[30000] │ 973.01µs │ 0.7% │ 9.88s │ 1,000 │ └──────────────────────────┴─────────────┴─────────────┴──────────┴───────┘ ================================== 2 benchmarked ================================== =============================== 2 passed in 13.20s ================================ ``` We can see that as expected each benchmark ran 100 rounds of 10 iterations each, totalling 1,000 iterations. Using `benchmark.pedantic()` is especially useful for bigger benchmarks where you need precise control over rounds, iterations, and warmup behavior. ### Benchmarking Async Functions To benchmark asynchronous functions, we can use the `benchmark` fixture along with `asyncio.run()` on a synchronous sub-function that calls our async code. Here's an example: ```python tests/test_async.py icon="python" theme={null} import asyncio import pytest async def simple_async_task() -> int: """A simple async task that simulates work""" await asyncio.sleep(0.1) # simulates async work for 100 ms return 42 @pytest.mark.benchmark def test_simple_async_task(benchmark): """Benchmark a simple async task""" result = benchmark(lambda: asyncio.run(simple_async_task())) assert result == 42 ``` Here is the output when you run this benchmark: ```shellsession title=terminal icon="square-terminal" theme={null} =============================== test session starts =============================== platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0 codspeed: 4.2.0 (enabled, mode: walltime, callgraph: not supported, timer_resolution: 41.7ns) CodSpeed had to disable the following plugins: pytest-benchmark benchmark: 5.2.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/user/projects/CodSpeedHQ/docs-guides/python configfile: pyproject.toml plugins: benchmark-5.2.1, codspeed-4.2.0 collected 1 item tests/test_async.py . [100%] Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩ │ test_simple_async_task │ 100.8ms │ 1.0% │ 2.95s │ 29 │ └────────────────────────┴─────────────┴─────────────┴──────────┴───────┘ ================================== 1 benchmarked ================================== ================================ 1 passed in 4.12s ================================ ``` Since asynchronous functions most likely involve I/O operations, their execution time can vary significantly based on external factors like network latency or disk speed. When benchmarking async code, consider running more iterations or rounds to obtain reliable measurements. If you are using CodSpeed in your CI to run your benchmarks, be sure to use the [Walltime instrument](/docs/instruments/walltime) to get accurate timing for async operations. ## Best Practices ### Use Assertions to Verify Correctness Since benchmarks are regular `pytest` tests, they should include assertions to verify correctness: ```python icon="python" theme={null} # ❌ BAD: No verification @pytest.mark.benchmark def test_computation(): result = expensive_computation() # Oops, forgot to check if result is correct! # ✅ GOOD: Verify the result without measuring the assertion def test_computation(benchmark): result = benchmark(expensive_computation) assert result == expected_value ``` This ensures you're benchmarking correct code, not broken code that happens to be fast. Or, as we briefly said in the introduction, you can turn existing tests into benchmarks by adding the `@pytest.mark.benchmark` decorator. ```python icon="python" theme={null} # Existing correctness test def test_sorting_algorithm(): data = [5, 2, 9, 1] result = sorting_algorithm(data) assert result == [1, 2, 5, 9] # Turn it into a benchmark using the benchmark fixture def test_sorting_algorithm(benchmark): data = [5, 2, 9, 1] result = benchmark(sorting_algorithm, data) assert result == [1, 2, 5, 9] ``` ### Keep Benchmarks Deterministic Your benchmarks should produce consistent results across runs: ```python icon="python" theme={null} # ❌ BAD: Non-deterministic due to random data def test_sort_random(benchmark): import random data = [random.randint(1, 1000) for _ in range(100)] benchmark(sorted, data) # ✅ GOOD: Use a fixed seed or deterministic data def test_sort_deterministic(benchmark): import random random.seed(42) # Fixed seed for reproducibility data = [random.randint(1, 1000) for _ in range(100)] benchmark(sorted, data) # ✅ EVEN BETTER: Use deterministic data def test_sort_worst_case(benchmark): data = list(range(100, 0, -1)) # Always the same benchmark(sorted, data) ``` ### Benchmarking Your Own Package Following Python best practices, your source code should live in a `src/` directory. Here's a typical project structure: ```shellsession title=terminal icon="square-terminal" theme={null} my_project/ ├── pyproject.toml ├── src/ │ └── mylib/ │ ├── __init__.py │ └── algorithms.py └── tests/ ├── test_algorithms.py # Regular unit tests └── benchmarks/ # Performance benchmarks └── test_algorithm_performance.py ``` Your source code in `src/mylib/algorithms.py`: ```python src/mylib/algorithms.py icon="python" theme={null} def quick_sort(arr: list[int]) -> list[int]: if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) ``` Then benchmark it in your tests: ```python tests/benchmarks/test_algorithm_performance.py icon="python" theme={null} from mylib.algorithms import quick_sort import pytest @pytest.mark.parametrize("size", [10, 100, 1000]) def test_quick_sort_performance(benchmark, size): # NOT MEASURED: Create test data data = list(range(size, 0, -1)) # MEASURED: The sorting algorithm result = benchmark(quick_sort, data) # NOT MEASURED: Verify correctness assert result == list(range(1, size + 1)) ``` Make sure your package is installed in development mode: ```bash icon="square-terminal" theme={null} uv pip install -e . ``` ## Running Benchmarks Continuously with CodSpeed So far, you've been running benchmarks locally. But local benchmarking has limitations: * **Inconsistent hardware**: Different developers get different results * **Manual process**: Easy to forget to run benchmarks before merging * **No historical tracking**: Hard to spot gradual performance degradation * **No PR context**: Can't see performance impact during code review This is where **CodSpeed** comes in. It runs your benchmarks automatically in CI and provides: * Automated performance regression detection in PRs * Consistent metrics with reliable measurements across all runs * Historical tracking to see performance over time with detailed charts * Flamegraph profiles to see exactly what changed in your code's execution For the full CodSpeed integration reference, see [Writing Benchmarks in Python](/docs/benchmarks/python). ### How to set up CodSpeed with pytest-codspeed Here's how to integrate CodSpeed with your `pytest-codspeed` benchmarks: Create a workflow file to run benchmarks on every push and pull request. **Important**: Use `actions/setup-python` to set up Python, not `uv install`. This is required for CodSpeed's CPU simulation to work correctly. Once the workflow runs, your pull requests will receive a performance report comment: Pull Request Result

After your benchmarks run in CI, head over to your CodSpeed dashboard to see detailed performance reports, historical trends, and flamegraph profiles for deeper analysis. Profiling Report on CodSpeed

Adding a new repository from the settings page

3. Select the organization or the user and add the repositories you want to use with CodSpeed: Github App organization selection

4. After connecting your repository, you can proceed to the [setup](#setup) to start tracking performance. ## Setup In the [CodSpeed settings](https://app.codspeed.io/settings), configure a repository by clicking on the "Setup" button: Repository Setup button

### Automated setup The [CodSpeed Wizard](/docs/ai/wizard) analyzes your repository, configures benchmarks, generates a CI workflow, and opens a pull request — all automatically. No manual configuration required. 1. From the repository setup screen, click **Start AI Setup** to let the Wizard handle the configuration: The setup modal showing AI Setup and Manual Setup options, with the Start AI Setup button highlighted

The setup modal showing AI Setup and Manual Setup options, with the Start AI Setup button highlighted

2. The Wizard analyzes your repository and configures it: The setup modal showing AI Setup and Manual Setup options, with the Start AI Setup button highlighted

3. The Wizard opens a setup **pull request**: Pull Request Result on Installation

4. Review and merge the pull request to complete setup. #### How it works The Wizard handles the full setup process automatically: 1. **Detects existing benchmarks**: finds and configures your benchmarks in the language and framework you're already using. 2. **Creates benchmarks if needed**: generates appropriate benchmark files when none exist in the repository. 3. **Generates CI workflows**: creates optimized GitHub Actions or GitLab CI configurations tailored to your setup. 4. **Opens a pull request**: submits all changes for your review before anything is merged. ### Manual setup If you prefer to configure CodSpeed yourself, follow the steps below. This example uses a Python repository with `pytest`. #### Create performance tests 1. Install the CodSpeed plugin for `pytest`: ```sh theme={null} pip install pytest-codspeed ``` 2. Write a performance test using the `@pytest.mark.benchmark` marker: ```python tests/test_sum_squares.py highlight={12} icon="python" theme={null} import pytest def sum_squares(arr): """Sum the squares of the numbers in an array.""" total = 0 for x in arr: total += x * x return total # Your tests can also be benchmarks @pytest.mark.benchmark def test_sum_squares(): assert sum_squares(range(1000)) == 332833500 ``` 3. Run your performance tests locally: ```shellsession title=terminal icon="square-terminal" theme={null} $ pytest tests/ --codspeed ============================= test session starts ==================== platform darwin -- Python 3.13.0, pytest-7.4.4, pluggy-1.5.0 codspeed: 3.0.0 (enabled, mode: walltime, timer_resolution: 41.7ns) rootdir: /home/user/codspeed-test, configfile: pytest.ini plugins: codspeed-3.0.0 collected 1 items tests/test_sum_squares.py . [ 100%] Benchmark Results ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┣━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━╋━━━━━━━━━━━━━╋━━━━━━━━━━╋━━━━━━━━┫ ┃test_sum_squares┃ 1,873ns ┃ 4.8% ┃ 3.00s ┃ 66,930 ┃ ┗━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━┻━━━━━━━━━━━━━┻━━━━━━━━━━┻━━━━━━━━┛ =============================== 1 benchmarked ======================== =============================== 1 passed in 4.12s ==================== ``` Your first performance test is ready. Next, track it in CI with CodSpeed. #### Run the tests in your CI 1. Create a new GitHub Actions workflow file to run your benchmarks: 2. Create a Pull Request installing the workflow to the repository and wait for the report in the comments: Pull Request Result on Installation

3. Merge it and congrats, CodSpeed is installed! #### Introduce a performance regression 1. Let's change the implementation of the `sum_squares` with a more concise and elegant one: ```python tests/test_sum_squares.py icon="python" theme={null} def sum_squares(arr): """Sum the squares of the numbers in an array.""" total = 0 # [!code --:4] for x in arr: total += x * x return total return sum(map(lambda x: x**2, arr)) # [!code ++] ``` 2. Open a Pull Request and wait for the CodSpeed report: Pull Request Regression Result

Before merging, the report shows this new implementation is slower. Pull Request Checks failing because of the performance regression

Merging it would have introduced a performance regression, but CodSpeed caught it before it shipped. This behavior can be enforced by configuring [performance checks](/docs/features/performance-checks/). ## Next Steps Learn how CodSpeed works Learn more about how to create performance tests with pytest-codspeed Understand the performance metrics generated by CodSpeed Make sure you or team members never merge unexpected performance regressions. # Allocation Exclusion Source: https://codspeed.io/docs/instruments/cpu/allocation-exclusion Exclude memory-allocation time from CPU simulation results to remove allocator variance. Allocators are a common source of benchmark variance. Their cost depends on the operating system, the allocator implementation, and its version. If you are optimizing your own code and the allocator itself is not what you are measuring, that cost is noise in your measurements. ## How it works CodSpeed tags every flamegraph frame that belongs to an allocator, covering the standard allocation functions, common allocators such as `jemalloc` and language-runtime allocators. With allocation exclusion enabled, CodSpeed sums the time spent in those frames, including everything they call, and subtracts it from the reported benchmark value. The flamegraph still shows the allocator frames, marked with an `Allocator` tag in the tooltip. Only the reported benchmark value changes. ## When to use it * Exclude allocations when allocator noise dominates a micro-benchmark and you are optimizing your own code. * Keep allocations included when the allocator itself is what you are measuring or optimizing. Allocation exclusion complements the allocator-variance strategies in [Reducing allocator variance](/docs/instruments/cpu/reducing-variance#reducing-allocator-variance). Tuning or swapping the allocator reduces variance, while exclusion leaves the allocator cost out of the reported value entirely. ## Enable it Pass the flag to the CodSpeed CLI: ```sh theme={null} codspeed run --exclude-allocations -- ``` In GitHub Actions, enable allocation exclusion with the dedicated action input: ```yaml .github/workflows/codspeed.yml highlight={17} theme={null} name: CodSpeed Benchmarks on: push: branches: - main pull_request: jobs: benchmarks: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: CodSpeedHQ/action@v5 with: mode: simulation exclude-allocations: true run: ``` ## Next Steps Techniques to make your benchmarks more stable across runs. Understand why benchmarks can regress without code changes. # CPU Simulation Instrument Source: https://codspeed.io/docs/instruments/cpu/index Learn how to use CodSpeed's CPU simulation instrument for consistent, hardware-agnostic performance measurement in your benchmarks ## What is CPU Simulation? CodSpeed instruments your benchmarks to measure the performance of your code, simulating the CPU behavior. A benchmark will be run **only once** and the CPU behavior will be simulated. This ensures that the measurement is as accurate as possible, taking into account not only the instructions executed but also the cache and memory access patterns. The simulation gives us an equivalent of the CPU cycles that includes cache and memory access. ### Estimating Cycles The CPU simulation takes into account the following factors: 1. **Executed instruction cycles** (`Icycles`): the baseline cost of your code, where each instruction is weighted by its cost on real hardware 2. **L1 cache misses**: data that must be fetched from L2/L3 cache, this can take 10-40 cycles 3. **LL (Last-level) cache misses**: data that must be fetched from RAM, this can take 100+ cycles The total number of cycles is calculated like this: $$ \text{cycles} \approx \text{Icycles} + (\text{L1 Misses} \times \text{L2/L3 Cost}) + (\text{LL Misses} \times \text{RAM Cost}) $$
Each executed instruction is charged a cost based on measured cost data for real CPUs, so expensive instructions count for more than cheap ones: ```asm theme={null} imul rax, rbx ; a few cycles div rbx ; ~20-30 cycles ``` This weighting is what makes the estimate track real hardware. Charging every instruction the same cost would report replacing a `div` with a multiply as almost no change, even though it is significantly faster in practice. For background on how per-instruction latency and throughput are measured, see the [Agner Fog instruction tables](https://www.agner.org/optimize/instruction_tables.pdf) and [uops.info](https://www.uops.info). ### Converting Cycles to Time Once we have the number of cycles for a benchmark, we transform it into an execution time measurement by using the following formula, where `FREQUENCY` is a constant set to the frequency (number of instructions executed per second) of a real CPU: $$ execution\_time = \frac{cycles}{FREQUENCY} $$ We then calculate the **execution speed** of the benchmark by taking the inverse of the execution time: $$ speed = \frac{1}{execution\_time} $$ This is the displayed metric in the CodSpeed reports. **Why choose execution speed over execution time?** A performance increase of a benchmark will increase its execution speed. Same for a performance regression. However, if execution time was used, a performance increase of a benchmark would result in a decrease in its execution time. This would be counter-intuitive. ### System Calls System calls play a critical role in the performance of software, but they present unique challenges for accurate measurement. **A system call is a request made by a program to the operating system's kernel**, typically for I/O operations such as reading from or writing to files, communicating over a network, or interacting with hardware devices. Due to their nature, **system calls introduce variability in execution time**. This variability is influenced by several factors, including system load, network latency, and disk I/O performance. As a result, the execution time of system calls can fluctuate significantly, making them the most inconsistent part of a program's execution time. To ensure that our execution speed measurements are as stable and reliable as possible, **CodSpeed CPU Simulation mode does not include system calls in the measurement**. Instead, we focus solely on the code executed within user space(the code you wrote), excluding any time spent in system calls. This approach allows us to provide a clear and consistent metric for the execution speed of your code, independent of your hardware and all variability that it can create. **Walltime measurement with CodSpeed Macro Runners** If your the code you wish to optimize and measure relies heavily on system calls, you can use CodSpeed Macro Runners combined with our WallTime instrument. You can find more information in the [Walltime](/docs/instruments/walltime) section of the docs. Still, **the wall time spent on system calls is recorded and this data is available in the trace view**, providing insight into how much time is consumed by system interactions. While these times are not included in the overall execution speed metric, they offer valuable information for performance analysis. **Roadmap for system calls** In the future, we plan to enhance CodSpeed by emulating system calls. This will allow us to more accurately anticipate the performance impact of system calls, further improving the reliability and comprehensiveness of our performance measurements. ## Legacy Terminology Previously, this instrument was referred to as "instrumentation" or "instrumentation mode". This terminology is being phased out in favor of "CPU simulation" to better reflect what the instrument does: simulating CPU behavior. While the old `instrumentation` value is still accepted for backward compatibility, it will be removed in a future release. We recommend updating your configuration and your integration library to use `simulation` instead. Going forward, what we will refer to as **"instrumentation"** represents the generic overlay that CodSpeed applies to your benchmarks to collect performance data and profiling information. It applies to both CPU Simulation and Walltime instruments. ## Automated Profiling When using the CPU simulation instrument, CodSpeed automatically collects [profiling data and generates flame graphs](/docs/features/profiling) for each benchmark when available. This allows you to quickly identify performance bottlenecks and their root causes. ### Inspector Metrics When you hover over a span in the flame graph, the inspector displays CPU simulation-specific metrics: Flamegraph inspector

* **Self time**: The simulated execution time spent in the function body only, excluding time spent in child function calls. * **Total time**: The simulated execution time spent in the function including all its children. The time bars are broken down into components that show what is limiting progress: * **Instructions**: Time spent executing CPU instructions. * **Cache**: Time spent due to CPU cache misses (L1, L2, L3). * **Memory**: Time spent waiting for main memory access. This breakdown helps you identify whether a function is instruction-bound, cache-bound, or memory-bound, guiding your optimization efforts. ## Compatibility To enable profiling with the CPU simulation instrument, ensure you meet the following minimum version requirements: | Language | Minimum required versions | | :------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Python](/docs/benchmarks/python) | Python 3.12+ and [`pytest-codspeed>=2.0.0`](https://github.com/CodSpeedHQ/pytest-codspeed/releases/tag/v2.0.0) | | [Rust](/docs/benchmarks/rust) | Any version of [`codspeed-divan-compat`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.8.0) or [`codspeed-criterion-compat`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.9.1) | | [C++](/docs/benchmarks/cpp) | Any version of [`codspeed-google-benchmark`](https://github.com/CodSpeedHQ/codspeed-cpp/releases/tag/v1.0.0) | | [Node.js](/docs/benchmarks/nodejs/overview) | A [supported Node.js version](/docs/benchmarks/nodejs/overview#supported-node-js-versions), plus [`@codspeed/vitest-plugin>=2.3.1`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v2.3.1), [`@codspeed/tinybench-plugin>=2.2.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v2.2.0), or [`@codspeed/benchmark.js-plugin>=2.2.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v2.2.0) | ## Usage with GitHub Actions To enable CPU Simulation in your GitHub Actions workflow, ensure you are using `mode: simulation` in the CodSpeed Action configuration. ## Next Steps Learn how to read flame graphs and use profiling to optimize your code Understand why benchmarks can regress without code changes Measure real-world execution time including system calls and I/O # Reducing Variance Source: https://codspeed.io/docs/instruments/cpu/reducing-variance Learn how to reduce variance in your benchmarks. As explained in the previous chapter on [Benchmark Variance](/docs/instruments/cpu/regression-causes), there are many possible reasons why your code can be variable. There is no silver bullet, but different solutions can be employed to reduce unexpected variance. ## Variance Categories Variance can be separated into different groups, which will help understand and fix multiple regressions. The categories include: * **Compiler/Linker variance**: Whenever the built binary changes, this can cause code to be executed differently. * **Cache variance**: This describes variance caused by different cache behavior. In CI, each benchmark process typically runs once per commit, so cold-cache effects can influence results. * **State-dependent variance**: This describes all the variance that is caused by changing the underlying state of the system. * **Allocator variance**: Allocators can execute different code paths, depending on the current state of the allocator. Changing the memory fragmentation at a previous point in time, can cause variance in benchmarks that are executed later. * **Environment variance**: Variance caused by the runtime environment. * **CPU variance**: If code behaves differently based on the CPU, variance can be introduced. This happens in heavily optimized libraries/programs that might try to detect cache sizes, CPU features or the number of CPU cores. * **Kernel variance**: Syscalls can cause variance in benchmarks, as the kernel might execute different code paths depending on the current state of the system. ```mermaid theme={null} flowchart TD top["Unrelated code change"] -->|"different binary layout"| compiler["Compiler/Linker variance"] top -->|"state changes between runs"| order["State-dependent
Variance"] top --> env["Environment variance"] compiler --> cold["Cache variance"] order --> alloc["Allocator variance"] env --> cpuv["CPU variance"] env --> kernelv["Kernel variance"] ``` ## Strategies ### One benchmark, one binary Most of the issues come from multiple benchmarks being written and run in the same binary. Seemingly unrelated changes to the code, can cause ripple effects that are hard to track down. To fix this, we can compile each benchmark into its own binary. This will fix unrelated variance, as compilers (usually) produce the same binary when given the same input. The only downside to this approach is the increased linker/compilation overhead. For N benchmarks, we will have to compile N binaries. We only recommend this approach for micro-benchmarks which observe a significant amount of variance. #### How to implement in Rust In Rust, this can be done by adding a feature flag for each benchmark, which allows us to compile each benchmark into its own binary. ```toml Cargo.toml theme={null} [features] bench_foo = [] bench_bar = [] ``` Then annotate each benchmark with the feature flag: ```rust highlight={1,8} theme={null} #[cfg(feature = "bench_foo")] #[divan::bench] fn bench_foo() { // // } #[cfg(feature = "bench_bar")] #[divan::bench] fn bench_bar() { // // } ``` Then run like this: ```bash theme={null} $ cargo codspeed build -m simulation --features bench_foo $ cargo codspeed run -m simulation $ cargo codspeed build -m simulation --features bench_bar $ cargo codspeed run -m simulation ``` For now, it's only possible to build and execute a single benchmark at a time, but we're exploring how to better integrate this into cargo-codspeed. ```yaml highlight={17,23} theme={null} name: CodSpeed Benchmarks on: [push, pull_request] jobs: benchmarks: runs-on: ubuntu-latest strategy: matrix: benchmark: [bench_foo, bench_bar] steps: - uses: actions/checkout@v5 - uses: dtolnay/rust-toolchain@stable - name: Build benchmark run: cargo codspeed build -m simulation --features ${{ matrix.benchmark }} - name: Run benchmark uses: CodSpeedHQ/action@v5 with: mode: simulation run: cargo codspeed run -m simulation ``` #### How to implement in C++ When using C++, we can achieve this by wrapping each `BENCHMARK()` in a define. This allows us to conditionally include/exclude benchmarks while building. ```cpp highlight={1,11} theme={null} #ifdef BENCHMARK_BM_StringCopy static void BM_StringCopy(benchmark::State &state) { std::string x = "hello"; for (auto _ : state) { std::string copy(x); benchmark::DoNotOptimize(copy); benchmark::ClobberMemory(); } } BENCHMARK(BM_StringCopy); #endif ``` Then build each benchmark into its own binary: ```bash theme={null} for define in $(rg -oN 'BENCHMARK_\w+' src | sort -u); do cmake -S . -B "build/$define" -DCODSPEED_MODE=simulation -D"$define"=ON cmake --build "build/$define" cp "build/$define/" "codspeed-results/$define" done ``` Then run each benchmark: ```bash theme={null} for define in $(rg -oN 'BENCHMARK_\w+' src | sort -u); do ./codspeed-results/$define done ``` In GitHub Actions we then do the same: ```yaml highlight={13-17, 24-26} theme={null} name: CodSpeed Benchmarks on: [push, pull_request] jobs: benchmarks: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - name: Build all benchmarks run: | for define in $(rg -oN 'BENCHMARK_\w+' src | sort -u); do cmake -S . -B "build/$define" -DCODSPEED_MODE=simulation -D"$define"=ON cmake --build "build/$define" cp "build/$define/" "codspeed-results/$define" done - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation run: | for define in $(rg -oN 'BENCHMARK_\w+' src | sort -u); do ./codspeed-results/$define done ``` ### Reducing allocator variance #### Tune your allocator Most allocators expose configuration options that affect determinism. Reducing the number of arenas, disabling caches, and controlling page purging behavior can all help stabilize benchmark results. Refer to the allocator documentation for available options: * [glibc `mallopt`](https://man7.org/linux/man-pages/man3/mallopt.3.html) * [jemalloc `MALLOC_CONF`](https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md) * `dirty_decay_ms:-1,muzzy_decay_ms:-1`: This disables returning unused pages back to the OS, which can otherwise randomly slowdown benchmarks. * [tcmalloc](https://google.github.io/tcmalloc/tuning.html) * [`SetProfileSamplingInterval(MAX)`](https://github.com/google/tcmalloc/blob/master/docs/sampling.md): Disables heap profile sampling * [`SetGuardedSamplingInterval(-1)`](https://github.com/google/tcmalloc/blob/b90d4ac374850b0bec6bbf9b520e8afcb6496517/tcmalloc/malloc_extension.h#L527C1-L534C60): Disables [GWP-ASan](https://google.github.io/tcmalloc/gwp-asan.html) guarded sampling, which otherwise probabilistically guards allocations to detect buffer overflows and use-after-free. * [`SetBackgroundProcessActionsEnabled(false)`](https://github.com/google/tcmalloc/blob/b90d4ac374850b0bec6bbf9b520e8afcb6496517/tcmalloc/malloc_extension.h#L555): Disables background memory release actions that can cause timing variance. #### Exclude allocations entirely If the allocator time is not relevant to what you are measuring, exclude allocation time from your reported results. See [Allocation Exclusion](/docs/instruments/cpu/allocation-exclusion). #### Use a custom allocator In many cases, variance is caused by `realloc` which either grows the allocation in place, or creates a new allocation and moves the previous allocation to the new one. Whether in-place growing succeeds depends on the OS memory state, making it **completely non-deterministic**. To fix this, we can always run the slow-path that never grows in-place. Here is an example in Rust (adapted from [oxc](https://github.com/oxc-project/oxc/blob/112580a408843f9f405d8d8acc9d03990a75eaff/tasks/benchmark/src/lib.rs)), which uses the default implementation of [`GlobalAlloc::realloc`](https://github.com/rust-lang/rust/blob/d933cf483edf1605142ac6899ff32536c0ad8b22/library/core/src/alloc/global.rs#L286-L303) that always allocates and then copies the memory. ```rust theme={null} use std::alloc::{GlobalAlloc, Layout, System}; #[global_allocator] static GLOBAL: NeverGrowInPlaceAllocator = NeverGrowInPlaceAllocator; struct NeverGrowInPlaceAllocator; unsafe impl GlobalAlloc for NeverGrowInPlaceAllocator { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { System.alloc(layout) } unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { System.dealloc(ptr, layout); } } ``` *** We're actively exploring how to implement this in our integrations. If you have further questions, please reach out to us via [Discord](https://discord.gg/MxpaCfKSqF) or [email](mailto:contact@codspeed.io). # Benchmark Variance Source: https://codspeed.io/docs/instruments/cpu/regression-causes Learn why micro-benchmarks can improve/regress despite no code changes, and how to identify the causes. CodSpeed's CPU Simulation is based on Valgrind, which operates on **compiled machine code** produced by your toolchain. The machine code is executed on a simulated CPU, which ensures that the performance is consistent across multiple runs. However, there are cases where the performance of a benchmark regresses, despite not making any changes to the underlying code. This can happen due to changes in the cache behavior, as cache misses are also included in the [cycle calculation](/docs/instruments/cpu#estimating-cycles). This article explains **regressions in micro-benchmarks**, caused by different cache behavior. These regressions are typically \< 1μs, which is why they are often not noticeable in walltime measurements. ## Toolchain Updates It is generally recommended to pin your toolchain (e.g. compilers, dependencies, etc.) to avoid unintended changes, which can be bugs, malware but also **performance regressions**. **Common issues:** * **Stable compiler toolchains:** Using `dtolnay/rust-toolchain@stable` in Github Actions is non-deterministic. When Rust stable updates (e.g. from 1.92 to 1.93), your toolchain and Rust compiler will update, which can change [compiler optimizations](#compiler-non-determinism). * **CI runner/image updates:** `runs-on: ubuntu-latest` can move to a new Ubuntu release, changing glibc, LLVM, and other system libraries. * **Dependency updates:** Changes in `Cargo.lock` (new crate versions) or JS/Python lockfiles can alter inlining decisions and code layout. * **Target architecture changes:** Compiling for different CPU microarchitectures (e.g. `x86-64-v2` vs `x86-64-v3` vs `x86-64-v4`) enables different instruction sets (SSE, AVX, AVX-512). In general, try to avoid using `latest` or `stable` tags and use a specific version instead. Commit your lockfiles to version control to ensure reproducible builds. ## CI Runner Variability Shared CI runners (GitHub Actions, GitLab CI, CircleCI) don't guarantee the same physical machine between jobs. The base and head runs can end up on different hardware: * **CPU model:** Different instruction sets, cache sizes, and microarchitectures change simulated cache behavior. * **System libraries:** Library versions can differ even on the same OS image if it was updated between runs. CodSpeed detects this and shows a warning: CodSpeed warning showing different runtime environments between base and head runs, with CPU brand mismatch highlighted

To fix this, use CodSpeed [Macro Runners](/docs/features/macro-runners): dedicated bare-metal machines where your benchmarks always run on the same hardware. If you manage your own runners, pin them to a single CPU type. For a deep dive, see our blog post [Why glibc Is Faster on GitHub Actions](https://codspeed.io/blog/why-glibc-faster-github-actions), that explores how CPU differences across CI runners cause benchmark variance. ## Compiler Non-determinism **When can this happen?** Any change to the code or compiler can trigger different compiler decisions. However, recompiling the entire codebase with the same source code, compiler version, and flags is usually deterministic. Compiling optimized code is hard, because it is a tradeoff between compilation resource usage (speed, memory, ...), runtime execution speed and binary size, because you don't want your simple code to take 1 hour to compile or take up 1GB of disk space. This tradeoff is balanced by using heuristics and thresholds, that cover most cases while being fast enough. An example for this is **inlining**: By inlining a function, the overhead of a call at runtime is removed and also allows the compiler to better optimize the function body. However, if every function is inlined the binary would be much bigger, which decreases the performance due to the [increased instruction cache pressure](https://stackoverflow.com/questions/49334487/inlining-and-instruction-cache-hit-rates-and-thrashing). There are many other optimizations that can affect cache behavior: * **Basic block reordering**: Moving cold error paths into separate functions, rearranging `if` branches, ... * **Loop transformations**: Loop unrolling, peeling, fusing, ... * **Bounds checks**: Compilers are often smart enough to eliminate bounds checks, but if they are not, they may hinder loop unrolling/vectorization. **How to detect this?** We recommend checking the cache misses and instruction counts in the tooltip of the flamegraph. ### Function Alignment Do you think this function always has the same performance? ```asm theme={null} ; rax = rdi + rsi foo: mov rax, rdi jmp label ; a lot of other code label: add rax, rsi ret ``` The answer is, it depends. The CPU fetches the next N instructions that should be executed and stores them in the I-cache (Instruction Cache). If the label is far away from the `foo` function, the CPU may need to fetch another cache line to get the instructions after the `jmp label` instruction. This is counted as an instruction cache miss, which may cost anywhere from 10-40 cycles (if found in L2/L3) to 100+ cycles (if it goes all the way to RAM). Because of that, compilers try to align functions and keep the hot paths close to each other to minimize cache misses. ## Allocators Most allocators are designed to be fast, while keeping fragmentation and memory usage at a minimum. Just like compilers, they use heuristics to decide when to allocate more memory, which can lead to unpredictable performance. Here are some examples that can cause different performance: * **Time-Based Memory Decay**: Allocators like [jemalloc](https://github.com/jemalloc/jemalloc) implement "decay" logic, where unused "dirty" memory pages are returned to the OS after a specific duration (e.g., 10 seconds). * **Adaptive Thread-Cache Sizing**: High-performance allocators (like [tcmalloc](https://github.com/google/tcmalloc)) dynamically resize thread-local caches based on "demand history." * **Memory Fragmentation Patterns**: Allocation order determines fragmentation. If one benchmark allocates 1MB while another allocates 64B, then the allocator may have to allocate more memory to satisfy the larger request, leading to fragmentation. Detecting allocator regressions is straightforward, because we can see the reduced performance of the allocator functions in the flamegraph: Flamegraph showing an allocator regression

Flamegraph showing an allocator regression

You can exclude this variance source entirely with [Allocation Exclusion](/docs/instruments/cpu/allocation-exclusion), which subtracts allocator time from your reported benchmark results. ## HashMaps Most hash map implementations randomize their hash seed on every program start. Rust's `std::collections::HashMap`, Python's `dict`, and Go's `map` all do this by default. The reason is security: a fixed seed lets an attacker craft inputs that all hash to the same bucket, turning O(1) lookups into O(n) and creating a denial-of-service vector (HashDoS). For a benchmark, that randomization shows up as run-to-run variance even when the input is identical: * **Bucket layout**: Keys land in different buckets between runs. Probe sequences differ, which changes the cache lines touched on each lookup. * **Iteration order**: Iterating a `HashMap` produces a different order on every run, so any work that depends on iteration order (allocations, recursive calls, downstream hashing) takes a different path. * **Resize timing**: With a different bucket distribution, the map hits its load-factor threshold at a different insertion, shifting where the next allocation happens and how big the peak working set gets. To remove this source of variance, swap the default hasher for a deterministic one in your benchmark, or use an ordered container like `BTreeMap` if iteration order matters. ## Filesystem Iteration Order Reading a directory on Linux using (e.g. using [`readdir`](https://man7.org/linux/man-pages/man3/readdir.3.html), or libraries built on top of it) does not return entries in any particular order. The reason for this is that the underlying implementations vary based on the filesystem (ext4, btrfs, xfs all behave differently). This means that the benchmarks can be fully deterministic when run on a single machine (and therefore the same filesystem), but show significant variance when run on different filesystems. Changes in the filesystem iteration order can have an impact on: * **Cache behavior**: Files get processed in a different order, so the data loaded into the page cache and the CPU caches differs between runs. * **Allocation order**: When per-file work allocates, the allocator sees a different request sequence. Peak memory and fragmentation change even though the total work is identical. * **Order-dependent code**: Anything downstream that consumes the iteration order (sorting later, hashing into a map, writing output) takes a different path. To remove this source of variance, sort the directory entries before iterating over them. ## Next Steps Now that you understand the common causes of benchmark regressions, you can use CodSpeed's profiling tools to identify them in your code. Learn how to read flamegraphs and find performance bottlenecks Learn strategies to reduce variance in your benchmarks # MongoDB Instrumentation Overview Source: https://codspeed.io/docs/instruments/databases/mongodb/index MongoDB instrument to measure the performance of queries directly in the CI This feature is currently in beta and is subject to change. If you are interested in testing it out, please reach out to us via [Discord](https://discord.gg/MxpaCfKSqF) or [email out support](mailto:contact@codspeed.io). [MongoDB](https://www.mongodb.com/) instrument to measure the performance of queries directly in the CI. ## Getting Started ### Prerequisites * MongoDB instrument enabled in your CodSpeed account ([reach out to us](mailto:contact@codspeed.io) to enable) * MongoDB instance running in your CI environment or the cloud We recommend: - using the [`art049/mongodb-cluster-action`](https://github.com/art049/mongodb-cluster-action) to run MongoDB in your CI environment - dynamic setup of a MongoDB instance during the setup phase of the benchmarks is supported, using tools such as [`testcontainers`](https://node.testcontainers.org/modules/mongodb/) or [`vitest-mongodb`](https://github.com/enochchau/vitest-mongodb/) ### Activate the `mongodb` instrument in the CodSpeed Action To activate the MongoDB instrument during the run of your benchmarks in the CodSpeed Action, make the following changes to your workflow file: ```yaml {6,11} theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 run: with: mode: simulation # Activate the MongoDB instrument instruments: mongodb # (Optional) Specify the name of the environment variable that contains the MongoDB # connection string and is used by your application to connect to MongoDB. # If not specified, you will have to provide the connection string dynamically during # the setup phase of your benchmarks. mongo-uri-env-name: MONGO_URL ``` **Using `mongo-uri-env-name`** If using the `mongo-uri-env-name` option, make sure to set the actual value of the environment variable in the `env` section of your workflow file. Otherwise, the MongoDB instrument will not be able to connect to your MongoDB instance. ```yaml {8} theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 run: with: mode: simulation instruments: mongodb mongo-uri-env-name: MONGO_URL env: MONGO_URL: mongodb://localhost:27017 ``` **Dynamic connection string** If the value of connection string cannot be set in the workflow file, you can omit the `mongo-uri-env-name` option and provide the connection string dynamically during the setup phase of your benchmarks. ```yaml theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation run: instruments: mongodb ``` ## Language specific examples If you are using the `mongo-uri-env-name` option, the integration should work out of the box at this stage 🎉 However, if you are not using the `mongo-uri-env-name` option, you will have to provide the connection string dynamically during the setup phase of your benchmarks. Select your language below to see how to do that. To view examples and how to dynamically provide the connection string during the setup phase of your benchmarks, select your language below: } /> * **Python**: Not supported yet (*coming soon*) * **Rust**: Not supported yet (*coming soon*) # NodeJS setup for MongoDB instrumentation Source: https://codspeed.io/docs/instruments/databases/mongodb/nodejs/index MongoDB instrument for NodeJS Make sure you have [activated the `mongodb` instrument](/docs/instruments/databases/mongodb/#activate-the-mongodb-instrument-in-the-codspeed-action) in your CodSpeed GitHub workflow. Make sure you are using the minimum required version of the NodeJS integrations: * [`@codspeed/vitest-plugin>=3.1.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v3.1.0) * [`@codspeed/tinybench-plugin>=3.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v3.0.0) * [`@codspeed/benchmark.js-plugin>=3.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v3.0.0) ## In-depth guides To see complete guides of the different integrations, check out the pages below: Setup and run MongoDB performance tests in a Node.js API using vitest, leveraging Docker. Setup and run MongoDB performance tests in a Node.js API using tinybench, leveraging Docker. ## Dynamically providing the connection string Each integration exports a `setupInstruments` function that can be used to dynamically setup the instruments. This function takes the actual connection string as an argument and returns the patched connection string that should be used to connect to the database. ```typescript theme={null} type SetupInstrumentsRequestBody = { /** * The full `MONGO_URL` that is usually used to connect to the database. */ mongoUrl: string; }; type SetupInstrumentsResponse = { /** * The patched `MONGO_URL` that should be used to connect to the database. */ remoteAddr: string; }; /** * Dynamically setup the CodSpeed instruments. */ declare function setupInstruments( body: SetupInstrumentsRequestBody ): Promise; ``` You can use this function to set up the instruments in your application: ```typescript src/bench.ts {12} theme={null} import { setupInstruments, withCodSpeed } from "@codspeed/tinybench-plugin"; import { MongoDBContainer } from "@testcontainers/mongodb"; import { registerCatControllerBenches } from "cats/cats.controller.tinybench"; import { Bench } from "tinybench"; async function setupDatabase() { const mongodbContainer = await new MongoDBContainer("mongo:7.0.5").start(); const mongoUrl = mongodbContainer.getConnectionString() + "/test?replicaSet=rs0&directConnection=true"; const { remoteAddr } = await setupInstruments({ mongoUrl }); process.env.MONGO_URL = remoteAddr; } const bench = withCodSpeed(new Bench()); (async () => { await setupDatabase(); registerCatControllerBenches(bench); await bench.run(); console.table(bench.table()); })(); ``` The example above uses the [`testcontainers`](https://node.testcontainers.org/modules/mongodb/) library to start a MongoDB container and get the connection string. Then, the `setupInstruments` function is used to patch the connection string and set it as the `MONGO_URL` environment variable. Finally, the `bench` is run as usual. The `setupInstruments` function should be called **only once** during the whole benchmark run, and **before** any connection to the database is established. Otherwise, the CodSpeed MongoDB instrument will not be able to collect the metrics. # MongoDB Benchmarking Guide for Node.js with tinybench Source: https://codspeed.io/docs/instruments/databases/mongodb/nodejs/tinybench If using [`Vitest`](https://vitest.dev/), is possible in your project, we highly recommend using it instead of `tinybench`. [Click here to see how to use the MongoDB instrument with Vitest](/docs/instruments/databases/mongodb/nodejs/vitest). Make sure you are using the minimum required version of the plugin: [`@codspeed/tinybench-plugin>=3.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v3.0.0) All the code shown on this page is available in the [`CodSpeedHQ/codspeed-nestjs-mongodb` repository](https://github.com/CodSpeedHQ/codspeed-nestjs-mongodb). It uses the following technologies: * [NestJS](https://nestjs.com/) * [MongoDB](https://www.mongodb.com/) * [tinybench](/docs/benchmarks/nodejs/tinybench) * [Docker](https://www.docker.com/) * [`mongoose`](https://mongoosejs.com/) * [`@nestjs/mongoose`](https://docs.nestjs.com/techniques/mongodb) ## Sample application We are going to use a simple NestJS application exposing a REST API to manage cats. The following `Cat` model is defined: ```typescript src/cats/schemas/cat.schema.ts theme={null} import { Prop, Schema, SchemaFactory } from "@nestjs/mongoose"; import { HydratedDocument } from "mongoose"; export type CatDocument = HydratedDocument; @Schema() export class Cat { @Prop({ index: 1, required: true }) name: string; @Prop({ required: true }) age: number; @Prop({ required: true }) breed: string; } export const CatSchema = SchemaFactory.createForClass(Cat); ``` The `CatsController` exposes the following endpoints: ```typescript src/cats/cats.controller.ts theme={null} import { Controller, Get, Param } from "@nestjs/common"; import { CatsService } from "./cats.service"; import { Cat } from "./schemas/cat.schema"; @Controller("cats") export class CatsController { constructor(private readonly catsService: CatsService) {} @Get("name/:name") async findByName(@Param("name") name: string): Promise { return this.catsService.findByName(name); } @Get("breed/:breed") async findByBreed(@Param("breed") breed: string): Promise { return this.catsService.findByBreed(breed); } } ``` ## Complete setup with Docker ### Setup dependencies Install the `@codspeed/tinybench-plugin`: ### Create benchmarks Let's create a script that defines benchmarks on the `cats` endpoints of the application. ```typescript src/cats/cats.controller.e2e.bench.ts theme={null} import { faker } from "@faker-js/faker"; import { INestApplication } from "@nestjs/common"; import { getModelToken } from "@nestjs/mongoose"; import { Test } from "@nestjs/testing"; import { AppModule } from "app.module"; import { Model } from "mongoose"; import request from "supertest"; import { Bench } from "tinybench"; import { CatsFactory } from "./cats.factory"; import { Cat } from "./schemas/cat.schema"; faker.seed(1); // enforce the same seed, to remove randomness from generated data const cats: Cat[] = Array.from({ length: 100 }, () => ({ name: ["river", "felix", "toto", "marcel"][faker.number.int(3)], age: faker.number.int(20), breed: ["chausie", "toyger", "abyssinian", "birman"][faker.number.int(3)], })); export function registerCatControllerBenches(bench: Bench) { let app: INestApplication; let catsModel: Model; let catsFactory: CatsFactory; // initialize the application before the benchmark async function beforeAll() { const moduleRef = await Test.createTestingModule({ imports: [AppModule], }).compile(); app = moduleRef.createNestApplication(); catsModel = moduleRef.get(getModelToken(Cat.name)); catsFactory = new CatsFactory(catsModel); await app.init(); await catsFactory.createMany(cats); } // clean up the application after the benchmark async function afterAll() { await catsModel.deleteMany(); await app.close(); } bench.add( "GET /cats/name/:name", async () => { await request(app.getHttpServer()).get("/cats/name/river"); }, { beforeAll, afterAll } ); bench.add( "GET /cats/breed/:breed", async () => { await request(app.getHttpServer()).get("/cats/breed/chausie"); }, { beforeAll, afterAll } ); } ``` Here we have defined 4 benchmarks for the `cats` endpoints: * `GET /cats`: retrieve all the cats * `GET /cats/name/:name`: retrieve all the cats with the given name * `GET /cats/breed/:breed`: retrieve all the cats with the given breed * `GET /cats/age/greater/:age`: retrieve all the cats with an age greater than the given age We finally have to register the benchmarks in the `src/bench.e2e.ts` file: ```typescript src/bench.e2e.ts theme={null} import { withCodSpeed } from "@codspeed/tinybench-plugin"; import { registerCatControllerBenches } from "cats/cats.controller.e2e.bench"; import { Bench } from "tinybench"; const bench = withCodSpeed(new Bench()); (async () => { registerCatControllerBenches(bench); await bench.run(); console.table(bench.table()); })(); ``` ### Setup Docker locally Add the following file to the root of the project: ```yaml docker-compose.yml theme={null} version: "3" services: mongodb: image: mongo:latest environment: - MONGODB_DATABASE="test" ports: - 27017:27017 volumes: - mongo:/data/db volumes: mongo: ``` Run the following command to start the MongoDB instance: ```bash theme={null} docker-compose up -d ``` ### Run the benchmarks locally To use `tinybench`, we recommend using `ts-node` with `swc`: To enforce using `swc` when running `ts-node`, add the following to your `tsconfig.json`: ```js tsconfig.json theme={null} { "ts-node": { "swc": true } } ``` Add the following script to your `package.json`: ```js package.json theme={null} { "scripts": { "bench:e2e": "NODE_ENV=test ts-node --swc -r tsconfig-paths/register src/bench.e2e.ts" } } ``` Run the following command to run the benchmarks: ```shellsession title=terminal icon="square-terminal" theme={null} $ pnpm bench:e2e ┌─────────┬──────────────────────────────┬─────────┬────────────────────┬──────────┬─────────┐ │ (index) │ Task Name │ ops/sec │ Average Time (ns) │ Margin │ Samples │ ├─────────┼──────────────────────────────┼─────────┼────────────────────┼──────────┼─────────┤ │ 0 │ 'GET /cats/name/:name' │ '260' │ 3840144.435868008 │ '±9.38%' │ 131 │ │ 1 │ 'GET /cats' │ '348' │ 2870076.392037528 │ '±4.93%' │ 175 │ │ 2 │ 'GET /cats/breed/:breed' │ '489' │ 2043167.1677803504 │ '±3.39%' │ 245 │ │ 3 │ 'GET /cats/age/greater/:age' │ '431' │ 2318777.595405225 │ '±3.97%' │ 216 │ └─────────┴──────────────────────────────┴─────────┴────────────────────┴──────────┴─────────┘ ``` ### Run the benchmarks in the CI Add the following file to the project: ```yaml .github/workflows/codspeed.yml {23-25,30-42} theme={null} name: CodSpeed on: push: branches: - "main" # or "master" pull_request: # required to have reports on PRs # `workflow_dispatch` allows CodSpeed to trigger backtest # performance analysis in order to generate initial data. workflow_dispatch: jobs: benchmarks: name: Run benchmarks runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - uses: pnpm/action-setup@v2 - uses: actions/setup-node@v6 with: cache: pnpm node-version-file: .nvmrc # easily setup a MongoDB cluster - uses: art049/mongodb-cluster-action@v0 id: mongodb-cluster-action - name: Install dependencies run: pnpm install - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation instruments: mongodb mongo-uri-env-name: MONGO_URL run: | pnpm bench:e2e env: # we need the MONGO_URL to be set in the environment before actually running # the benchmark command so we set it here instead of inside the `run` command MONGO_URI: ${{ steps.mongodb-cluster-action.outputs.connection-string }} ``` With this configuration, the CodSpeed MongoDB instrument will be activated and data from MongoDB queries will be sent to CodSpeed. ## Setup using testcontainers Instead of relying on an externally provided Docker instance, we can leverage [`testcontainers`](https://node.testcontainers.org/modules/mongodb/) to start a MongoDB instance dynamically during the benchmarks. For this setup, we assume that the state of the application is similar to the one described in the above section. ### Setup tinybench + testcontainers Install the `testcontainers` dependencies: Change the `src/bench.e2e.ts` file to the following: ```typescript src/bench.e2e.ts theme={null} import { setupInstruments, withCodSpeed } from "@codspeed/tinybench-plugin"; import { MongoDBContainer } from "@testcontainers/mongodb"; import { registerCatControllerBenches } from "cats/cats.controller.tinybench"; import { Bench } from "tinybench"; async function setupDatabase() { const mongodbContainer = await new MongoDBContainer("mongo:7.0.5").start(); const mongoUrl = mongodbContainer.getConnectionString() + "/test?replicaSet=rs0&directConnection=true"; const { remoteAddr } = await setupInstruments({ mongoUrl }); process.env.MONGO_URL = remoteAddr; } const bench = withCodSpeed(new Bench()); (async () => { await setupDatabase(); registerCatControllerBenches(bench); await bench.run(); console.table(bench.table()); })(); ``` On macOS, we recommend using [`colima`](https://github.com/abiosoft/colima) to run Docker containers. However there are [issues using `testcontainers` on macOS](https://node.testcontainers.org/supported-container-runtimes/#colima). To bypass those issues, some environment variables need to be set when running the tests: To make `testcontainers` work on macOS with `colima`, the following environment variables need to be set: ```bash theme={null} TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock NODE_OPTIONS="$NODE_OPTIONS --dns-result-order=ipv4first" ``` We will add a function to enforce that they are set when running `tinybench`. Add the following function to your `src/bench.e2e.ts` file: ```typescript src/bench.e2e.ts theme={null} function checkColimaTestcontainersDarwin() { if ( process.platform === "darwin" && (process.env.TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE === undefined || !process.env.NODE_OPTIONS.includes("--dns-result-order=ipv4first")) ) { throw new Error( 'On macOs, run with the following command to make testcontainers + colima work: `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock NODE_OPTIONS="$NODE_OPTIONS --dns-result-order=ipv4first" `' ); } } ``` And use it at the top of the `setupDatabase` function: ```typescript src/bench.e2e.ts {2} theme={null} async function setupDatabase() { checkColimaTestcontainersDarwin(); await setupMongoDB(); } ``` Now the execution will stop with an explicit error message if the environment variables are not set when running on macOS. ### Run the benchmarks locally You can now run the benchmarks locally without having to start a MongoDB instance: ```bash theme={null} pnpm bench:e2e ``` ### Run the benchmarks in the CI You can now simplify the `codspeed.yml` file to the following: * Remove the `mongodb-cluster-action` step * Remove the `mongo-uri-env-name` input * Remove the `MONGO_URI` environment variable ```yaml .github/workflows/codspeed.yml {26-32} theme={null} name: CodSpeed on: push: branches: - "main" # or "master" pull_request: # required to have reports on PRs # `workflow_dispatch` allows CodSpeed to trigger backtest # performance analysis in order to generate initial data. workflow_dispatch: jobs: benchmarks: name: Run benchmarks runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - uses: pnpm/action-setup@v2 - uses: actions/setup-node@v6 with: cache: pnpm node-version-file: .nvmrc - name: Install dependencies run: pnpm install - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation instruments: mongodb run: pnpm bench:e2e ``` # MongoDB Benchmarking Guide for Node.js with vitest Source: https://codspeed.io/docs/instruments/databases/mongodb/nodejs/vitest Make sure you are using the minimum required version of the plugin: [`@codspeed/vitest-plugin>=3.1.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v3.1.0) All the code shown on this page is available in the [`CodSpeedHQ/codspeed-nestjs-mongodb` repository](https://github.com/CodSpeedHQ/codspeed-nestjs-mongodb). It uses the following technologies: * [NestJS](https://nestjs.com/) * [MongoDB](https://www.mongodb.com/) * [Vitest](/docs/benchmarks/nodejs/vitest) * [Docker](https://www.docker.com/) * [`mongoose`](https://mongoosejs.com/) * [`@nestjs/mongoose`](https://docs.nestjs.com/techniques/mongodb) ## Sample application We are going to use a simple NestJS application exposing a REST API to manage cats. The following `Cat` model is defined: ```typescript src/cats/schemas/cat.schema.ts theme={null} import { Prop, Schema, SchemaFactory } from "@nestjs/mongoose"; import { HydratedDocument } from "mongoose"; export type CatDocument = HydratedDocument; @Schema() export class Cat { @Prop({ index: 1, required: true }) name: string; @Prop({ required: true }) age: number; @Prop({ required: true }) breed: string; } export const CatSchema = SchemaFactory.createForClass(Cat); ``` The `CatsController` exposes the following endpoints: ```typescript src/cats/cats.controller.ts theme={null} import { Controller, Get, Param } from "@nestjs/common"; import { CatsService } from "./cats.service"; import { Cat } from "./schemas/cat.schema"; @Controller("cats") export class CatsController { constructor(private readonly catsService: CatsService) {} @Get("name/:name") async findByName(@Param("name") name: string): Promise { return this.catsService.findByName(name); } @Get("breed/:breed") async findByBreed(@Param("breed") breed: string): Promise { return this.catsService.findByBreed(breed); } } ``` ## Complete setup with Docker ### Setup SWC + Vitest To use `swc` and `vitest` with `nest-js`, follow the [setup guide on the NestJS website](https://docs.nestjs.com/recipes/swc#vitest). At the end of this setup, you should be able to run e2e tests with a running MongoDB instance, using the following command: ```bash theme={null} pnpm test:e2e # equivalent to pnpm vitest run --config ./vitest.config.e2e.ts ``` ### Setup CodSpeed and Vitest for e2e benchmarks Install the dependencies: `vite-tsconfig-paths` is used to resolve the paths defined in the `tsconfig.json` file automatically. Rename the file `vitest.config.e2e.ts` to `vitest.config.e2e.mts` since `@codspeed/vitest-plugin` is only available in ESM. Apply the following modifications to the file: ```typescript vitest.config.e2e.mts {7, 12-15} theme={null} import codspeedPlugin from "@codspeed/vitest-plugin"; import swc from "unplugin-swc"; import tsconfigPaths from "vite-tsconfig-paths"; import { defineConfig } from "vitest/config"; export default defineConfig({ plugins: [swc.vite(), tsconfigPaths(), codspeedPlugin()], test: { root: "./", passWithNoTests: true, include: ["**/*.e2e.spec.ts"], benchmark: { include: ["**/*.e2e.bench.ts"] }, // ensure we running only one test at a time since they are using the same database // this could be removed by using a different database for each test poolOptions: { forks: { singleFork: true } }, }, }); ``` ### Create benchmarks Similar to how we would create an e2e test in NestJS in a `*.e2e.spec.ts` file, we can create a benchmark in a `*.e2e.bench.ts` file. ```typescript src/cats/cats.controller.e2e.bench.ts theme={null} import { faker } from "@faker-js/faker"; import { INestApplication } from "@nestjs/common"; import { getModelToken } from "@nestjs/mongoose"; import { Test } from "@nestjs/testing"; import { AppModule } from "app.module"; import { Model } from "mongoose"; import request from "supertest"; import { afterAll, beforeAll, beforeEach, bench, describe, expect, } from "vitest"; import { CatsFactory } from "./cats.factory"; import { Cat } from "./schemas/cat.schema"; faker.seed(1); // enforce the same seed, to remove randomness from generated data const cats: Cat[] = Array.from({ length: 100 }, () => ({ name: ["river", "felix", "toto", "marcel"][faker.number.int(3)], age: faker.number.int(20), breed: ["chausie", "toyger", "abyssinian", "birman"][faker.number.int(3)], })); describe("Cats (bench)", () => { let app: INestApplication; let catsModel: Model; let catsFactory: CatsFactory; // initialize the application before the benchmarks beforeAll(async () => { const moduleRef = await Test.createTestingModule({ imports: [AppModule], }).compile(); app = moduleRef.createNestApplication(); catsModel = moduleRef.get(getModelToken(Cat.name)); catsFactory = new CatsFactory(catsModel); await app.init(); }); // reset the database before each benchmark beforeEach(async () => { await catsModel.deleteMany(); await catsFactory.createMany(cats); }); afterAll(async () => { await app.close(); }); bench("GET /cats/name/:name", async () => { const response = await request(app.getHttpServer()).get("/cats/name/river"); // the response should contain 29 cats with the name "river" expect(response.body).toHaveLength(29); expect(response.body[0]).toEqual( expect.objectContaining({ _id: expect.any(String), age: expect.any(Number), breed: expect.any(String), name: "river", }) ); }); bench("GET /cats/breed/:breed", async () => { const response = await request(app.getHttpServer()).get( "/cats/breed/chausie" ); // the response should contain 27 cats with the breed "chausie" expect(response.body).toHaveLength(27); }); }); ``` Here we have defined 4 benchmarks for the `cats` endpoints: * `GET /cats`: retrieve all the cats * `GET /cats/name/:name`: retrieve all the cats with the given name * `GET /cats/breed/:breed`: retrieve all the cats with the given breed * `GET /cats/age/greater/:age`: retrieve all the cats with an age greater than the given age Note the use the usage of [`expect`](https://vitest.dev/api/expect.html#expect) in the benchmarks. ```typescript theme={null} const response = await request(app.getHttpServer()).get("/cats/name/river"); expect(response.body).toHaveLength(29); expect(response.body[0]).toEqual( expect.objectContaining({ _id: expect.any(String), age: expect.any(Number), breed: expect.any(String), name: "river", }) ); ``` This allows for a better experience when authoring benchmarks, as it provides a way to ensure that everything went well. This is optional, you can remove the assertions if you want: ```typescript theme={null} await request(app.getHttpServer()).get("/cats/name/river"); ``` ### Setup Docker locally Add the following file to the root of the project: ```yaml docker-compose.yml theme={null} version: "3" services: mongodb: image: mongo:latest environment: - MONGODB_DATABASE="test" ports: - 27017:27017 volumes: - mongo:/data/db volumes: mongo: ``` Run the following command to start the MongoDB instance: ```bash theme={null} docker-compose up -d ``` ### Run the benchmarks locally Add the following script to your `package.json`: ```js package.json theme={null} { "scripts": { "bench:e2e": "vitest -c vitest.config.e2e.mts bench" } } ``` Run the following command to run the benchmarks: ```shellsession title=terminal icon="square-terminal" theme={null} $ pnpm bench:e2e Benchmarking is an experimental feature. Breaking changes might not follow SemVer, please pin Vitest's version when using it. DEV v1.2.0 /Users/user/projects/CodSpeedHQ/codspeed-nestjs-mongodb [CodSpeed] @codspeed/vitest-plugin v5.0.1 - setup [CodSpeed] running suite src/cats/cats.controller.e2e.bench.ts [CodSpeed] src/cats/cats.controller.e2e.bench.ts::Cats (bench)::GET /cats/name/:name done [CodSpeed] src/cats/cats.controller.e2e.bench.ts::Cats (bench)::GET /cats/breed/:breed done [CodSpeed] running suite src/cats/cats.controller.e2e.bench.ts done ✓ src/cats/cats.controller.e2e.bench.ts (2) 698ms · Cats (bench) (2) ``` ### Run the benchmarks in the CI Add the following file to the project: ```yaml .github/workflows/codspeed.yml {23-25,29-42} icon="github" theme={null} name: CodSpeed on: push: branches: - "main" # or "master" pull_request: # required to have reports on PRs # `workflow_dispatch` allows CodSpeed to trigger backtest # performance analysis in order to generate initial data. workflow_dispatch: jobs: benchmarks: name: Run benchmarks runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - uses: pnpm/action-setup@v2 - uses: actions/setup-node@v6 with: cache: pnpm node-version-file: .nvmrc # easily setup a MongoDB cluster - uses: art049/mongodb-cluster-action@v0 id: mongodb-cluster-action - name: Install dependencies run: pnpm install - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation instruments: mongodb mongo-uri-env-name: MONGO_URL run: pnpm bench:e2e env: # we need the MONGO_URL to be set in the environment before actually running # the benchmark command so we set it here instead of inside the `run` command MONGO_URL: ${{ steps.mongodb-cluster-action.outputs.connection-string }} ``` With this configuration, the CodSpeed MongoDB instrument will be activated and data from MongoDB queries will be sent to CodSpeed. ## Setup using testcontainers Instead of relying on an externally provided Docker instance, we can leverage [`testcontainers`](https://node.testcontainers.org/modules/mongodb/) to start a MongoDB instance dynamically during the benchmarks. For this setup, we assume that the state of the application is similar to the one described in the above section. ### Setup Vitest + testcontainers Install the `testcontainers` dependencies: Create a new file `src/global.d.ts` with the following content: ```typescript src/global.d.ts theme={null} declare var __MONGO_URI__: string; ``` This will make the `globalThis.__MONGO_URI__` variable available in the whole application with the correct type. ⚠️ Make sure to use `var` and not `let` or `const`, as otherwise the TypeScript type will not be set. Create a new file `src/testUtils/setup-vitest.ts` with the following content: ```typescript src/testUtils/setup-vitest.ts theme={null} import { setupInstruments } from "@codspeed/vitest-plugin"; import { MongoDBContainer, StartedMongoDBContainer, } from "@testcontainers/mongodb"; import { beforeAll } from "vitest"; let mongodbContainer: StartedMongoDBContainer; async function setupMongoDB() { // if the database is already setup so we can skip this step if (globalThis.__MONGO_URI__) return; mongodbContainer = await new MongoDBContainer("mongo:7.0.5").start(); const mongoUrl = mongodbContainer.getConnectionString() + "/test?replicaSet=rs0&directConnection=true"; const { remoteAddr } = await setupInstruments({ mongoUrl }); globalThis.__MONGO_URI__ = remoteAddr; } async function setup() { await setupMongoDB(); } beforeAll(async () => { await setup(); }); ``` **`testcontainers` on macOS** On macOS, we recommend using [`colima`](https://github.com/abiosoft/colima) to run Docker containers. However there are [issues using `testcontainers` on macOS](https://node.testcontainers.org/supported-container-runtimes/#colima). To bypass those issues, some environment variables need to be set when running the tests: To make `testcontainers` work on macOS with `colima`, the following environment variables need to be set: ```bash theme={null} TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock NODE_OPTIONS="$NODE_OPTIONS --dns-result-order=ipv4first" ``` We will add a function to enforce that they are set when running `vitest`. Add the following function to your `src/testUtils/setup-vitest.ts` file: ```typescript src/testUtils/setup-vitest.ts theme={null} function checkColimaTestcontainersDarwin() { if ( process.platform === "darwin" && (process.env.TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE === undefined || !process.env.NODE_OPTIONS.includes("--dns-result-order=ipv4first")) ) { throw new Error( 'On macOs, run with the following command to make testcontainers + colima work: `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock NODE_OPTIONS="$NODE_OPTIONS --dns-result-order=ipv4first" `' ); } } ``` And use it at the top of the the `setupMongoDB` function: ```typescript src/testUtils/setup-vitest.ts theme={null} async function setupMongoDB() { checkColimaTestcontainersDarwin(); ... } ``` Now the execution will stop with an explicit error message if the environment variables are not set when running on macOS. Add the file as a `setupFiles` entry in `vite.config.e2e.mts`: ```typescript vitest.config.e2e.mts {7, 16} theme={null} import codspeedPlugin from "@codspeed/vitest-plugin"; import swc from "unplugin-swc"; import tsconfigPaths from "vite-tsconfig-paths"; import { defineConfig } from "vitest/config"; export default defineConfig({ plugins: [swc.vite(), tsconfigPaths(), codspeedPlugin()], test: { root: "./", passWithNoTests: true, include: ["**/*.e2e.spec.ts"], benchmark: { include: ["**/*.e2e.bench.ts"] }, // ensure we running only one test at a time since they are using the same database // this could be removed by using a different database for each test poolOptions: { forks: { singleFork: true } }, setupFiles: ["./src/testUtils/setup-vitest.ts"], }, }); ``` We can now change the `src/app.module.ts` file to use the `globalThis.__MONGO_URI__` variable instead of the `MONGO_URL` environment variable when it is defined: ```typescript src/app.module.ts theme={null} MongooseModule.forRootAsync({ useFactory: async () => ({ uri: globalThis.__MONGO_URI__ ?? process.env.MONGO_URL, }), }), ``` ### Run the benchmarks locally You can now run the benchmarks locally without having to start a MongoDB instance: ```sh theme={null} pnpm bench:e2e ``` ### Run the benchmarks in the CI You can now simplify the `codspeed.yml` file to the following: * Remove the `mongodb-cluster-action` step * Remove the `mongo-uri-env-name` input * Remove the `MONGO_URI` environment variable ```yaml .github/workflows/codspeed.yml {26-33} theme={null} name: CodSpeed on: push: branches: - "main" # or "master" pull_request: # required to have reports on PRs # `workflow_dispatch` allows CodSpeed to trigger backtest # performance analysis in order to generate initial data. workflow_dispatch: jobs: benchmarks: name: Run benchmarks runs-on: ubuntu-latest permissions: # optional for public repositories contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v5 - uses: pnpm/action-setup@v2 - uses: actions/setup-node@v6 with: cache: pnpm node-version-file: .nvmrc - name: Install dependencies run: pnpm install - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation instruments: mongodb run: pnpm bench:e2e ``` # Database Instrument Overview Source: https://codspeed.io/docs/instruments/databases/overview The database instruments measure the performance of database queries directly in the CI The database instruments allow measuring the performance of **database queries** directly in the CI. For each benchmark, information about the database queries is displayed in the CodSpeed dashboard. The following performance issues can be detected: * **Un-optimized queries**: queries that are not using indexes or are not using them efficiently * **N+1 queries**: queries that are executed multiple times for each item in a list instead of using a single query (*coming soon*) Example of a benchmark with an un-optimized MongoDB query

Example of a benchmark with an un-optimized MongoDB query

The following benefits can be expected from using the database instruments: * Full visibility on the performance of your database queries, **directly in the CI** * **Detect and fix problematic queries** before they are deployed to production ## How it works When enabled, the CodSpeed Action will automatically spawn a proxy server that will intercept all database queries. Each query will be executed against a real database and the response sent back to the application. Databases Instruments architecture

For each query, the CodSpeed proxy will communicate with the real database to gather the **explain plan** and other metrics about it. When the run is finished, the data is sent to CodSpeed where it is processed. ## Supported Databases } /> If there is another database you would like to see supported, please reach out to us via [Discord](https://discord.gg/MxpaCfKSqF) or [email out support](mailto:contact@codspeed.io). # Performance Instruments Source: https://codspeed.io/docs/instruments/index Pick the instrument you want to use. Instruments are a key component of CodSpeed. They are used to measure the performance of your code using different techniques. ## Choosing an Instrument Not sure which instrument to use? This decision tree will help you find the right one: ```mermaid theme={null} flowchart TD start["What do you want to measure?"] -->|" Execution Time "| time["Benchmark has I/O or runs longer than 5s?*"] start -->|" Memory Usage "| memory["Memory "] time -->| No | simulation["Simulation "] time -->| Yes | walltime["Walltime "] click simulation "/instruments/cpu" click walltime "/instruments/walltime" click memory "/instruments/memory" ``` * **[CPU Simulation](/docs/instruments/cpu)**: Simulates CPU behavior to measure performance, taking into account instructions executed, cache and memory access patterns. Benchmarks run only once for consistent, hardware-agnostic results. * **[Walltime](/docs/instruments/walltime)**: Measures real-world execution time on bare-metal runners managed by CodSpeed, with low noise and high precision. * **[Memory](/docs/instruments/memory)**: Captures detailed memory usage and heap allocations to help you identify and optimize allocations. **\*** Simulation is the recommended default, but use Walltime when your benchmark contains I/O (which can introduce variance in simulation) or is too large (simulation adds overhead that slows down long-running benchmarks significantly). ## Language Support | | [CPU Simulation](/docs/instruments/cpu) | [Walltime](/docs/instruments/walltime) | [Memory](/docs/instruments/memory) | | :------------------------------------------ | :-------------------------------------: | :------------------------------------: | :--------------------------------: | | [Python](/docs/benchmarks/python) | | | | | [Rust](/docs/benchmarks/rust) | | | | | [C++](/docs/benchmarks/cpp) | | | | | [Node.js](/docs/benchmarks/nodejs/overview) | | | | | [Go](/docs/benchmarks/go) | | | | | [Java](/docs/benchmarks/java) | | | | If you need an instrument that isn't available for your language yet, let us know via [Discord](https://discord.gg/MxpaCfKSqF) or [email our support](mailto:contact@codspeed.io). ## Executor Instruments Measure performance by simulating the CPU behavior Track memory usage, allocations, and leaks in your benchmarks Measure performance by measuring the real time taken ## Ad Hoc Instruments Those ad hoc instruments can be used on top of an executor instrument to measure specific performance metrics. Measure the performance of database queries # Memory Source: https://codspeed.io/docs/instruments/memory/index Track memory usage, heap allocations, and memory leaks in your benchmarks. The memory instrument captures detailed memory usage of your benchmarks, helping you identify and optimize allocations before shipping to production. ## How does it work? CodSpeed builds your benchmarks to **run only once** while measuring the memory. The profiling is done using a custom eBPF program ensuring stability and minimal overhead (depending on how allocation-heavy the benchmark is). The tracking is done by either instrumenting the dynamically loaded allocator libraries or your benchmark executable (when using a statically linked allocator). We track all allocation related functions (e.g. `malloc`, `free`, ...) in your benchmark. ## What does it measure? Memory metrics for a benchmark, showing peak physical memory, peak allocated memory, average allocation size, total allocated memory and allocation count

Memory metrics for a benchmark, showing peak physical memory, peak allocated memory, average allocation size, total allocated memory and allocation count

* **Peak Physical Memory**: The maximum memory resident in RAM at any single point during execution. This is the most important metric for memory-constrained systems, as it determines the minimum RAM requirements for your application and helps prevent out-of-memory errors. This metric is equivalent to [Resident Set Size (RSS)](https://en.wikipedia.org/wiki/Resident_set_size) * **Peak Allocated Memory**: The maximum memory consumed at any single point during execution. This helps identify memory leaks and excessive memory usage, which can lead to performance degradation or crashes. * **Average Allocation Size**: The average size of each heap allocation. Smaller allocations can lead to better cache locality and less memory fragmentation. * **Total Allocations**: The total amount of heap memory allocated throughout your benchmark execution. Fewer heap allocations typically mean better cache locality and less pressure on the memory allocator, making this a key optimization target for performance-critical code. * **Allocation Count**: The number of individual allocation operations performed during the benchmark. Since each allocation has overhead, high allocation counts can indicate excessive temporary object creation, impacting both performance and memory fragmentation. * **Memory Usage Over Time**: The timeline shows how the peak memory evolves throughout benchmark execution. This graph reveals memory patterns, like steady-state behavior, gradual growth, or periodic spikes. Example of memory showing heap allocations, peak memory usage, allocation count, and memory leak detection in a benchmark

Example of memory showing heap allocations, peak memory usage, allocation count, and memory leak detection in a benchmark

### Allocated vs. physical memory The two metrics answer different questions. Peak Allocated Memory is the largest amount of memory requested through the application's allocator (e.g., `malloc`). Peak Physical Memory is the largest amount of memory resident in RAM. Physical memory can be lower than allocated memory when the application or allocator requests large amounts of memory that it doesn't immediately use. This memory is lazily allocated by the operating system, meaning that it will only be backed by physical memory once it's actually accessed. Physical memory can be higher than allocated memory when: * Fragmentation is high. * The allocator keeps memory after the application frees it. * Large files, executables, shared libraries, or shared memory are included. ## Usage with GitHub Actions **Requirements:** * `CodSpeedHQ/action >= 4` * A supported benchmark framework (see [Language Support](#language-support)) To enable memory in your GitHub Actions workflow, use `mode: memory` in the CodSpeed Action configuration: The CodSpeed action will automatically: * Instrument your benchmarks to capture memory metrics * Run your benchmarks once with memory tracking enabled * Upload results to the CodSpeed dashboard For compiled languages you must pass the `memory` mode when building, this varies across languages: * **Rust**: Pass the [`--measurement-mode`/`-m` flag](/docs/reference/codspeed-rust/cargo-codspeed#the-measurement-mode-flag) to `cargo-codspeed` * **C++ (Bazel/CMake)**: Define the [`CODSPEED_MODE` flag](/docs/benchmarks/cpp#the-codspeed_mode-flag) Checkout the [language-specific docs](/docs/benchmarks/overview) for information on how to build your benchmarks ## Compatibility ### Language Support To use the memory instrument, ensure you meet the following minimum version requirements: | Language | Minimum required versions | | :------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Rust](/docs/benchmarks/rust) | [`codspeed-divan-compat >= 4.2.0`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v4.2.0) or [`codspeed-criterion-compat >= 4.2.0`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v4.2.0) | | [C++](/docs/benchmarks/cpp) | [`codspeed-google-benchmark >= 2.1.0`](https://github.com/CodSpeedHQ/codspeed-cpp/releases/tag/v2.1.0) | | [Python](/docs/benchmarks/python) | [`pytest-codspeed >= 4.3.0`](https://github.com/CodSpeedHQ/pytest-codspeed/releases/tag/v4.3.0) | | [Node.js](/docs/benchmarks/nodejs/overview) | [`@codspeed/vitest-plugin >= 5.2.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.2.0) or [`@codspeed/tinybench-plugin >= 5.2.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.2.0) | If you want to use memory with other languages, please reach out on [Discord](https://discord.gg/MxpaCfKSqF) or [email our support](mailto:contact@codspeed.io). ### Supported Allocators CodSpeed supports both dynamic and statically linked allocators. This is achieved by locating the libraries on your system in common paths, or by detecting allocator functions inside your benchmark binary. However, this means that if your allocator or benchmark binary is in a non-standard path, memory profiling can fail. Currently, the following allocators are supported: * **[libc](https://www.gnu.org/software/libc/)** (glibc, musl) * **libc++ / libstdc++** (C++ standard library) * **[jemalloc](https://jemalloc.net/)** * **[mimalloc](https://github.com/microsoft/mimalloc)** * **[tcmalloc](https://github.com/google/tcmalloc)** We're planning to extend this list. If an allocator is missing, feel free to [open an issue](https://github.com/CodSpeedHQ/codspeed/issues/new) or [contact us](mailto:contact@codspeed.io). ### Kernel support Physical memory tracking is supported on Linux kernel 6.8 and newer. On older kernels, CodSpeed does not collect Peak Physical Memory. GitHub-hosted `ubuntu-22.04` and `ubuntu-24.04` runners currently meet this requirement. ## Limitations * **Instrumentation overhead:** The memory instrument tracks **every individual allocation**, which works best for small benchmarks with \<2M allocations. For bigger benchmarks, try splitting the input or benchmark into smaller parts. * **Multi-threaded variance:** Concurrent threads can allocate in non-deterministic order, causing allocation counts and peak memory to vary between runs. * **Spawned sub-processes:** Allocations in spawned sub-processes are only tracked with dynamically linked allocators. Statically linked allocators (e.g. jemalloc in Rust) are not supported. ## Best Practices To get the most out of memory, consider these recommendations: * **Run benchmarks with realistic workloads** - Use production-representative data sizes and patterns to capture actual memory behavior rather than toy examples * **Focus optimization on hot paths** - Prioritize reducing allocations in frequently called code, as allocation count in hot paths can significantly impact both memory and CPU performance * **Combine with CPU profiling** - Memory and CPU metrics together reveal the full performance story; high allocation counts often correlate with CPU overhead * **Track trends over time** - Compare memory metrics across benchmark runs to catch regressions early and validate that optimizations remain effective as code evolves Just like performance regressions, memory regressions can be caught in CI. If you notice unexpected increases in heap allocations or allocation counts, it's often a sign that code changes have introduced inefficiencies. ## Next Steps Learn about CodSpeed's CPU simulation instrument for performance measurement Learn how to write effective benchmarks for your code Explore automated profiling features for deeper performance insights Complete guide to setting up CodSpeed with GitHub Actions # Walltime Instrument Overview Source: https://codspeed.io/docs/instruments/walltime/index Learn how to use CodSpeed's walltime instrument for measuring real-world execution time in your benchmarks. The walltime instruments allow measuring the **walltime** of your benchmarks directly in the CI. It leverages **bare-metal runners managed and provided by CodSpeed** to measure the performance of your benchmarks with **low noise** and **high precision**. Example of a walltime benchmark run

## What Does the Walltime Instrument Measure? The walltime instrument measures the **actual elapsed time** (also known as "wall clock time") of your benchmark execution. Unlike CPU simulation which measures simulated CPU cycles, walltime captures the real-world duration including: * **All code execution**: Both user-space code and system calls are included in the measurement, giving you a complete picture of actual runtime performance. * **I/O operations**: Network requests, file system operations, and other I/O bound tasks are fully captured, making this instrument ideal for benchmarks that interact with external systems. * **Parallelism effects**: Multi-threaded code benefits are accurately measured since walltime reflects the actual elapsed time, not CPU time across threads. This makes the walltime instrument particularly valuable when you need to measure performance beyond what the CPU simulation instrument can capture, such as integration tests on API endpoints or workloads that rely on external dependencies. **Multiple benchmark processes** With the walltime instrument, you should try and avoid running multiple benchmark processes in parallel since this can lead to noisy measurements. Thus, using `pytest-xdist` or similar tools is not recommended. ## Automated Profiling When using the walltime instrument, CodSpeed automatically collects [profiling data and generates flame graphs](/docs/features/profiling) for each benchmark. This allows you to quickly identify performance changes and their root causes. Function list

### Inspector Metrics Flamegraph inspector

When you hover over a span in the flame graph, the inspector displays the following metrics: **Common metrics:** * **Self time**: The measured execution time spent in the function body only, excluding time spent in child function calls. * **Total time**: The measured execution time spent in the function including all its children. **Execution events** The walltime instrument also collects hardware events during execution. This happens automatically when events are available. All displayed event counts are cumulative and include events from child function calls. * **CPU Cycles**: The number of CPU cycles elapsed. * **Instructions**: The number of CPU instructions executed. * **Memory R/W**: The number of memory read and write operations performed. * **Memory Access Pattern**: A breakdown of how memory accesses were served: * **L1 Cache Hits**: Memory accesses served from the fastest CPU cache. * **L2 Cache Hits**: Memory accesses served from the second-level cache. * **Cache Misses**: Memory accesses that required fetching from main memory. * **Memory access distribution**: Total bytes read from and written to memory, for each level of cache. It is calculated based on the number of events, and the average size of each access, namely a word for a cache access, and a cache line for a cache miss. **Sampling accuracy** Event counts are collected using hardware performance counter sampling. The deeper you navigate into leaf functions, the more susceptible these counts suffer from to sampling-related inaccuracies. For the most reliable data, focus on higher-level functions in the call stack. ## Compatibility To enable profiling with the walltime instrument, ensure you meet the following minimum version requirements: | Language | Minimum required versions | | :------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Python](/docs/benchmarks/python) | Python 3.12+ and [`pytest-codspeed >= 3.0.0`](https://github.com/CodSpeedHQ/pytest-codspeed/releases/tag/v3.0.0) | | [Rust](/docs/benchmarks/rust) | [`codspeed-divan-compat >= 2.8.0`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.8.0) or [`codspeed-criterion-compat >= 2.9.1`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.9.1) | | [C++](/docs/benchmarks/cpp) | [`codspeed-google-benchmark >= 1.0.0`](https://github.com/CodSpeedHQ/codspeed-cpp/releases/tag/v1.0.0) | | [Node.js](/docs/benchmarks/nodejs/overview) | [`@codspeed/vitest-plugin >= 5.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.0.0) or [`@codspeed/tinybench-plugin >= 5.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.0.0) | | [Go](/docs/benchmarks/go) | [`codspeed-go >= 0.3.0`](https://github.com/CodSpeedHQ/codspeed-go/releases/tag/v0.3.0) | | [Java](/docs/benchmarks/java) | JDK 21+ with the [CodSpeed JMH integration](/docs/benchmarks/java) | ## Usage with GitHub Actions **Requirements:** * `CodSpeedHQ/action >= 3.1.0` The CI setup is exactly the same as [the one with the CPU simulation](/docs/integrations/ci/github-actions#2-create-the-benchmarks-workflow) but instead of running on a GitHub-hosted runner, you'll need to request a "codspeed-macro" runner. [Macro Runners](/docs/features/macro-runners) are bare-metal machines managed by CodSpeed and will provide you with a more stable and precise environment to run your benchmarks. Simply replace the `runs-on: ubuntu-latest` line with `runs-on: codspeed-macro` and use the `mode: walltime` option in the action: Your benchmarks will now run on a CodSpeed-managed runner, the action and the benchmark integration will automatically collect walltime data and you'll be able to see the new measurements in the CodSpeed dashboard. **Caching** If you're using caches in the GitHub action workflow, make sure your cache keys include `runner.arch` to avoid cache misses since CodSpeed Macro runners are running on the ARM64 architecture. For example: ```yaml {4} theme={null} - uses: actions/cache@v4 with: path: /home/.cache/pip key: pip-${{ runner.arch }}-${{ hashFiles('pyproject.toml') }} ``` ### Usage on personal GitHub accounts At the moment, the **macro runners are only available for organizations** and not for personal accounts. This is because registering GitHub self-hosted runners on repositories instead of organizations would require the **Repository: Administration (Read/Write)** permission, which is too broad. To use the macro runners on a repository owned by a personal GitHub account, the only solution is to create a new organization and transfer the repository to that organization. ### Usage on public repositories By default, the macro runners are only available for the private repositories of your organization. If you want to use them on a public repository, you'll need to explicitly allow them from your GitHub organization settings (under **Organization Settings** > **Actions** > **Runner groups** > **Default**). Allowing macro runners on a public repository

Allowing macro runners on a public repository

## Next Steps Learn more about CodSpeed managed bare-metal runners designed for Walltime benchmarks Learn more about automated profiling for Walltime benchmarks Learn why system processes are missing from macOS profiles and how to work around it # Profiling System Processes on macOS Source: https://codspeed.io/docs/instruments/walltime/macos-profiling Why macOS System Integrity Protection prevents CodSpeed from profiling Apple system binaries, and how to work around it. On macOS, CodSpeed profiles a benchmark by injecting a small profiling library into each process of the benchmark's process tree, using the `DYLD_INSERT_LIBRARIES` environment variable. [System Integrity Protection (SIP)](https://support.apple.com/en-us/102149) strips all `DYLD_*` environment variables whenever a protected Apple system binary is executed, e.g., `/bin/sh`, `/usr/bin/env`, the system `/usr/bin/python3`, or the built-in coreutils. Once such a binary is executed, the profiling library is removed from that process and every process it spawns. **The entire subtree becomes invisible to the profiler.** ## What this looks like In the following execution tree, 🔍 marks a profiled process and 🔒 marks a SIP-protected system binary. Processes without an icon are not profiled: ```text title="Execution tree" theme={null} codspeed run ├─ 🔍 /opt/homebrew/bin/python3 bench.py profiled └─ 🔒 /bin/bash -c "python3 bench.py" SIP strips DYLD_* └─ python3 bench.py not profiled ``` JavaScript tooling is particularly affected: package manager and `node_modules/.bin` shims go through `/usr/bin/env` and `/bin/sh`, so the profiling variable is stripped several times along the chain: ```text title="Execution tree (pnpm run bench)" theme={null} codspeed run └─ 🔒 /usr/bin/env -> node └─ 🔍 node (pnpm CLI) └─ 🔒 /bin/sh -> /bin/bash └─ 🔍 node (vitest runner) └─ 🔍 node × 5 workers ``` CodSpeed re-sets `DYLD_INSERT_LIBRARIES` when launching benchmark runtimes such as Node.js and Go, re-enabling profiling for the rest of the process tree. Only the SIP-protected system processes themselves are never profiled. ## When to ignore the warning Check which binaries the warning mentions: * If they are launchers like `/bin/sh` or `/usr/bin/env` and your flamegraph contains your benchmark code, ignore the warning. The launcher only accounts for a negligible amount of time. * If the work you want to profile happens inside a system binary itself, e.g., you benchmark the system `/usr/bin/python3`, use one of the workarounds below. ## Workarounds ### Use a non-system toolchain The recommended fix is to make sure no SIP-protected binary appears in the benchmark's process tree. This requires no system changes and is the only workaround that also applies in CI. SIP only protects the binaries that ship with macOS, under `/usr/bin`, `/bin`, `/usr/sbin`, and `/sbin`. Toolchains installed by a package manager, e.g., Homebrew, `uv`, `pyenv`, `nvm`, or `rustup`, live outside these paths and are profiled normally: 1. Install the toolchain with a package manager instead of using the system one under `/usr/bin`. 2. Invoke it directly, so the kernel never executes a protected binary, e.g., `uv run python3 bench.py` instead of `python3 bench.py`. ### Disable SIP For local profiling on a machine you control, you can disable the part of SIP that strips `DYLD_*` variables. With it disabled, system binaries are profiled like any other process. Follow [Apple's instructions for disabling System Integrity Protection](https://developer.apple.com/documentation/security/disabling-and-enabling-system-integrity-protection): boot into Recovery Mode, run one of the following commands in the Terminal, and reboot: ```sh theme={null} csrutil enable --without debug # disable only the debugging restrictions csrutil disable # disable SIP entirely ``` Prefer the first command: the debugging restrictions are the part of SIP that strips `DYLD_*` variables, so disabling only them keeps the filesystem and kernel extension protections in place. To re-enable SIP when you are done, boot into Recovery Mode again and run: ```sh theme={null} csrutil enable ``` Verify the current status at any time with: `csrutil status`. ## Profiling in CI SIP cannot be disabled on CI-hosted macOS runners, e.g., GitHub-hosted macOS runners. The only available workaround is to [use a non-system toolchain](#use-a-non-system-toolchain): point the benchmark command at the exact non-system toolchain binary and avoid `env` or shell shims where possible. System processes will not appear in CI profiles, and that is expected. It does not affect the correctness of the benchmark measurements for your own code. ## Next Steps Learn how the walltime instrument measures real-world execution time Learn how to read flamegraphs and use profiling data to optimize your code # Running Benchmarks in Buildkite Source: https://codspeed.io/docs/integrations/ci/buildkite Learn how to setup CodSpeed and run benchmarks within your Buildkite CI workflow Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance. Read our detailed post on [how CI noise affects benchmark consistency](https://codspeed.io/blog/benchmarks-in-ci-without-noise). [CodSpeed instruments](/docs/instruments) are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in Buildkite is to use the [CodSpeed Runner](https://github.com/CodSpeedHQ/codspeed) directly. ## Setup For now, only the following OS and versions are supported on the agents: * Ubuntu 22.04 and later * Debian 12 and later ### 1. Setup the `CODSPEED_TOKEN` secret First, you need to get your CodSpeed Token. There are multiple ways to retrieve it: * Once you enable a repository on CodSpeed, you'll be prompted to copy the token * You can also find it on the repository settings page Upload Token from the settings page

**Token Scope**: Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token. Then, create a new [pipeline secret with your preferred method](https://buildkite.com/docs/pipelines/secrets) with the name `CODSPEED_TOKEN` and the value of your token. ### 2. Install the `codspeed-runner` CLI in your agent Install the CLI on your agent by running the following commands: ```bash theme={null} CODSPEED_RUNNER_VERSION= # refer to https://github.com/CodSpeedHQ/codspeed/releases for available versions curl -fsSL https://codspeed.io/$CODSPEED_RUNNER_VERSION/install.sh | bash ``` Refer to the [releases page](https://github.com/CodSpeedHQ/codspeed/releases) to see all available versions. ### 3. Create the benchmarks pipeline In a new or existing pipeline, add a step that will run your benchmarks with `codspeed-runner`: ```yml .buildkite/pipeline.yml theme={null} steps: - label: "Run benchmarks with CodSpeed" command: | ... # your build commands codspeed-runner --token=$CODSPEED_TOKEN -- ``` ### 3. Check the results Once the workflow is created, your pull requests will receive a performance report comment and will also receive some additional checks: Pull Request Result

## Next Steps Understand the performance metrics generated by CodSpeed Make sure you or team members never merge unexpected performance regressions Get detailed flame graphs and performance traces for your benchmarks Run your benchmarks in parallel to speed up your CI # Configuring CircleCI for CodSpeed Source: https://codspeed.io/docs/integrations/ci/circleci/configuration Learn how to configure CircleCI to run benchmarks with CodSpeed. ## Authentication In order to upload benchmark results to CodSpeed, the CircleCI job needs to authenticate with CodSpeed. There are two supported methods for authentication: OpenID Connect (OIDC) and static CodSpeed tokens. ### OIDC (Recommended) CodSpeed recommends using [OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/) for authentication. Using this method, a token is generated on-the-fly during the workflow run. This token is then used to authenticate securely with CodSpeed without needing to store long-lived credentials, but grants no additional permissions to the workflow. On CircleCI, this works by default: a job that sets no `CODSPEED_TOKEN` authenticates with an OIDC token, and there is nothing to configure for it. If a job cannot use OIDC, authenticate it with a [CodSpeed token](#codspeed-token). ### CodSpeed token Some jobs cannot use OIDC and need a static CodSpeed token instead: * Pull requests opened from a fork, whose token names the fork rather than your repository. * Pipelines triggered by a custom webhook, whose token names no repository at all. * Jobs running on an image that does not ship the `circleci` CLI, which the CodSpeed CLI needs to request the token. Retrieve your CodSpeed token from your repository settings on CodSpeed: Upload Token from the settings page

Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token. Then add it as a [project environment variable](https://circleci.com/docs/set-environment-variable/#set-an-environment-variable-in-a-project) or in a [context](https://circleci.com/docs/contexts/), with the name `CODSPEED_TOKEN`. The CodSpeed CLI reads it from the job environment. ## Project settings ### Pipeline trigger CodSpeed recommends the **PR opened or pushed to, default branch and tag pushes** event for the benchmarks pipeline: it builds the default branch, recording the baselines, and builds pull requests, which is what CodSpeed reports on. See [the setup guide](/docs/integrations/ci/circleci#2-create-the-benchmarks-pipeline) for how to set it. ### Running the benchmarks in an existing pipeline You can add the benchmarks to a pipeline you already have. Its trigger covers every job in it, so restricting the benchmarks to pull requests restricts the rest too. CircleCI does not rebuild a branch it has already built, so a branch built before you changed the trigger receives no performance report until you push a new commit to it. ## Legacy GitHub OAuth projects CircleCI has [two types of GitHub integration](https://circleci.com/docs/guides/integration/using-the-circleci-github-app-in-an-oauth-org/#two-types-of-github-integration): the GitHub App, which the [setup guide](/docs/integrations/ci/circleci) follows, and the GitHub OAuth app it replaces. CodSpeed supports both. An OAuth project holds a single pipeline, and its configuration has to live in `.circleci/config.yml`. The benchmarks job goes in that file, alongside your other jobs, rather than in a dedicated one: ```yaml .circleci/config.yml theme={null} version: 2.1 jobs: tests: docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: pytest tests/ benchmarks: docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- pytest tests/ --codspeed workflows: ci: jobs: - tests - benchmarks ``` The job needs no upload token: with `CODSPEED_TOKEN` unset, it authenticates with an [OIDC token](#oidc-recommended). Its [trigger](#pipeline-trigger) is the one that pipeline already has, so [the same tradeoff applies](#running-the-benchmarks-in-an-existing-pipeline). If you use an OAuth project with a trigger that builds **every push** and the pull request is opened after the branch has already been built, the CodSpeed report may not appear on the pull request. In that case, push a new commit to the branch to get the report. To bypass these restrictions, CodSpeed recommends installing the CircleCI GitHub App, which can be installed alongside an existing GitHub OAuth pipeline, and giving the benchmarks a pipeline of their own as the [setup guide](/docs/integrations/ci/circleci) does. The CircleCI Pipelines page listing a GitHub App pipeline built from .circleci/codspeed.yml and a GitHub OAuth pipeline built from .circleci/config.yml

The CircleCI Pipelines page listing a GitHub App pipeline built from .circleci/codspeed.yml and a GitHub OAuth pipeline built from .circleci/config.yml

## Advanced ### CLI version The examples pin the CLI version in the install URL, which is what CodSpeed recommends: the tools an instrument needs are pinned to it, so a version that moves under you can [shift the measurements](/docs/instruments/cpu/regression-causes). All versions are listed on the [releases page](https://github.com/CodSpeedHQ/codspeed/releases). To install the latest version on every job instead, drop the version from the URL: ```yaml .circleci/codspeed.yml theme={null} steps: - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" ``` The second command puts `codspeed` on the `PATH` of the steps that follow, since every `run` step starts a fresh shell. Refer to the CircleCI documentation on [setting an environment variable in a shell command](https://circleci.com/docs/set-environment-variable/#set-an-environment-variable-in-a-shell-command) for more details. ### Running benchmarks in parallel CI jobs Splitting a benchmark suite across several jobs cuts the time a run takes. On CircleCI, there are two ways to do it: a matrix of jobs, or the `parallelism` key. CodSpeed only supports emitting results from your benchmarks if you split them within a single CI workflow. If you run benchmarks in multiple CI workflows, CodSpeed will not be able to aggregate the results correctly, and you may see incomplete or missing data in your CodSpeed reports. A CircleCI pipeline can hold several workflows, so keep every benchmark job in the same one. #### Matrix jobs A matrix declares one job per shard. Reach for it when the shards differ by more than a number, for example when each one runs a different benchmark command. For example with `pytest`. The `--test-group` options come from [`pytest-test-groups`](/docs/benchmarks/python#running-benchmarks-in-parallel-ci-jobs), which the benchmarks job needs installed alongside your other dependencies: ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: parameters: shard: type: integer docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- \ pytest tests/ --codspeed --test-group=<< parameters.shard >> --test-group-count=2 workflows: benchmarks: jobs: - benchmarks: matrix: parameters: shard: [1, 2] ``` CodSpeed aggregates the results of every job of the workflow into a single report: A CircleCI pipeline run with a benchmarks workflow holding the benchmarks-1 and benchmarks-2 jobs

A CircleCI pipeline run with a benchmarks workflow holding the benchmarks-1 and benchmarks-2 jobs

#### The `parallelism` key `parallelism` runs a job on several identical containers. They all run the same steps, so the split comes from `CIRCLE_NODE_INDEX`, numbered from `0`, and `CIRCLE_NODE_TOTAL`. The example shifts the index by one, since `pytest-test-groups` numbers its groups from `1`: ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: parallelism: 2 docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- \ pytest tests/ --codspeed \ --test-group=$((CIRCLE_NODE_INDEX + 1)) \ --test-group-count=$CIRCLE_NODE_TOTAL workflows: benchmarks: jobs: - benchmarks ``` CodSpeed records each container as its own part of the run. A job that does not split its benchmarks runs the whole benchmark suite on every container. CodSpeed does not support the same benchmark running several times in a run, and the pull request receives this comment instead of a performance report: Multiple Benchmark Variations Error Message

CircleCI can also split a list of file names across the containers itself, which fits benchmark commands that take files as arguments. Refer to the CircleCI documentation on [test splitting and parallelism](https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/). Learn more about [benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks). ### Benchmarks in several languages Benchmarks written in several languages run in one job per language, in the same workflow. CodSpeed aggregates their results into a single performance report. For example, with Python and Rust benchmarks: ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: python-benchmarks: docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- pytest tests/ --codspeed rust-benchmarks: docker: - image: cimg/rust:1.82 steps: - checkout - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: cargo install cargo-codspeed --locked # Build the benchmark target(s) - run: cargo codspeed build - run: name: Run the benchmarks command: | codspeed run --mode simulation -- cargo codspeed run workflows: benchmarks: jobs: - python-benchmarks - rust-benchmarks ``` ### Caching the installed instruments The CodSpeed CLI installs the tools its instruments need, such as valgrind for CPU simulation, on every job. Install them in a step of their own with `codspeed setup`, pointed at a directory you cache, and later jobs restore them instead of installing them again: ```yaml .circleci/codspeed.yml theme={null} steps: - restore_cache: keys: - v1-codspeed-instruments-{{ arch }} - run: name: Install the CodSpeed instruments command: codspeed setup --mode simulation --setup-cache-dir ~/.cache/codspeed - save_cache: key: v1-codspeed-instruments-{{ arch }} paths: - ~/.cache/codspeed - run: name: Run the benchmarks command: | codspeed run --mode simulation -- pytest tests/ --codspeed ``` Only the setup step takes `--setup-cache-dir`, since the instruments it installs are already in place when the benchmarks run. Caching right after it, rather than after the benchmarks, means a benchmark failure does not cost the next run the install. The tools are pinned to the CLI version, so include that version in the cache key when you install a pinned CLI rather than the latest one. ### Executors Both the `machine` and the `docker` executor are supported: * `machine`, with an Ubuntu 22.04 or later image, for example `ubuntu-2404:current`. * `docker`, with an image based on Ubuntu 22.04 or later, or Debian 12 or later. CircleCI's [`cimg` images](https://circleci.com/developer/images) qualify. Keep a benchmark job's executor stable over time, so that a run and the baseline it is compared against are measured in the same [runtime environment](/docs/instruments/cpu/regression-causes#ci-runner-variability). Two jobs can use different executors, as [benchmarks in several languages](#benchmarks-in-several-languages) do. CircleCI has full support for the [CPU simulation](/docs/instruments/cpu) and [memory](/docs/instruments/memory) instruments. The [walltime](/docs/instruments/walltime) instrument will run, but not produce reliable results without a dedicated machine, and [macro runners](/docs/features/macro-runners) are not available on CircleCI yet. # Running Benchmarks in CircleCI Source: https://codspeed.io/docs/integrations/ci/circleci/index Learn how to setup CodSpeed and run benchmarks within your CircleCI pipeline Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance. Read our detailed post on [how CI noise affects benchmark consistency](https://codspeed.io/blog/benchmarks-in-ci-without-noise). [CodSpeed instruments](/docs/instruments) are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in CircleCI is to use the [CodSpeed Runner](https://github.com/CodSpeedHQ/codspeed) directly. ## Prerequisites * A repository hosted on GitHub, enabled on [CodSpeed](https://app.codspeed.io/login). CodSpeed reports the results on the pull requests of your [repository provider](/docs/integrations/providers), which is GitHub for a CircleCI pipeline. * [Benchmarks](/docs/benchmarks/overview) in that repository. * The repository set up as a CircleCI project. Refer to the CircleCI documentation on [creating a project](https://circleci.com/docs/guides/getting-started/create-project/). For now, only the following OS and versions are supported on the runners: * Ubuntu 22.04 and later * Debian 12 and later CodSpeed recommends giving the benchmarks a pipeline of their own, so the pipelines you already have keep running as they do today. This needs the CircleCI GitHub App, which can be installed alongside an existing GitHub OAuth pipeline. On the legacy GitHub OAuth integration, a project holds a single pipeline and its configuration has to live in `.circleci/config.yml`, so the benchmarks job goes in that file instead of a dedicated one. CodSpeed supports it: see [legacy GitHub OAuth projects](/docs/integrations/ci/circleci/configuration#legacy-github-oauth-projects). ## 1. Add the benchmarks job Create a `.circleci/codspeed.yml` file with a job that installs the CodSpeed CLI and runs your benchmarks with it: ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: machine: image: ubuntu-2404:current steps: - checkout # ... # Setup your environment here: # - Configure your Python/Rust/Node.js version # - Install your dependencies # - Build your benchmarks (if using a compiled language) # ... - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- "" workflows: benchmarks: jobs: - benchmarks ``` ### Sample configurations ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- pytest tests/ --codspeed workflows: benchmarks: jobs: - benchmarks ``` More info on [how to setup Python benchmarks](/docs/benchmarks/python). ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: docker: - image: cimg/rust:1.82 steps: - checkout - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: cargo install cargo-codspeed --locked # Build the benchmark target(s) - run: cargo codspeed build - run: name: Run the benchmarks command: | codspeed run --mode simulation -- cargo codspeed run workflows: benchmarks: jobs: - benchmarks ``` More info on [how to setup Rust benchmarks](/docs/benchmarks/rust). ```yaml .circleci/codspeed.yml theme={null} version: 2.1 jobs: benchmarks: docker: - image: cimg/node:22.11 steps: - checkout - run: npm install - run: name: Install the CodSpeed CLI command: | curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV" - run: name: Run the benchmarks command: | codspeed run --mode simulation -- node -r esbuild-register benches/bench.ts workflows: benchmarks: jobs: - benchmarks ``` More info on [how to setup Node.js benchmarks](/docs/benchmarks/nodejs). Commit the file on a branch. Nothing builds it yet: the pipeline that will run it is created next. ## 2. Create the benchmarks pipeline In **Project Settings** → **Project Setup**, add a pipeline: * **Config source** and **Checkout source**: your repository. * **Config filepath**: `.circleci/codspeed.yml`, the file from the previous step. The CircleCI add pipeline form, with the config filepath set to .circleci/codspeed.yml

The CircleCI add pipeline form, with the config filepath set to .circleci/codspeed.yml

Then add a GitHub App trigger to that pipeline, on the **PR opened or pushed to, default branch and tag pushes** event: The CircleCI trigger event menu, with PR opened or pushed to, default branch and tag pushes selected

The CircleCI trigger event menu, with PR opened or pushed to, default branch and tag pushes selected

The pipeline now builds: * Every push to the default branch. These runs record the baseline that pull requests are compared against. * Pull requests, which is what CodSpeed reports on. The CircleCI Pipelines page listing the benchmarks pipeline, built from .circleci/codspeed.yml and triggered on PR opened or pushed to, default branch and tag pushes

The CircleCI Pipelines page listing the benchmarks pipeline, built from .circleci/codspeed.yml and triggered on PR opened or pushed to, default branch and tag pushes

Pipelines and their triggers are configured in the CircleCI web app, not in a config file. Refer to the CircleCI documentation on [pipelines](https://circleci.com/docs/guides/orchestrate/pipelines/) and [GitHub trigger event options](https://circleci.com/docs/guides/orchestrate/github-trigger-event-options/) for the other events a trigger supports. ## 3. Open a pull request Open a pull request from the branch carrying `.circleci/codspeed.yml`. The benchmarks pipeline runs, and the pull request receives a performance report comment and a status check: Pull Request Result

Merge it to record the first baseline on your default branch, and the next pull requests are compared against it. If you opened the pull request before creating the pipeline, push a new commit to the branch. CircleCI does not rebuild a branch it has already built. ## 4. Next Steps Now that everything is up and running (and hopefully green 🎉), you can start enhancing your pipeline to get the most out of CodSpeed. Learn how to configure authentication methods and advanced options for CircleCI Understand the performance metrics generated by CodSpeed Make sure you or team members never merge unexpected performance regressions Get detailed flame graphs and performance traces for your benchmarks Run your benchmarks in parallel to speed up your CI # Configuring GitHub Actions for CodSpeed Source: https://codspeed.io/docs/integrations/ci/github-actions/configuration Learn how to configure GitHub Actions to run benchmarks with CodSpeed. ## Authentication CodSpeed needs to validate the authenticity of the benchmark results being uploaded. In order to upload benchmark results to CodSpeed, the GitHub Actions workflow needs to authenticate with CodSpeed. There are three supported methods for authentication: * OIDC: recommended way for both private and public repositories * Static CodSpeed tokens (legacy) * Tokenless uploads (public repositories only) ### OIDC (Recommended) CodSpeed recommends using [OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/) for authentication. Using this method, a token is generated on-the-fly during the workflow run. This token is then used to authenticate securely with CodSpeed without needing to store long-lived credentials, but grants no additional permissions to the workflow. To enable OIDC, scope the `permissions` block to the benchmark job: ```yaml icon="github" .github/workflows/codspeed.yml highlight={10-12} theme={null} name: Benchmarks on: push: { branches: [main] } pull_request: jobs: benchmarks: runs-on: ubuntu-latest permissions: contents: read # required for actions/checkout id-token: write # required for OIDC authentication with CodSpeed steps: - uses: actions/checkout@v4 - uses: CodSpeedHQ/action@v5 with: run: npm run bench ``` Scoping `id-token: write` to the benchmark job, rather than the whole workflow, follows GitHub's least-privilege guidance for OIDC. Since GitHub's OIDC permission is not restricted by audience, keeping it on the benchmark job avoids granting unrelated jobs the ability to request tokens for other services. See GitHub's [OIDC with reusable workflows](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/using-openid-connect-with-reusable-workflows) documentation for further reading. For more information, see the GitHub Actions OIDC documentation: * [Overview of OIDC on GitHub](https://docs.github.com/en/actions/concepts/security/openid-connect) * [How to configure OpenID Connect in GitHub Actions](https://docs.github.com/en/actions/reference/security/oidc#workflow-permissions-for-the-requesting-the-oidc-token) ### CodSpeed token (Legacy) While we recommend using OpenID Connect (OIDC) for improved security, you can use a static CodSpeed token for authentication. Retrieve your CodSpeed token from your repository settings page: Upload Token from the settings page

**Token Scope**: Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token. Then, create a new [encrypted secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) in your repository with the name `CODSPEED_TOKEN` and the value of your token. Then pass the token explicitly to the action using the `with` key: ```yaml highlight={5} theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 with: mode: simulation token: ${{ secrets.CODSPEED_TOKEN }} ``` ### Tokenless CodSpeed allows tokenless uploads for public repositories, allowing runs to be triggered from public forks directly. However, we still recommend using OIDC for improved security. To use tokenless uploads on public repositories, you can simply omit the `token` input and the whole `permissions` section when [creating the benchmarks workflow](#1-create-the-benchmarks-workflow). ## Advanced ### Defining environment variables You can define environment variables for your benchmarks by using the `env` key in the section of the action: ```yaml theme={null} - name: Run benchmarks uses: CodSpeedHQ/action@v5 env: MY_ENV_VAR: "my-value" with: mode: simulation ``` ### Running benchmarks in parallel CI jobs With Github Actions, you can leverage matrix jobs to improve the performance of your benchmarks. For example, using `pytest`. The `--test-group` options come from [`pytest-test-groups`](/docs/benchmarks/python#running-benchmarks-in-parallel-ci-jobs), which the benchmarks job needs installed alongside your other dependencies: CodSpeed only supports emitting results from your benchmarks if you split them within a single CI workflow. If you run benchmarks in multiple CI workflows, CodSpeed will not be able to aggregate the results correctly, and you may see incomplete or missing data in your CodSpeed reports. CodSpeed does not yet support running the same benchmarks multiple times. If your matrix has several dimensions (e.g. a runtime version), please ensure that each benchmark will only run once. If the same benchmark is run multiple times, you will receive the following comment on your pull request: Multiple Benchmark Variations Error Message

Learn more about [benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks). ### Running multiple instruments serially If you want to measure multiple aspects of performance, you can run multiple instruments in a single CI step by passing a comma-separated list to the `mode` input.\ Please note that for compiled languages, you will need to have built your benchmarks with support for all the instruments you are trying to run beforehand. For example, to run both the `simulation` and `memory` instruments: Walltime needs a dedicated runner (`codspeed-macro`) for reliable results. Do not run walltime serially on default runners as they are shared with other users and introduce lots of noise. See the [macro runner documentation](/docs/features/macro-runners) for more information. Support for running multiple instruments serially was added in CodSpeed v4.12.0. ### Using container images When running benchmarks in a container (such as `ubuntu:latest` or other base images), git is typically not installed by default. Without git in the path, `actions/checkout` falls back to downloading a tarball of your code, which does not setup a local git repository. CodSpeed relies on git to determine the current commit hash and other metadata, so it's important to have git installed and the repository checked out. ```yaml theme={null} jobs: benchmarks-in-container: runs-on: ubuntu-latest container: image: ubuntu:latest steps: - name: Install git before fetching repository run: | apt-get update apt-get install -y git git config --global --add safe.directory $GITHUB_WORKSPACE - name: Checkout code uses: actions/checkout@v4 ``` The `git config --global --add safe.directory` command is required to prevent git from triggering file system permission errors when accessing the repository. More details can be found in this [GitHub Issue](https://github.com/actions/checkout/issues/766) ### Compatibility For now, only the following OS and versions are supported on the runners: * Ubuntu 22.04 and later * Debian 12 and later # Running Benchmarks in GitHub Actions Source: https://codspeed.io/docs/integrations/ci/github-actions/index Learn how to setup CodSpeed and run benchmarks within your GitHub Actions CI workflow Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard GitHub-hosted runners can exhibit significant performance variance. Read our detailed post on [how CI noise affects benchmark consistency](https://codspeed.io/blog/benchmarks-in-ci-without-noise). [CodSpeed instruments](/docs/instruments) are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitHub Actions is to use the [CodSpeed GitHub Action](https://github.com/CodSpeedHQ/action). ## 1. Create the benchmarks workflow Create a new workflow to run the benchmarks for your repository. You can do this by creating the `codspeed.yml` file in the `.github/workflows` directory with the following content: The most important step of this workflow is the usage of [`CodSpeedHQ/action`](https://github.com/CodSpeedHQ/action). This action will configure the CodSpeed environment and upload the benchmarks results. Keep both triggers in the workflow's `on` section: * `push` on your default branch (e.g., `main`). These runs record the baseline that every comparison is measured against. * `pull_request`. This is what reports the performance impact of a pull request, measured against that baseline. If CodSpeed runs only on `pull_request` (or a manual `workflow_dispatch`), there are no runs on the default branch, so pull requests have no baseline to compare against and the default branch shows no results in CodSpeed. Learn more about [baseline report selection](/docs/features/understanding-the-metrics#baseline-report-selection). **Public repositories** CodSpeed allows tokenless uploads for public repositories, allowing runs to be triggered from public forks directly. To enable this, you can simply omit the `permissions` section. Learn more about [authentication methods](/docs/integrations/ci/github-actions/configuration#authentication). ### Sample configurations * [Python (with `pytest-codspeed`)](https://github.com/CodSpeedHQ/action/blob/main/examples/python-pytest-codspeed.yml) * [Rust (with `cargo-codspeed`)](https://github.com/CodSpeedHQ/action/blob/main/examples/rust-cargo-codspeed.yml) * [Node.js (with `codspeed-node` and TypeScript)](https://github.com/CodSpeedHQ/action/blob/main/examples/nodejs-typescript-codspeed.yml) ## 2. Check the results Once the workflow is created, your pull requests will receive a performance report comment and will also receive some additional checks: Pull Request Result

Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token. Then, create a new [CI/CD variable](https://docs.gitlab.com/ee/ci/variables/#for-a-project) in your repository with the name `CODSPEED_TOKEN` and the value of your token. Set the `CODSPEED_TOKEN` variable as `"Masked"`, but not `"Protected"`, because it must be available on all branches. Upload Token from the settings page

## Advanced ### Running benchmarks in parallel CI jobs With GitlabCI, you can split your benchmarks across multiple jobs. For example with `pytest`. The `--test-group` options come from [`pytest-test-groups`](/docs/benchmarks/python#running-benchmarks-in-parallel-ci-jobs), which the benchmarks jobs need installed alongside your other dependencies: ```yaml .gitlab-ci.yml theme={null} workflow: # run on merge requests and pushes on the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH codspeed-1: stage: test image: python:3.12 id_tokens: CODSPEED_TOKEN: aud: codspeed.io before_script: - pip install -r requirements.txt - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash - source $HOME/.cargo/env script: - codspeed run --mode instrumentation -- pytest tests/ --codspeed --test-group=1 --test-group-count=2 codspeed-2: stage: test image: python:3.12 id_tokens: CODSPEED_TOKEN: aud: codspeed.io before_script: - pip install -r requirements.txt - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash - source $HOME/.cargo/env script: - codspeed run --mode instrumentation -- pytest tests/ --codspeed --test-group=2 --test-group-count=2 ``` CodSpeed only supports emitting results from your benchmarks if you split them within a single CI workflow. If you run benchmarks in multiple CI workflows, CodSpeed will not be able to aggregate the results correctly, and you may see incomplete or missing data in your CodSpeed reports. Learn more about [benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks). # Running Benchmarks in GitLab CI Source: https://codspeed.io/docs/integrations/ci/gitlab-ci/index Learn how to setup CodSpeed and run benchmarks within your GitLab CI workflow Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance. Read our detailed post on [how CI noise affects benchmark consistency](https://codspeed.io/blog/benchmarks-in-ci-without-noise). [CodSpeed instruments](/docs/instruments) are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitLab CI is to use the [CodSpeed Runner](https://github.com/CodSpeedHQ/codspeed) directly. For now, only the following OS and versions are supported on the runners: * Ubuntu 22.04 and later * Debian 12 and later ## 1. Create the benchmarks job Create a new job to run the benchmarks for your repository. For example, you can add this job to your existing pipeline by adding a section in your `.gitlab-ci.yml` file with the following content: ```yaml .gitlab-ci.yml {15-25} theme={null} workflow: # run on merge requests and pushes on the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH codspeed: stage: test image: # for example: python:3.12 or rust:1.82 id_tokens: # OpenID Connect token to authenticate with CodSpeed CODSPEED_TOKEN: aud: codspeed.io before_script: # ... # Setup your environment here: # - Configure your Python/Rust/Node version # - Install your dependencies # - Build your benchmarks (if using a compiled language) # ... # Pin the CodSpeed CLI version. Refer to https://github.com/CodSpeedHQ/codspeed/releases for available versions - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash # Or always use the latest version: # - curl -fsSL https://codspeed.io/install.sh | bash - source $HOME/.cargo/env script: - codspeed run --mode simulation -- "" ``` Make sure to run the job on `merge_request_event`. This is required to have reports on merge requests correctly working. Learn more about [baseline report selection](/docs/features/understanding-the-metrics#baseline-report-selection). ### Sample configurations ```yaml .gitlab-ci.yml theme={null} workflow: # run on merge requests and pushes on the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH codspeed: stage: test image: python:3.12 id_tokens: # OpenID Connect token to authenticate with CodSpeed CODSPEED_TOKEN: aud: codspeed.io before_script: - pip install -r requirements.txt - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash - source $HOME/.cargo/env script: - codspeed run --mode simulation -- pytest tests/ --codspeed ``` More info on [how to setup Python benchmarks](/docs/benchmarks/python). ```yaml .gitlab-ci.yml theme={null} workflow: # run on merge requests and pushes on the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH codspeed: stage: test image: rust:1.82 id_tokens: # OpenID Connect token to authenticate with CodSpeed CODSPEED_TOKEN: aud: codspeed.io before_script: - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash - source $HOME/.cargo/env - cargo install cargo-codspeed --locked # Build the benchmark target(s) - cargo codspeed build script: # Run the benchmarks - codspeed run --mode simulation -- cargo codspeed run ``` More info on [how to setup Rust benchmarks](/docs/benchmarks/rust). ```yaml .gitlab-ci.yml theme={null} workflow: # run on merge requests and pushes on the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH codspeed: stage: test image: nodejs:22 id_tokens: # OpenID Connect token to authenticate with CodSpeed CODSPEED_TOKEN: aud: codspeed.io before_script: - npm install - curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash - source $HOME/.cargo/env script: - codspeed run --mode simulation -- node -r esbuild-register benches/bench.ts ``` More info on [how to setup Node.js benchmarks](/docs/benchmarks/nodejs). CodSpeed only currently supports docker images that are based on Ubuntu or Debian. ## 2. Check the results Once the workflow is created, your merge requests will receive a performance report comment and will also receive some additional checks: Merge Request Setup Comment

## 3. Next Steps Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed. Learn how to configure authentication methods and advanced options for GitLab CI Understand the performance metrics generated by CodSpeed Make sure you or team members never merge unexpected performance regressions Get detailed flame graphs and performance traces for your benchmarks Run your benchmarks in parallel to speed up your CI # Running Benchmarks in CI Source: https://codspeed.io/docs/integrations/ci/index Running performance tests in your CI/CD pipeline ensures that performance regressions are caught early in the development process, before they reach production. CodSpeed integrates seamlessly with your existing CI workflows to provide automated performance monitoring. ## Supported CI Providers } /> } /> } /> } /> If your provider is not listed here, please [open an issue](https://github.com/CodSpeedHQ/codspeed) or contact us on [Discord](https://discord.com/invite/MxpaCfKSqF). ## Benefits of CI Performance Testing ### Catch Regressions Early Performance issues are detected immediately when code is committed, not weeks later in production. Integrating performance tests into your CI pipeline creates an automated safety net that catches performance regressions as soon as they're introduced. Instead of discovering slowdowns after deployment, your team gets immediate feedback during code review. ### Maintain Performance Standards Set performance thresholds to automatically fail builds that introduce significant regressions. CI-based performance testing enforces consistent performance standards across your codebase. You can configure [performance checks](/docs/features/performance-checks/) to automatically block merges when benchmarks exceed acceptable regression thresholds, ensuring your application maintains its performance characteristics over time. ### Streamlined Development Workflow Performance testing in CI eliminates the friction of manual performance validation: * **Automated execution**: Benchmarks run automatically on every pull request * **Consistent environment**: Tests execute in standardized CI environments, reducing variability * **Zero maintenance overhead**: No need to remember to run performance tests manually * **Team visibility**: Performance results are shared directly in pull request comments ### Historical Performance Tracking CodSpeed maintains a complete history of your application's performance evolution across all commits and branches. Your CI pipeline becomes a continuous performance monitoring system, building a comprehensive history of how your application's performance changes over time. This historical data helps identify performance trends and understand the impact of specific changes. ### Data-Driven Optimization CI performance testing provides the metrics needed to make informed optimization decisions: * Compare performance across different implementations * Validate that optimizations actually improve performance * Identify which changes have the most significant performance impact * Track performance improvements over multiple iterations # Integrating with GitHub Source: https://codspeed.io/docs/integrations/providers/github Learn how to integrate CodSpeed with a GitHub repository. ## Setup 1. Go to your [CodSpeed settings](https://app.codspeed.io/settings) If you are importing a repository from an organization, you need to be either a **Provider Admin** or a **Admin** of this organization. Check out the [CodSpeed Roles & Permissions page](/docs/features/roles-and-permissions) for more information. 2. Click the "Import" button: Adding a new repository from the settings page

3. Select the organization or the user you want to import the repository from: Github App organization selection

4. Select the repositories you want to benchmark and click on "Install" 5. Your repositories should appear in the list of repositories. Repositories list after installation

You can now continue and [setup the continuous reporting](/docs/integrations/ci). ## GitHub Permissions CodSpeed requires the following permissions to work properly with GitHub: ### Account permissions * **Email addresses (Read-only)**: required to notify users about installation and configuration changes. **Act on your behalf** When you connect CodSpeed to your GitHub account, you might see a message saying the app wants to “Act on your behalf.” This message is a display inconsistency on GitHub's side and doesn't reflect what CodSpeed can actually do. GitHub Authorization Request

The only permission CodSpeed requires from your personal GitHub account is read access to your email address, which allows us to identify your account and contact you if needed. All interactions will otherwise be performed through the CodSpeed GitHub App, which is installed on repositories you authorize but anyway, CodSpeed can **never** perform actions on your behalf. For more technical background, see [this GitHub discussion](https://github.com/orgs/community/discussions/37117). ### Repository Permissions * **Actions (Read/Write)**: required to access the execution logs of the workflow runs and to annotate/cancel CodSpeed workflows. * **Checks (Read/Write)**: required to create check runs for the [performance reports](/docs/integrations/providers#performance-reports-in-pull-requests). * **Commit statuses (Read-only)**: required to access commit status checks. * **Contents (Read/Write)**: * Read access: required to access branches and commits. * Write access: required to recommend performance improvements by pushing commits to a dedicated branch. This permission does not bypass any branch protection rules you have set up on your repository * **Pull requests (Read/Write)**: required to access pull requests and annotate them with [performance reports](/docs/integrations/providers#performance-reports-in-pull-requests) comments. * **Issues (Read/Write)**: required to access issues and respond to pull request comments mentioning [@codspeedbot](/docs/ai/wizard). * **Metadata (Read-only)**: mandatory for all GitHub Apps. * **Workflows (Write)**: required to [automate CodSpeed setup](/docs/ai/wizard) on repositories. ### Organization permissions * **Members (Read-only)**: required to access the list of members of the organization. * **Self-hosted runners (Read/Write)**: required to register [CodSpeed Macro runners](/docs/instruments/walltime) at the organization level. If you have any concerns about the permissions we require, please [contact us](mailto:support@codspeed.io). # Integrating with GitLab Source: https://codspeed.io/docs/integrations/providers/gitlab Learn how to integrate CodSpeed with GitLab. ## Setup 1. Go to your [CodSpeed settings](https://app.codspeed.io/settings) **Required permissions**: If you are importing a repository from an organization, you need to be either a **Provider Admin** or a **Admin** of this organization. Check out the [CodSpeed Roles & Permissions page](/docs/features/roles-and-permissions) for more information. 2. Generate a Personal Access Token on GitLab. CodSpeed accepts fine-grained and legacy tokens. **Bot Account**: CodSpeed uses a single user to access GitLab resources on behalf of it. You can generate one either for an existing user, or for a new dedicated bot account (recommended). This user/account will be the one to publish comments on merge requests. **Permissions**: a token keeps the permissions it was generated with, and GitLab offers no way to change them afterwards. Grant everything CodSpeed needs before generating the token. To add a permission later, generate a new token and save it in your [CodSpeed settings](https://app.codspeed.io/settings). Open the [fine-grained token form](https://gitlab.com/-/user_settings/personal_access_tokens/granular/new) and fill it in: * **Expiration date**: set it to one year. It defaults to one month, and your CodSpeed integration stops working once the token expires. * **Group and project access**: select **All groups and projects that I'm a member of**. * **Add resource permissions**: grant the permissions below, switching between the **Group and project** and **User** tabs of the **Resource access** selector. On the **Group and project** tab: * **CI/CD**: Commit Status (Create), Job (Read). * **Project Planning**: Work Item (Create, Delete, Read, Update). * **Projects**: Project (Read). * **Repository**: Branch (Read), Code (Download, Push), Commit (Read), Merge Request (Create, Read, Update), Repository (Read). On the **User** tab: * **Projects**: Project (Read). * **System Access**: Personal Access Token (Read), User (Read). For more details about why CodSpeed needs these permissions, check out the [GitLab permissions](#gitlab-permissions) section. Create a fine-grained Personal Access Token on GitLab

Create a fine-grained Personal Access Token on GitLab

Open the [legacy token form](https://gitlab.com/-/user_settings/personal_access_tokens?name=Codspeed+Token\&scopes=api) and fill it in: * **Expiration date**: set it to one year. It defaults to one month, and your CodSpeed integration stops working once the token expires. * **Select scopes**: check `api`, which covers everything CodSpeed does. Create a legacy Personal Access Token on GitLab

Create a legacy Personal Access Token on GitLab

A group can [enforce fine-grained tokens](https://docs.gitlab.com/auth/tokens/fine_grained_access_tokens/#enforce-fine-grained-personal-access-tokens), after which legacy tokens stop working on all of its projects. 3. Fill the Personal Access Token in [CodSpeed settings](https://app.codspeed.io/settings) GitLab Personal Access Token section

4. Your repositories should appear in the list of repositories. Repositories list after installation

You can now continue and [setup the continuous reporting](/docs/integrations/ci) ## GitLab permissions The `api` scope of a legacy token covers everything CodSpeed does. A fine-grained token grants only the permissions you select, listed here with what CodSpeed uses them for. ### Group and project * **CI/CD** * **Commit Status (Create)**: required to publish the [performance report](/docs/integrations/providers#performance-reports-in-pull-requests) as a commit status. * **Job (Read)**: required to follow the GitLab CI jobs running your benchmarks. * **Project Planning** * **Work Item (Create, Delete, Read, Update)**: required to publish and update the performance report comment on merge requests. * **Projects** * **Project (Read)**: required to read repository settings, such as the default branch and the visibility. * **Repository** * **Branch (Read)**: required to resolve the head commit of a branch. * **Code (Download, Push)**: required to clone a repository and push the branch of an optimization opened by the [CodSpeed wizard](/docs/ai/wizard). * **Commit (Read)**: required to read commits and the statuses published on them. * **Merge Request (Create, Read, Update)**: required to read merge requests, open the ones the wizard suggests, and request reviewers on them. * **Repository (Read)**: required to find the commit two runs have in common, which is the baseline a merge request is compared against. GitLab classifies merge request comments as work item notes, which is why the report comment needs the **Work Item** permissions. They also cover issues and epics in the same boundary. ### User * **Projects** * **Project (Read)**: required to list the repositories you can import. * **System Access** * **Personal Access Token (Read)**: required to read the expiration date and the permissions of the token itself. * **User (Read)**: required to identify the account CodSpeed acts as when it pushes and comments. # Integrating with a Repository Provider Source: https://codspeed.io/docs/integrations/providers/index CodSpeed integrates directly with your repository provider to deliver performance insights where your team already collaborates. By connecting with your Git forge, CodSpeed brings performance data directly into your development workflow through pull request comments and status checks. ## Supported Repository Providers } /> } /> If your provider is not listed here, please [send us an email](mailto:support@codspeed.io) or contact us on [Discord](https://discord.com/invite/MxpaCfKSqF). ## Benefits ### Performance Reports in Pull Requests Performance results appear automatically in pull request comments, making performance data visible to all reviewers. Instead of hunting through CI logs or external dashboards, your team gets performance insights directly in the pull request interface. CodSpeed automatically posts detailed performance reports as comments, showing benchmark results, comparisons with the base branch, and highlighting any significant changes. Pull Request Regression Result

### Automated Status Checks Configure status checks to prevent merging pull requests that introduce performance regressions beyond your tolerance thresholds. Repository provider integration enables CodSpeed to participate in your branch protection rules through status checks. You can configure [performance checks](/docs/features/performance-checks/) to automatically block merges when performance degrades beyond acceptable limits, ensuring performance standards are enforced at the repository level. Pull Request Checks failing because of the performance regression

### Centralized Performance Tracking All performance data is linked to specific commits, branches, and pull requests, creating a comprehensive audit trail. Your repository becomes the single source of truth for both code changes and their performance implications. CodSpeed maintains performance history tied directly to your Git commits, making it easy to understand when and why performance changed. # bencher (libtest) compatibility layer documentation Source: https://codspeed.io/docs/reference/codspeed-rust/bencher A compatibility layer for `bencher`, a port of the unstable libtest benchmarking framework

## Overview `codspeed-bencher-compat` is a Bencher compatibility layer for CodSpeed that allows seamless integration of Rust's built-in benchmarking framework with CodSpeed performance measurement. This crate acts as a drop-in replacement for the standard `bencher` crate, maintaining your existing benchmark code while enabling CodSpeed integration. Bencher is a port of the unstable `libtest` benchmarking framework. It is not recommended for new projects, consider using [Divan](/docs/reference/codspeed-rust/divan) for new projects as it offers better ergonomics and features. ## Installation Install the compatibility layer and rename it to `bencher` to maintain compatibility with existing code: ```bash theme={null} cargo add --dev codspeed-bencher-compat --rename bencher ``` This will install the `codspeed-bencher-compat` crate and rename it to `bencher` in your `Cargo.toml`. This way, you can keep your existing imports and the compatibility layer will take care of the rest. Using the compatibility layer won't change the behavior of your benchmark suite and bencher will still run it as usual. If you prefer, you can also install `codspeed-bencher-compat` as is and change your imports to use this new crate name. ## Usage Let's start with the example from the [Bencher documentation](https://docs.rs/bencher/latest/bencher/), creating a benchmark suite for 2 simple functions (in `benches/example.rs`): ```rust theme={null} use bencher::{benchmark_group, benchmark_main, Bencher}; fn a(bench: &mut Bencher) { bench.iter(|| { (0..1000).fold(0, |x, y| x + y) }) } fn b(bench: &mut Bencher) { const N: usize = 1024; bench.iter(|| { vec![0u8; N] }); bench.bytes = N as u64; } benchmark_group!(benches, a, b); benchmark_main!(benches); ``` The last step in creating the Bencher benchmark is to add the new benchmark target in your `Cargo.toml`: ```toml title="Cargo.toml" theme={null} [[bench]] name = "example" harness = false ``` And that's it! You can now run your benchmark suite with CodSpeed: ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed build Finished release [optimized] target(s) in 0.12s Finished built 1 benchmark suite(s) $ cargo codspeed run Collected 1 benchmark suite(s) to run Running example Using codspeed-bencher-compat v1.0.0 compatibility layer NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/example.rs::a (group: benches) Checked: benches/example.rs::b (group: benches) Done running bencher_example Finished running 1 benchmark suite(s) ``` ## Resources * [Official bencher documentation](https://docs.rs/bencher/latest/bencher/index.html) * [Official bencher GitHub repository](https://github.com/bluss/bencher) # cargo-codspeed documentation Source: https://codspeed.io/docs/reference/codspeed-rust/cargo-codspeed The `cargo` subcommand to benchmark your Rust code with CodSpeed

## Overview `cargo-codspeed` is a Cargo subcommand for running performance benchmarks with CodSpeed. It provides an easy way to build and run benchmarks for Rust projects with CodSpeed integration. ## Installation ### With `cargo` ```bash theme={null} cargo install cargo-codspeed --locked ``` ### With `cargo-binstall` (recommended in CI) [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall) enables you to install binaries directly without having to build from the source(with `cargo install`) every time. If you don't have installed yet, you can install it with: ```bash theme={null} cargo install cargo-binstall ``` You can then install `cargo-codspeed` with: ```bash theme={null} cargo binstall cargo-codspeed ``` ## Usage ### Building Benchmarks Build your benchmarks for CodSpeed: ```bash theme={null} cargo codspeed build ``` ### Running Benchmarks Run the previously built benchmarks: ```bash theme={null} cargo codspeed run ``` ### The `--measurement-mode` flag When building benchmarks, you can specify the measurement mode using the `--measurement-mode` / `-m` flag. This determines which CodSpeed instrument can be used to run your benchmarks. ```bash theme={null} # First build the benchmarks: cargo codspeed build -m simulation # Then run them: cargo codspeed run -m simulation ``` The `--measurement-mode` flag can take the following values: * **`simulation`**: Runs benchmarks once on a [simulated CPU](/docs/instruments/cpu) for consistent measurements. * **`walltime`**: Measures [wall-clock time](/docs/instruments/walltime) for real-world scenarios. * **`memory`**: Benchmarks are run once using [memory profiling](/docs/instruments/memory) to track heap allocations and memory usage. If you omit the `--measurement-mode` flag, the default behavior is to build for simulation mode. # criterion.rs compatibility layer documentation Source: https://codspeed.io/docs/reference/codspeed-rust/criterion A compatibility layer for `criterion.rs`

## Overview `codspeed-criterion-compat` is a Criterion.rs compatibility layer for CodSpeed that allows you to seamlessly integrate existing Criterion benchmarks with CodSpeed performance measurement. This crate acts as a drop-in replacement for Criterion, maintaining your existing benchmark code while enabling CodSpeed integration. ## Installation ```sh theme={null} cargo add --dev codspeed-criterion-compat --rename criterion ``` This will install the `codspeed-criterion-compat` crate and rename it to `criterion` in your `Cargo.toml`. This way, you can keep your existing imports and the compatibility layer will take care of the rest. Using the compatibility layer won't change the behavior of your benchmark suite and criterion will still run it as usual. If you prefer, you can also install `codspeed-criterion-compat` as is and change your imports to use this new crate name. ## Usage Let's start with the example from the [Criterion.rs documentation](https://bheisler.github.io/criterion.rs/book/getting_started.html), creating a benchmark suite for the Fibonacci function (in `benches/my_benchmark.rs`): ```rust theme={null} use criterion::{black_box, criterion_group, criterion_main, Criterion}; fn fibonacci(n: u64) -> u64 { match n { 0 => 1, 1 => 1, n => fibonacci(n-1) + fibonacci(n-2), } } pub fn criterion_benchmark(c: &mut Criterion) { c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20)))); } criterion_group!(benches, criterion_benchmark); criterion_main!(benches); ``` The last step in creating the Criterion benchmark is to add the new benchmark target in your `Cargo.toml`: ```toml title="Cargo.toml" theme={null} [[bench]] name = "my_benchmark" harness = false ``` And that's it! You can now run your benchmark suite with `cargo-codspeed`: ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed build Finished release [optimized] target(s) in 0.12s Finished built 1 benchmark suite(s) $ cargo codspeed run Collected 1 benchmark suite(s) to run Running my_benchmark Using codspeed-criterion-compat v1.0.0 compatibility layer NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/bencher_example.rs::fib_20 (group: benches) Done running bencher_example Finished running 1 benchmark suite(s) ``` ## Compatibility ### Not (yet) supported * `iter_custom` * `with_filter` ## Resources * [Official criterion.rs documentation](https://bheisler.github.io/criterion.rs/book/index.html) * [Official criterion.rs GitHub repository](https://github.com/bheisler/criterion.rs) # divan compatibility layer documentation Source: https://codspeed.io/docs/reference/codspeed-rust/divan A compatibility layer for `divan`

## Overview `codspeed-divan-compat` is a Divan compatibility layer for CodSpeed that allows seamless integration of Divan benchmarks with CodSpeed performance measurement. This crate acts as a drop-in replacement for Divan, maintaining your existing benchmark code while enabling CodSpeed integration. Divan is the **recommended benchmarking framework** for Rust when using CodSpeed due to its modern design and excellent performance characteristics. ## Installation ```sh theme={null} cargo add --dev codspeed-divan-compat --rename divan ``` This will install the `codspeed-divan-compat` crate and rename it to `divan` in your `Cargo.toml`. This way, you can keep your existing imports and the compatibility layer will take care of the rest. Using the compatibility layer won't change the behavior of your benchmark suite and divan will still run it as usual. If you prefer, you can also install `codspeed-divan-compat` as is and change your imports to use this new crate name. ## Usage Let's start with the example from the [divan documentation](https://docs.rs/divan/0.1.17/divan/index.html#getting-started), creating a benchmark suite for the Fibonacci function (in `benches/my_benchmark.rs`): ```rust theme={null} fn main() { // Run registered benchmarks. divan::main(); } // Register a `fibonacci` function and benchmark it over multiple cases. #[divan::bench(args = [1, 2, 4, 8, 16, 32])] fn fibonacci(n: u64) -> u64 { if n <= 1 { 1 } else { fibonacci(n - 2) + fibonacci(n - 1) } } ``` The last step in creating the divan benchmark is to add the new benchmark target in your `Cargo.toml`: ```toml title="Cargo.toml" theme={null} [[bench]] name = "my_benchmark" harness = false ``` And that's it! You can now run your benchmark suite with `cargo-codspeed`: ```shellsession title=terminal icon="square-terminal" theme={null} $ cargo codspeed build Finished release [optimized] target(s) in 0.12s Finished built 1 benchmark suite(s) $ cargo codspeed run Collected 1 benchmark suite(s) to run Running my_benchmark NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment. Checked: benches/my_benchmark.rs::fibonacci[1] Checked: benches/my_benchmark.rs::fibonacci[2] Checked: benches/my_benchmark.rs::fibonacci[4] Checked: benches/my_benchmark.rs::fibonacci[8] Checked: benches/my_benchmark.rs::fibonacci[16] Checked: benches/my_benchmark.rs::fibonacci[32] Done running my_benchmark Finished running 1 benchmark suite(s) ``` ## Compatibility ### Not (yet) supported * [`divan::bench(crate = xxx)`](https://docs.rs/divan/latest/divan/attr.bench.html#crate): due to how the compatibility layer works internally, we do not plan to support this feature. * [`divan::bench_group`](https://docs.rs/divan/latest/divan/attr.bench_group.html): we do not support benchmark grouping yet, if you need it don't hesitate to create an issue. ## Resources * [Official divan documentation](https://docs.rs/divan/latest/divan/index.html) * [Official divan GitHub repository](https://github.com/nvzqz/divan) # codspeed-rust documentation Source: https://codspeed.io/docs/reference/codspeed-rust/index Crates to benchmark your Rust code CodSpeed provides compatibility layers for popular Rust benchmarking frameworks and a dedicated cargo subcommand to help you measure the performance of your Rust code. A cargo subcommand to benchmark your Rust code with CodSpeed integration. ## Compatibility layers The compatibility layer for divan, the recommended benchmarking framework for Rust. The compatibility layer for criterion.rs. The compatibility layer for bencher. # API References Source: https://codspeed.io/docs/reference/index 🚧 We're in the process of integrating all the API references, stay tuned! A pytest plugin for benchmarking Python code. A collection of tools to benchmark your Rust code. # pytest-codspeed documentation Source: https://codspeed.io/docs/reference/pytest-codspeed A `pytest` plugin for benchmarking performance of Python code

## Overview `pytest-codspeed` is a pytest plugin for measuring and tracking performance of Python code. It provides benchmarking capabilities with support for both wall-time and CPU Simulation measurements. ### Installation ### Example Usage ```shellsession title=terminal icon="square-terminal" theme={null} $ pytest tests/ --codspeed ============================= test session starts ==================== platform darwin -- Python 3.13.0, pytest-7.4.4, pluggy-1.5.0 codspeed: 3.0.0 (enabled, mode: walltime, timer_resolution: 41.7ns) rootdir: /home/user/codspeed-test, configfile: pytest.ini plugins: codspeed-3.0.0 collected 1 items tests/test_sum_squares.py . [ 100%] Benchmark Results ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┓ ┃ Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃ ┣━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━╋━━━━━━━━━━━━━╋━━━━━━━━━━╋━━━━━━━━┫ ┃test_sum_squares┃ 1,873ns ┃ 4.8% ┃ 3.00s ┃ 66,930 ┃ ┗━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━┻━━━━━━━━━━━━━┻━━━━━━━━━━┻━━━━━━━━┛ =============================== 1 benchmarked ======================== =============================== 1 passed in 4.12s ==================== ``` ### Command Line Options Enable the CodSpeed benchmarking plugin for the test session. *(This is automatically enabled when running under the CodSpeed runner or from the GitHub action)* The measurement instrument to use for measuring performance. * `auto`: Automatically select the measurement instrument based on the environment. * `simulation`: Use [the CPU simulation instrument](/docs/instruments/cpu). * `walltime`: Use [the wall-clock time instrument](/docs/instruments/walltime). Automatically enabled on macro runners. * `memory`: Use [the memory instrument](/docs/instruments/memory) to track heap allocations. The time to warm up the benchmark for (in seconds), **only for walltime mode**. The maximum time to run a benchmark for (in seconds), **only for walltime mode**. The maximum number of rounds to run a benchmark for, **only for walltime mode**. All of these walltime-specific command line options can be overridden by more specific settings set by the benchmark marker. For example, if you set `warmup_time` in the benchmark marker, it will take precedence over the `--codspeed-warmup-time` command line option. ## Creating Benchmarks There are multiple ways to mark tests as benchmarks at different levels: ### The `pytest.mark.benchmark` marker Marking a test with the `pytest.mark.benchmark` marker will automatically mark it as a benchmark. This means that the entire test function will be measured. For more fine-grained control, see the [benchmark fixture](#with-the-benchmark-fixture) section. ```python highlight={3} title="test_sum_powers.py" theme={null} import pytest @pytest.mark.benchmark def test_sum_squares(): input = [1, 2, 3, 4, 5] output = sum(i**2 for i in input) assert output == 55 ``` You can also mark all the tests contained in a test file as benchmarks by using the `pytestmark` variable at the module level. ```python highlight={3} title="test_sum_powers.py" theme={null} import pytest pytestmark = pytest.mark.benchmark def test_sum_squares(): input = [1, 2, 3, 4, 5] output = sum(i**2 for i in input) assert output == 55 def test_sum_cubes(): input = [1, 2, 3, 4, 5] output = sum(i**3 for i in input) assert output == 225 ``` ### The `benchmark` fixture When more fine-grained control is needed, the `benchmark` fixture can be used. This fixture is exposed by the `pytest-codspeed` plugin, allowing to select exactly the code to be measured. A fixture is a function that can be used to set up and tear down the state of a test. More information about fixtures can be found in the [pytest documentation](https://docs.pytest.org/en/6.2.x/fixture.html). #### Direct invocation The fixture can be used directly in the test function: ```python theme={null} def test_sum_squares(benchmark): data = [1, 2, 3, 4, 5] benchmark(sum, data) # Only the `sum` function is measured ``` The fixture behaves as an identity function: calling `benchmark(target, *args, **kwargs)` will have the same effect as calling `target(*args, **kwargs)`. The return value will also be passed along to make it possible to write assertions on the result. For example: ```python theme={null} def test_sum_squares(benchmark): input = [1, 2, 3, 4, 5] output = benchmark(sum, [i**2 for i in input]) assert output == 55 ``` It's also possible to use it with lambda functions: ```python theme={null} def test_sum_squares(benchmark): input = [1, 2, 3, 4, 5] output = benchmark(lambda: sum(i**2 for i in input)) assert output == 55 ``` #### As a decorator If you want to measure a block of code containing multiple function calls, you can use the fixture as a decorator: ```python theme={null} def test_sum_squares_cubes(benchmark): input = [1, 2, 3, 4, 5] @benchmark def measured_function(): squares = sum(i**2 for i in input) cubes = sum(i**3 for i in input) return squares + cubes ``` When using the fixture, the marker is not necessary anymore, except if you want to customize the execution. **The benchmark fixture can only be used once per test function**. For example, the following code will raise an error: ```python theme={null} def test_invalid(benchmark): benchmark(func1) # OK benchmark(func2) # ERROR: RuntimeError ``` ### Benchmark options The `@pytest.mark.benchmark` marker accepts several options to customize the benchmark execution: The group name to use for the benchmark. This can be useful to organize related benchmarks together. *(Will be supported soon in the UI)* The minimum time of a round (in seconds). Only available in walltime mode. The maximum time to run the benchmark for (in seconds). Only available in walltime mode. The maximum number of rounds to run the benchmark for. Takes precedence over max\_time. Only available in walltime mode. Example usage: ```python theme={null} @pytest.mark.benchmark( group="sorting", min_time=0.1, max_time=1.0, max_rounds=100 ) def test_sorting_algorithm(benchmark): data = [1, 2, 3, 4, 5] benchmark(quicksort, data) ``` The `min_time`, `max_time` and `max_rounds` options are only available in walltime mode. When using CPU simulation mode (Valgrind), these options are ignored. ### Pedantic mode (advanced) For fine-grained control over benchmark execution protocol, you can use the `benchmark.pedantic` method. For example: ```python theme={null} def test_pedantic_mode(benchmark): def setup(): # Setup code that shouldn't be measured data = list(range(1000)) return (data,), {} # Returns (args, kwargs) for target def target(data): # Code to benchmark return sorted(data) def teardown(data): # Cleanup code that shouldn't be measured data.clear() result = benchmark.pedantic( target, setup=setup, teardown=teardown, rounds=5, # Number of rounds to run warmup_rounds=1 # Number of warmup rounds ) ``` The `benchmark.pedantic` method accepts the following parameters: The function to benchmark. This is the main code that will be measured. Positional arguments to pass to the target function. Keyword arguments to pass to the target function. Optional setup function that runs before each round. If it returns a tuple of (args, kwargs), these will be passed to the target function. Optional teardown function that runs after each round. Receives the same arguments as the target function. Number of warmup rounds to run before the actual benchmark. These rounds are not included in the measurements. Number of rounds to run the benchmark for. This parameter is ignored when using the CPU simulation mode. Number of iterations to run within each round. The total number of executions will be $rounds \times iterations$. This parameter is ignored when using the CPU simulation mode. ## Recipes ### Parametrized benchmarks `pytest-codspeed` fully supports pytest's parametrization out of the box: ```python theme={null} import pytest @pytest.mark.parametrize("size", [10, 100, 1000]) def test_parametrized_benchmark(benchmark, size): data = list(range(size)) benchmark(sum, data) ``` For complex parameters or values that may change over time, attach explicit ids so each case keeps a stable, readable name across runs: ```python theme={null} @pytest.mark.parametrize( "size", [10, 100, 1000], ids=["small", "medium", "large"], ) def test_parametrized_benchmark(benchmark, size): data = list(range(size)) benchmark(sum, data) ``` See [Naming Parametrized Cases](/docs/guides/how-to-benchmark-python-with-pytest#naming-parametrized-cases) in the pytest guide for the full rationale. ## Compatibility `pytest-codspeed` is designed to be fully backward compatible with [pytest-benchmark](https://pypi.org/project/pytest-benchmark/). You can use both plugins in the same project, though only one will be active at a time. ## Running the benchmarks continuously To run the benchmarks continuously in your CI, you can use `pytest-codspeed` along with the CodSpeed runner. We have first-class support for the following CI providers: } /> } /> } /> } /> If your provider is not listed here, please [open an issue](https://github.com/CodSpeedHQ/codspeed) or contact us on [Discord](https://discord.com/invite/MxpaCfKSqF). # Security Source: https://codspeed.io/docs/security CodSpeed's compliance certifications, data handling practices, and the permissions it requests For a detailed overview of CodSpeed's security practices, policies, and controls, visit the [Trust Center](https://trust.codspeed.io). For anything not covered there, contact [security@codspeed.io](mailto:security@codspeed.io). ## SOC 2 CodSpeed is SOC 2 Type II compliant. To request a copy of the report, contact [security@codspeed.io](mailto:security@codspeed.io). SOC 2 Badge

## GDPR CodSpeed is GDPR compliant. A Data Processing Agreement (DPA) is incorporated into the [Terms of Service](https://codspeed.io/terms) and applies to every customer. How personal data is collected, used, and protected is described in the [Privacy Statement](https://codspeed.io/privacy). GDPR Badge

## Source Code CodSpeed never stores your source code. When the [CodSpeed Wizard](/docs/ai/wizard) is used, your repository is fetched directly from GitHub to perform its analysis, but is not retained after the operation completes. ## GitHub Permissions For a full breakdown of the permissions CodSpeed requests from GitHub and why each is needed, see the [GitHub integration permissions](/docs/integrations/providers/github#github-permissions). ## Data Deletion Repositories can be deleted from the dashboard at any time, and account or organization deletion can be requested from support. See [Data Deletion](/docs/data-deletion) for the full process. # Troubleshooting Source: https://codspeed.io/docs/troubleshooting Understand the warnings CodSpeed reports on your benchmarks, and fix them ## Optimized-out benchmarks CodSpeed reports that a benchmark **measured no execution time** when it detects no execution of the code under test. The compiler likely removed that code, since nothing in the benchmark uses its result. Keep the compiler from optimizing the benchmark away, then run it again: # What is CodSpeed? Source: https://codspeed.io/docs/what-is-codspeed Integrated CI tools for software engineering teams to anticipate the impacts of the next delivery on performances. CodSpeed is a continuous performance testing platform to track, compare and optimize the performance of your codebase during development, before having performance issues in production. Running benchmarks using a traditional statistical approaches is usually unreliable on CI environments. To measure the performance of your code in an accurate and reproducible way, CodSpeed uses various instruments when running your code. Those instruments isolate noisy neighbors and all the side effect they can have on when measuring the performance of our workloads. CodSpeed then brings concise performance reports during your development process. To help you improve your codebase performance and find bottlenecks, CodSpeed gives you extended insights, including detailed execution profiles for all your performance tests. For the most convenience, all these perks are directly available within your repository provider (Pull Requests comments and Merge checks). ## What can be measured? CodSpeed can measure the performance of different types of code, thanks to its instruments: * The core of CodSpeed is the [CPU instrument](/docs/instruments/cpu). Use it measure the performance of algorithms, data processing steps, mathematical operations, and basically any kind CPU-bound tasks. Extensive data is collected during the execution of the code, which is used to create consistent [performance measurement](/docs/features/understanding-the-metrics/) and [detailed flame-graphs](/docs/features/profiling) of the code. * The [Walltime instrument](/docs/instruments/walltime/) measures the real time taken by your code to execute. It is useful to measure end-to-end performance of your code, including system calls, I/O operations, network requests, and database queries. It leverages CodSpeed's [Macro Runners](/docs/features/macro-runners/) to provide a stable and isolated environment for running your benchmarks with low noise and high precision. * The [Databases instrument](/docs/instruments/databases/) measure the performance of database queries. They allow you to track and compare the performance of your database queries during development. ## How does it work? Overview of CodSpeed's architecture

The core of the performance measurement is done directly within your CI environment through the CodSpeed Action, allowing you to stay the sole owner of your data and to keep your codebase private. Once the performance data is generated, only the benchmark results are sent to the CodSpeed servers to be analyzed. Then, performance reports are generated and published directly within your Pull/Merge Request comments. ## How long does it take to install? The fastest way to get started is the [CodSpeed Wizard](/docs/ai/wizard), which automates the entire setup: it analyzes your repository, configures benchmarks, generates a CI workflow, and opens a pull request — all in minutes. If you prefer to set things up manually, you can plug your existing benchmark suites into CodSpeed in less than 5 minutes since CodSpeed's benchmark API is compatible with the most popular benchmarking frameworks for each language. If you don't have any benchmarks yet, you can get started by [creating benchmarks](/docs/benchmarks/overview). You can create your first benchmarks in just a few minutes by reusing existing unit tests.