> ## Documentation Index
> Fetch the complete documentation index at: https://codspeed.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Walltime Instrument Overview

> Learn how to use CodSpeed's walltime instrument for measuring real-world execution time in your benchmarks. 

export const CIWorkflow = ({minimal = false, enableWorkflowDispatch = true, runsOn = "ubuntu-latest", highlight = [], mode, modes, submodules = false, preSteps = [], buildSteps = ["# ...", "# Setup your environment here:", "#  - Configure your Python/Rust/Node version", "#  - Install your dependencies", "#  - Build your benchmarks (if using a compiled language)", "# ..."], benchmarkCommand = ["<Insert your benchmark command here>"], jobName = "Run benchmarks", env = {}}) => {
  const modeList = modes || (mode ? [mode] : undefined);
  if (!modeList || modeList.length === 0) {
    throw new Error("mode or modes is required");
  }
  const indent = (lines, depth) => {
    const reindentedLines = lines.map(l => l.length === 0 ? l : (" ").repeat(depth) + l);
    return reindentedLines.join("\n");
  };
  const workflowDispatchSection = enableWorkflowDispatch ? "  # `workflow_dispatch` allows CodSpeed to trigger backtest\n" + "  # performance analysis in order to generate initial data.\n" + "  workflow_dispatch:\n" : "";
  let yaml = "";
  if (!minimal) {
    yaml += `
name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
`;
    yaml += workflowDispatchSection;
  }
  yaml += `
jobs:
  benchmarks:
    name: ${jobName}
    runs-on: ${runsOn}`;
  if (!minimal) {
    yaml += `
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed`;
  }
  if (preSteps.length > 0) yaml += "\n" + indent(preSteps, 4);
  yaml += `
    steps:
      - uses: actions/checkout@v5`;
  if (submodules) {
    const value = typeof submodules === "string" ? submodules : "true";
    yaml += `\n        with:\n          submodules: ${value}`;
  }
  yaml += "\n" + indent(buildSteps, 6);
  const modeValue = modeList.join(",");
  yaml += `
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v4
        with:
          mode: ${modeValue}`;
  if (benchmarkCommand.length > 0) {
    const indentedBenchCommand = benchmarkCommand.length > 1 ? benchmarkCommand[0] + "\n" + indent(benchmarkCommand.slice(1), 12) : benchmarkCommand;
    const runLine = indent(["run: "], 10) + indentedBenchCommand;
    yaml += `\n${runLine}`;
  }
  const envEntries = Object.entries(env);
  if (envEntries.length > 0) {
    const envLines = ["env:", ...envEntries.map(([k, v]) => `  ${k}: ${v}`)];
    yaml += "\n" + indent(envLines, 8);
  }
  return <CodeBlock language="yaml" highlight={JSON.stringify(highlight)} {...minimal || ({
    filename: ".github/workflows/codspeed.yml",
    icon: "github"
  })}>
      {yaml}
    </CodeBlock>;
};

The walltime instruments allow measuring the **walltime** of your benchmarks
directly in the CI. It leverages **bare-metal runners managed and provided by
CodSpeed** to measure the performance of your benchmarks with **low noise** and
**high precision**.

<Frame caption="Example of a walltime benchmark run">
  <img src="https://mintcdn.com/codspeed/_9gjNuBHMdsdxHot/instruments/walltime/assets/walltime-benchmark-sample.png?fit=max&auto=format&n=_9gjNuBHMdsdxHot&q=85&s=30cd503a285ece751baac90699d28f1f" className="rounded-xl w-full max-w-lg mx-auto" alt="Example of a walltime benchmark run" width="1374" height="994" data-path="instruments/walltime/assets/walltime-benchmark-sample.png" />
</Frame>

<Info>
  **Supported languages and integrations**

  At the moment, the walltime instrument is supported for the following languages
  and integrations:

  * **Python**:
    [`pytest-codspeed >= 3.0.0`](https://github.com/CodSpeedHQ/pytest-codspeed/releases/tag/v3.0.0)
  * **Rust**
    * [`codspeed-divan-compat >= 2.8.0`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.8.0)
    * [`codspeed-criterion-compat >= 2.9.1`](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v2.9.1)
  * **C++**:
    [`codspeed-google-benchmark >= 1.0.0`](https://github.com/CodSpeedHQ/codspeed-cpp/releases/tag/v1.0.0)
  * **Node.js**
    * [`@codspeed/vitest-plugin >= 5.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.0.0)
    * [`@codspeed/tinybench-plugin >= 5.0.0`](https://github.com/CodSpeedHQ/codspeed-node/releases/tag/v5.0.0)
  * **Go**:
    [`codspeed-go >= 0.3.0`](https://github.com/CodSpeedHQ/codspeed-go/releases/tag/v0.3.0)

  If you want to use the walltime instrument with a different language or
  integration, please reach out on [Discord](https://discord.gg/MxpaCfKSqF) or
  [email our support](mailto:contact@codspeed.io).
</Info>

## What Does the Walltime Instrument Measure?

The walltime instrument measures the **actual elapsed time** (also known as
"wall clock time") of your benchmark execution. Unlike CPU simulation which
measures simulated CPU cycles, walltime captures the real-world duration
including:

* **All code execution**: Both user-space code and system calls are included in
  the measurement, giving you a complete picture of actual runtime performance.

* **I/O operations**: Network requests, file system operations, and other I/O
  bound tasks are fully captured, making this instrument ideal for benchmarks
  that interact with external systems.

* **Parallelism effects**: Multi-threaded code benefits are accurately measured
  since walltime reflects the actual elapsed time, not CPU time across threads.

This makes the walltime instrument particularly valuable when you need to
measure performance beyond what the CPU simulation instrument can capture, such
as integration tests on API endpoints or workloads that rely on external
dependencies.

<Info>
  **Multiple benchmark processes**

  With the walltime instrument, you should try and avoid running multiple
  benchmark processes in parallel since this can lead to noisy measurements.

  Thus, using `pytest-xdist` or similar tools is not recommended.
</Info>

## Automated Profiling

When using the walltime instrument, CodSpeed automatically collects
[profiling data and generates flame graphs](/features/profiling) for each
benchmark. This allows you to quickly identify performance changes and their
root causes.

<img src="https://mintcdn.com/codspeed/CInbng288QuXBkrC/features/assets/function-list.png?fit=max&auto=format&n=CInbng288QuXBkrC&q=85&s=4e535467c1e8a6410668d5eb6bf902a0" alt="Function list" width="2729" height="1584" data-path="features/assets/function-list.png" />

### Inspector Metrics

<img src="https://mintcdn.com/codspeed/J_9QffCKgr3Hbs6D/instruments/walltime/assets/walltime-tooltip-explanation.excalidraw.png?fit=max&auto=format&n=J_9QffCKgr3Hbs6D&q=85&s=aca9e0b2ac5ef2f731696253c9a93e78" alt="Flamegraph inspector" width="1632" height="900" data-path="instruments/walltime/assets/walltime-tooltip-explanation.excalidraw.png" />

When you hover over a span in the flame graph, the inspector displays the
following metrics:

**Common metrics:**

* **Self time**: The measured execution time spent in the function body only,
  excluding time spent in child function calls.
* **Total time**: The measured execution time spent in the function including
  all its children.

**Execution events**

The walltime instrument also collects hardware events during execution. This
happens automatically when events are available. All displayed event counts are
cumulative and include events from child function calls.

* **CPU Cycles**: The number of CPU cycles elapsed.
* **Instructions**: The number of CPU instructions executed.
* **Memory R/W**: The number of memory read and write operations performed.
* **Memory Access Pattern**: A breakdown of how memory accesses were served:
  * **L1 Cache Hits**: Memory accesses served from the fastest CPU cache.
  * **L2 Cache Hits**: Memory accesses served from the second-level cache.
  * **Cache Misses**: Memory accesses that required fetching from main memory.
  * **Memory access distribution**: Total bytes read from and written to memory,
    for each level of cache. It is calculated based on the number of events, and
    the average size of each access, namely a word for a cache access, and a
    cache line for a cache miss.

<Info>
  **Sampling accuracy**

  Event counts are collected using hardware performance counter sampling. The
  deeper you navigate into leaf functions, the more susceptible these counts
  suffer from to sampling-related inaccuracies. For the most reliable data, focus
  on higher-level functions in the call stack.
</Info>

## Usage with GitHub Actions

**Requirements:**

* `CodSpeedHQ/action >= 3.1.0`

The CI setup is exactly the same as
[the one with the CPU simulation](/integrations/ci/github-actions#2-create-the-benchmarks-workflow)
but instead of running on a GitHub-hosted runner, you'll need to request a
"codspeed-macro" runner. [Macro Runners](/features/macro-runners) are bare-metal
machines managed by CodSpeed and will provide you with a more stable and precise
environment to run your benchmarks. Simply replace the `runs-on: ubuntu-latest`
line with `runs-on: codspeed-macro` and use the `mode: walltime` option in the
action:

<CIWorkflow minimal runsOn="codspeed-macro" mode="walltime" highlight={[4, 19]} />

Your benchmarks will now run on a CodSpeed-managed runner, the action and the
benchmark integration will automatically collect walltime data and you'll be
able to see the new measurements in the CodSpeed dashboard.

<Tip>
  **Caching**

  If you're using caches in the GitHub action workflow, make sure your cache keys
  include `runner.arch` to avoid cache misses since CodSpeed Macro runners are
  running on the ARM64 architecture.

  For example:

  ```yaml {4} theme={null}
  - uses: actions/cache@v4
    with:
      path: /home/.cache/pip
      key: pip-${{ runner.arch }}-${{ hashFiles('pyproject.toml') }}
  ```
</Tip>

### Usage on personal GitHub accounts

At the moment, the **macro runners are only available for organizations** and
not for personal accounts. This is because registering GitHub self-hosted
runners on repositories instead of organizations would require the **Repository:
Administration (Read/Write)** permission, which is too broad.

To use the macro runners on a repository owned by a personal GitHub account, the
only solution is to create a new organization and transfer the repository to
that organization.

### Usage on public repositories

By default, the macro runners are only available for the private repositories of
your organization. If you want to use them on a public repository, you'll need
to explicitly allow them from your GitHub organization settings (under
**Organization Settings** > **Actions** > **Runner groups** > **Default**).

<img src="https://mintcdn.com/codspeed/_9gjNuBHMdsdxHot/instruments/walltime/assets/github-public-repo-access.png?fit=max&auto=format&n=_9gjNuBHMdsdxHot&q=85&s=fd62b02d6c9d620f4fa8723334d568c3" className="rounded-xl w-full max-w-lg mx-auto" alt="Allowing macro runners on a public repository" width="1400" height="372" data-path="instruments/walltime/assets/github-public-repo-access.png" />

## Next Steps

<CardGroup cols={2}>
  <Card title="Macro Runners" icon="server" href="/features/macro-runners">
    Learn more about CodSpeed managed bare-metal runners designed for Walltime
    benchmarks
  </Card>

  <Card title="Profiling" icon="bars-sort" href="/features/profiling">
    Learn more about automated profiling for Walltime benchmarks
  </Card>

  <Card title="Profiling on macOS" icon="apple" href="/instruments/walltime/macos-profiling">
    Learn why system processes are missing from macOS profiles and how to work
    around it
  </Card>
</CardGroup>