
One of the easiest ways to track performance regressions in your JavaScript applications is to use Vitest's `bench` API.
It lets you write benchmarks as easily as unit tests:

```ts
import { bench, describe, test } from "vitest";

describe("fibonacci", () => {
  test("should return 55 when called with 10", () => {
    expect(fibonacci(10)).toBe(55);
  });

  bench("fibo 10", () => {
    fibonacci(10);
  });
});
```

Then you can run the benchmarks with `vitest bench`:

```bash
pnpm vitest bench
```

The problem that we now face is that the results are **inconsistent between runs**. Running the command multiple times in a CI/CD runner or on your machine will yield different results.
Thus it will be hard to spot performance regressions with confidence.

This is where using CodSpeed comes in handy. CodSpeed provides a way to **run benchmarks in a consistent manner**, and thus **detect performance regressions with confidence**.

We are excited to announce that CodSpeed now has first-class support for Vitest!

## How to use Vitest with CodSpeed

Install the `@codspeed/vitest-plugin` package and `vitest` as dev dependencies:

```bash
pnpm add -D @codspeed/vitest-plugin vitest
```

<Alert status="info">
  The CodSpeed plugin is only compatible with
  [vitest@1.0.0](https://www.npmjs.com/package/vitest/v/1.0.0) and above.
</Alert>

Create or update the `vitest.config.ts` file at the root of your project:

```ts title="vitest.config.ts"
import { defineConfig } from "vitest/config";
import codspeedPlugin from "@codspeed/vitest-plugin";

export default defineConfig({
  plugins: [codspeedPlugin()],
  // ...
});
```

<Alert>
  If you are on a project using `vite` and `vitest`, you can also add the plugin
  to the `vite.config.ts` file.
</Alert>

Let's create a fibonacci function and create some benchmarks on it.
Create a new file `src/fibonacci.ts`:

```ts
export function fibonacci(n: number): number {
  if (n <= 1) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
}
```

Create a new file `src/fibonacci.bench.ts`:

```ts
import { bench, describe } from "vitest";
import { fibonacci } from "./fibonacci";

describe("fibonacci", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });

  bench("fibo 15", () => {
    fibonacci(15);
  });
});
```

Now we can run the benchmarks with `vitest bench`:

```bash
$ vitest bench

[CodSpeed] bench detected but no instrumentation found, falling back to default vitest runner
 RUN  v1.0.3 ...

 ✓ src/fibonacci.bench.ts (2) 1521ms
   ✓ fibonacci (2) 1520ms
     name                hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · fibo 10   1,760,687.42  0.0005  1.0279  0.0006  0.0006  0.0007  0.0007  0.0016  ±0.50%   880344   fastest
   · fibo 15     173,932.15  0.0052  0.0594  0.0057  0.0059  0.0064  0.0070  0.0147  ±0.08%    86967


 BENCH  Summary

  fibo 10 - src/fibonacci.bench.ts > fibonacci
    10.12x faster than fibo 15
```

We can see that CodSpeed has been detected and that the benchmarks have been run.
For the moment we can see the default output of Vitest's `bench` API.

## Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to detect the CI environment and properly configure the instrumented environment.

### Enable the repository on CodSpeed

The first step is to log in to CodSpeed and import the repository you want to track. You can find more information about this in the [CodSpeed documentation](https://docs.codspeed.io/importing-repositories).

After installing the GitHub app and importing the repository in CodSpeed, you will be prompted a `CODSPEED_TOKEN` secret to copy. This token is used to authenticate the CodSpeed GitHub Actions workflow.

### Add a CodSpeed GitHub Actions workflow

Open a Pull Request on your repo with the previous changes, and the following GitHub Actions workflow `.github/workflows/codspeed.yml`.
It will run the benchmarks and report the results to CodSpeed on every push to the `main` branch and every pull request:

```yaml title=".github/workflows/codspeed.yml"
name: codspeed-benchmarks

on:
  push:
    branches:
      - "main"
  pull_request:

jobs:
  benchmarks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: pnpm/action-setup@v2
      - uses: actions/setup-node@v3
        with:
          cache: pnpm
      - name: Install dependencies
        run: pnpm install
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v2
        with:
          run: pnpm vitest bench
          # token retrieved from the CodSpeed app at the previous step
          token: ${{ secrets.CODSPEED_TOKEN }}
```

If everything went well, the workflow should run and after it finishes you should see a comment from the CodSpeed GitHub app on the Pull Request with the run report.

### Visualizing the benchmarks on the CodSpeed app

By clicking on the CodSpeed comment in the Pull Request, you will be redirected to the CodSpeed report of the PR.
You will see a list of the benchmarks, and when clicking on the `fibo 10` benchmark its flame graph will appear:

<Figure>
  <FlameGraph height={150} flameGraphsPath="fibo10-vitest" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

And that's it for the installation of CodSpeed with Vitest, now we can explore a bit more what CodSpeed offers.

## Visualizing a performance change

One of the main features of CodSpeed is to detect and visualize the performance changes of a pull request.

Let's now say that we are on a more complex project where we have a `parsePr` function that parses a GitHub pull request.
We created a pull request with some performance improvements, and the following is in the CodSpeed report:

<PerfDiff
  base={1.5e-3}
  head={1.2e-3}
  name="parsePr::short body"
  description="Improve parsing performance"
/>
<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-vitest" />
  <Caption>
    The Diff graph shows the resulting graph of the difference between the head
    and base runs, allowing us to pinpoint where the performance changes are
    inside the code
  </Caption>
</Figure>

We observe a 25% improvement in the overall benchmark, driven by enhancements in the `parsePr` and `parseIssueFixed` functions.
Although the `logMetrics` function has experienced regression, its impact on the overall benchmark is less pronounced compared to the observed improvements.
Additionally, a new function, `modifyTitle`, is introduced and denoted by the distinctive blue color.

Thanks to the differential flame graph, it is now really easy to understand at a glance what code changes impact the performance.

## Wrapping up

And there you have it—CodSpeed now supports Vitest to make performance testing a breeze for your JavaScript and TypeScript projects! With an easy-to-use bench API and the magic touch of CodSpeed's consistency, tracking regressions is as simple as adding a unit test.

Get ready to supercharge your CI with performance reports, flame graphs, and more!

---

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles and product updates.

## Resources

- [Vitest integration](https://docs.codspeed.io/benchmarks/nodejs/vitest) in the CodSpeed documentation
- [@codspeed/vitest-plugin](https://github.com/CodSpeedHQ/codspeed-node/tree/main/packages/vitest-plugin) on GitHub
- [Trace Generation](https://docs.codspeed.io/features/trace-generation) in the CodSpeed documentation
- [Vitest bench API](https://vitest.dev/api/#bench)



Flame Graphs are a visualization tool that helps you understand how your software is performing.
They display the call stack of a program and the time spent in each function.
They are a powerful tool to quickly identify performance bottlenecks.

Differential Flame Graphs combine two flame graphs to highlight the differences between them. They allow you
to easily spot performance regressions and improvements and gain invaluable insights into your software's performance.

Below you can find an example of normal and differential flame graphs following a change in a codebase.
At a glance, we can understand where the code has become slower. Here it is the `parse_issue_fixed` function as it
has the brightest red color.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

Let's explore how to read flame graphs and how CodSpeed automates flame graph generation in your CI pipeline.

<Alert>
  CodSpeed supports Rust, Node.js, Python and generates flame graphs out of the
  box. More languages are coming soon!
</Alert>

## Spotting a performance regression

Let's dive into the previous example.

We have a function `parse_issue_fixed` that parses a GitHub pull request body and extracts the issue
number that it fixes. Given `body = "fixes #123"`, the function returns `123`. Here `body` can be
a multiline string, and the issue number can be anywhere in the string.

<Alert>
  On GitHub, adding the string `fixes #123` in a pull request body will
  automatically close the issue #123 when the pull request is merged.
</Alert>

We notice that the code is quite long and not easily understandable. We refactor it to use a regular expression instead.
This gives us the following diff:

```diff
+import re

+FIXES_REGEX = re.compile(r"fixes #(\d+)")
+
+
 def parse_issue_fixed(body: str) -> int | None:
-    prefix = "fixes #"
-    index = body.find(prefix)
-    if index == -1:
-        return None
-
-    start = index + len(prefix)
-    end = start
-    while end < len(body) and body[end].isdigit():
-        end += 1
-    return int(body[start:end])
+    match = FIXES_REGEX.search(body)
+    return int(match.group(1)) if match else None
```

Great, we have a much shorter and more readable function. But did we introduce a performance regression?

We already have a benchmark in place for the function `parse_pr`, a higher-level function that
parses a GitHub pull request body and calls multiple functions to retrieve information,
including `parse_issue_fixed`. The input is a multiline string of approximately 50kB.
We chose a large input to better understand the performance characteristics of the function and make sure the function performs well when parsing large pull request bodies.

Let's check out the flame graphs for the benchmark to analyze the performance impact of our change.
The **Base** flame graph is the one before the change, the **Head** flame graph is the one after
the change, and finally, the **Diff** flame graph is the difference between the two.

<PerfDiff
  base={1.2e-3}
  head={1.525e-3}
  name="test_parse_pr"
  description="Refactor with regular expression"
/>
<div style={{ marginBottom: "16px" }}>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
</div>

Here we can see that the `parse_issue_fixed` function is bright red, thus denoting that it
was slower after the change and had the biggest performance impact on the benchmark.
When hovering over the `parse_issue_fixed` frame, we can see that the performance
impact is a _-21.5% regression_ on the overall benchmark.

So in our use case, using a regular expression will result in slower executions compared
to using `str.find`.

So yes, our change for the sake of readability introduced a performance regression.
We can now make an informed decision on whether we want to keep the change or not.

Now that we understand how a differential flame graph works and how useful it is,
let's explore how to generate one.

## Generating flame graphs manually is unreliable and time-consuming

The first thing to note is that manually generated flame graphs on a dev machine are **not consistent**.

Running the same script multiple times with the same input will result in different flame graphs.
Indeed, other processes running on the machine will impact the execution time of the script and its different functions.
This make it hard to spot performance regressions with confidence.

The second thing to note is that generating flame graphs manually is **tedious**.

The first tool to generate flame graphs is the eponym FlameGraph by Brendan Gregg written in Perl.
Since then, many other tools have been created in and for other languages. The general steps are the same:

- Run your script with a profiler and output the profile in a file
- Run the tool on the profile to generate a flame graph svg
- Open the generated svg in your browser and explore the data

This is already a quite heavy process: you have to create a script to isolate the part of your program
you want to generate a flame graph for, then run a bunch of commands and finally open
a file in your browser.

This is why we decided to automate the generation of flame graphs in your CI pipeline with CodSpeed,
and find a way to make sure that the flame graphs are consistent.

## Automating flame graphs with CodSpeed

Since we already had wrappers of benchmarking libraries in different languages, we decided to augment them with
profiling capabilities.

### Steps to follow to get flame graphs in your CI pipeline

Write benchmarks with one of our wrappers in Rust, Node.js, or Python.

For example, in Python, use our `pytest` extension `pytest-codspeed` (which API is compatible with `pytest-benchmark`)
and add a test marked as a benchmark:

```python
import pytest
from parse_pr import parse_pr

@pytest.mark.benchmark
def test_parse_pr():
    parse_pr(body=very_long_body_string, pr_number=126, title="Refactor some code")
```

Run the benchmarks in the `@codspeed/action` GitHub Action:

```yml
- name: Run benchmarks
  uses: CodSpeedHQ/action@v2
  with:
    run: pytest tests/ --codspeed
```

The action will run the benchmarks on a "virtualized" CPU with Valgrind.
Instead of measuring the execution time, it measures the number of CPU cycles and memory accesses.
Each benchmark is only ran once and it is enough to get **consistent data**.

This data is then sent to CodSpeed's servers and flame graphs are generated.
A comment is added to the GitHub Pull Request, with a link to the CodSpeed app, where you can browse
the benchmarks and their flame graphs.

And voilà! You now have flame graphs in your CI pipeline, with the benefits of consistency and automation.
Moreover, now that are using CodSpeed, **performance regressions will automatically be detected and reported**
in your pull requests 🎉

<Alert>
  Head out to our [documentation](https://docs.codspeed.io) to view integrations
  for the different languages.
</Alert>

## Going further

Flame graphs are not only useful for spotting performance regressions but they can also be used to
understand the performance impact of a change. Being an improvement or new calls to a function.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-complex" />
  <Caption>
    A more complex flame graph with regressions, improvements, and added code
  </Caption>
</Figure>

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles.

## Resources

- [Trace Generation](https://docs.codspeed.io/features/trace-generation) in the CodSpeed documentation
- [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed), plugin for pytest
- [@codspeed/action](https://github.com/CodSpeedHQ/action), GitHub Action to run benchmarks and generate flame graphs
- [Flame Graphs](https://www.brendangregg.com/flamegraphs.html) by Brendan Gregg
- [Differential Flame Graphs](https://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html) by Brendan Gregg
- [Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
- [Valgrind](https://valgrind.org/)
- [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark)



During the last advent of code, while starting with Rust, I discovered that the Rust standard library was missing the usual `div_mod`(a.k.a. `div_rem`) function that would simultaneously return the quotient and remainder. I got confused since I already used the operator in C++ with [std::div](https://en.cppreference.com/w/cpp/numeric/math/div) or Python with [divmod](https://docs.python.org/3/library/functions.html#divmod). It seemed a basic way of improving performance since both values would come out with the same `DIV` assembly instruction, nicely avoiding extra CPU cycles.

Looking it up, I found [an existing RFC](https://github.com/rust-lang/rfcs/issues/1887) created in 2017, but it hasn't received much attention since.

So, do we need a `div_rem` implementation in the standard library? I decided to investigate.

## A naive approach

Let's naively implement the function:

```rust
pub fn naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    (a / b, a % b)
}
```

To dissect what is executed we're gonna analyze the compilation steps.

<Figure>

![Rust Compilation Chain](compilation-chain.excalidraw.png)

  <Caption>The compilation chain of our Rust program</Caption>
</Figure>

First, let's have a look at the LLVM Intermediate Representation(IR). If you're not
familiar with it, is an extremely low-level programming language close to assembly but
without being tied to specific hardware. It's extremely useful for compiler writers
since it's a common language that can be used to target multiple architectures. It's
also a great tool to understand what the compiler is doing and how it "interpret" our
code.

For the previous code, here is the annotated LLVM IR we can obtain:

```llvm
; Define a function called 'example::naive_div_rem
define { i32, i32 } @example::naive_div_rem(i32 %a, i32 %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for our return tuple
  %_4 = icmp eq i32 %b, 0; Check if divisor %b is zero
  br i1 %_4, label %panic, label %bb1; If %b is zero, panic, else continue
bb1:
  %_3 = udiv i32 %a, %b; Perform unsigned division(only get the quotient)
  %_6 = icmp eq i32 %b, 0; Check again for the divisor being zero (for the modulo)
  br i1 %_6, label %panic1, label %bb2; If %b is zero, panic, else continue
bb2:
  %_5 = urem i32 %a, %b; Compute the unsigned remainder
  ; ...
  ; Stores quotient and the remainder in a tuple, load them back,
  ; and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %8
; ... omitted panic handlers ...
}
```

We can witness that this code contains redundant checks for the division and modulo by 0. Also, we can note two operations to get our quotient and remainder: `udiv` and
`urem`. This is expected at this stage since LLVM doesn't have an instruction combining
them but hopefully, the compiler will be able to optimize this while generating the assembly.

Let's have a look at the next compilation step so we can dive into the assembly code generated from the IR(`x86_64` with optimizations enabled):

```asm
example::naive_div_rem:
    test    esi, esi  ; Test if esi (divisor) is zero
    je      .LBB0_2   ; Jump to panic block if esi is zero
    mov     eax, edi  ; Move the dividend into eax
    xor     edx, edx  ; Zero out edx to prepare for the division
    div     esi       ; Unsigned division, quotient goes into eax, remainder into edx
    ret
.LBB0_2:
    ; ... "attempt to divide by zero" panic handler ...
```

And voila! We can see that the compiler was able to optimize our code and fused the two
operations into a single `div` instruction. So having a dedicated implementation
wouldn't be that helpful for performance while using `rustc` since this optimization
should almost always be done by the IR compiler.

You might have noticed quite a few things in the assembly code, let's have a look at them by creating our own assembly version.

## An inline assembly version

```rust
use std::arch::asm;

pub fn asm_div_rem(a: u32, b: u32) -> (u32, u32) {
    let mut tmp: u32 = a;
    let mut remainder: u32 = 0;
    unsafe {
        asm!(
            "div {divisor}",
            inout("eax") tmp,
            inout("edx") remainder,
            divisor = in(reg) b,
            options(pure, nomem, nostack),
        );
    }
    (tmp, remainder)
}
```

A few things to note in this implementation:

- The `div` instruction divides the `eax` register by its operand value. It will store the quotient in `eax` and the remainder in `edx`.
- `tmp` holds the dividend before our assembly is executed. It will contain the quotient after. This is because its value is tied to the `eax` register and it will be used both for input and output thanks to the `inout` operand.
- `remainder` is also passed with the `inout` operand, allowing us to clear the `edx` register before the operation is done since `remainder` is 0 before the assembly is executed. At the end, the register will hold the remainder value that we return in the output tuple.
- The divisor is passed through any general purpose register as a simple input value that we pass directly to the `div` instruction.
- Finally, we pass a few flags that will help the compiler to know how our inline assembly will behave with `options`. Here: our block has no side effect, we're neither reading nor writing to the memory and we don't push anything to the stack.

Compiling this, we can see that it ends up a bit more concise than the previous compiled assembly we built:

```asm
example::asm_div_rem:
    mov     eax, edi ; Move the value of the dividend from EDI to EAX.
    xor     edx, edx ; Zero out EDX to prepare for the division.
    div     rsi      ; Signed divide the dividend in EAX by the value in RSI('b').
    ret              ; Return from the function. quotient -> EAX, remainder -> EDX.
```

It should be faster but we're cheating a bit on rust here
since we're not taking into account the panic check that was done in the previous version.

<PerfDiff
  base={32.7e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to an inline assembly version"
/>

It's faster! The overhead of those panic checks is not that dramatic in the end. Branch
prediction is probably contributing a lot to this. It was eluded in the first IR
snippets but in there, the rust compiler hints the rest of the pipeline that those
checks are likely to succeed. The compiler's ability to optimize and hint that these
checks are more likely to pass than fail can lead to surprisingly efficient code, even
when it appears to be doing "much more" than the assembly version.

## The cost of panic handling: an unchecked version

Now that we built our unchecked version in assembly, we can get quite close to it by using unchecked div and rem from `std::intrisics`:

```rust
#![feature(core_intrinsics)]
pub unsafe fn unchecked_naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    let quotient = std::intrinsics::unchecked_div(a, b);
    let remainder = std::intrinsics::unchecked_rem(a, b);
    (quotient, remainder)
}
```

And its intermediate representation:

```llvm
; Define a function named 'example::unchecked_naive_div_rem' that takes two arguments
define { i32, i32 } @example::unchecked_naive_div_rem(i32 noundef %a, i32 noundef %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for a tuple to store the quotient and remainder
  %quotient = udiv i32 %a, %b; Perform unsigned division of %a by %b
  %remainder = urem i32 %a, %b; Compute the unsigned remainder of %a / %b
  ; ...
  ; Stores quotient and remainder in a tuple, loads them back, and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %1
}
```

As you can see this IR doesn't contain anything about overflow checks and the performance should be close to or even better than our assembly version.

<PerfDiff
  base={30.5e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to the unchecked version"
/>

That's exactly the same time our assembly version takes. So let's have a look at the assembly:

```asm
example::unchecked_naive_div_rem:
    mov     eax, edi
    xor     edx, edx
    div     esi
    ret
```

And it's the same as our assembly version! So bothering writing our assembly version is not really worth it in the end.

To conclude, there is no performance drawback of not having a single `div_rem` function
since the compiler optimizations will take care of this burden for us. It might seem odd
to write those two operations but unless very specific optimization killer cases they
will be fused as a single assembly instruction.

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles.

### Resources

- [Compiler Explorer](https://godbolt.org/): an amazing tool to visualize the assembly and IR generated by the compiler
- [Understanding LLVM IR](https://mukulrathi.com/create-your-own-programming-language/llvm-ir-cpp-api-tutorial/#understanding-llvm-ir)
- [The faker's guide to reading (x86) assembly](https://www.timdbg.com/posts/fakers-guide-to-assembly/)
- [Guide to x86 Assembly](https://www.cs.virginia.edu/~evans/cs216/guides/x86.html)



### Why does Continuous Performance analysis matter?

**Performance is the cornerstone of many problems in software development.** Application reactivity, infrastructure costs, energy consumption, and carbon footprint all depend on the performance of the underlying software. **By performance, we often mean execution speed** but more generally, it refers to the efficiency or throughput of a system.

Today, most of the performance monitoring is done through **Application Performance Monitoring** **solutions**(often referred to as APM), provided by companies such as Datadog, Sentry, Blackfire, and many others. These platforms bring **several interesting insights about the production environment's health**: client-side UX monitoring, endpoint latency checks, and even continuous production profiling.

However, these solutions are monitoring tools; **they are built to check if everything is okay in production,** and not really to run experiments. They need real users serving as guinea pigs to experience poor performance to be able to report anomalies.

<Figure>

![Performance feedback in the **current** Software Development Life-Cycle](SDLC_Traditional.png)

  <Caption>Performance feedback in the <b>current</b> Software Development Life-Cycle</Caption>
</Figure>

So as a developer, to understand the performance impact of my changes, I need to wait for end users to try out my changes in production!? Then, if something is wrong, maybe I’ll try to improve it, someday…

In an ideal world, **performance checks should be included way earlier in the development lifecycle**; just as an additional testing flavor, **nurturing continuous improvement**.

<Figure>

![Performance feedback in an **ideal** Software Development Life-Cycle](SDLC_Ideal.png)

  <Caption>
    Performance feedback in an <b>ideal</b> Software Development Life-Cycle
  </Caption>
</Figure>

This much shorter feedback loop would **provide visibility with consistent metrics to the teams while they are building** and not once everything is already shipped in production environments. Besides, **guesstimating performance is hard and often plain wrong**. Accurate performance reports help in those cases and can serve as an educational tool for software developers.

### Building a consistent performance metric

Measuring software performance in various environments to gather reproducible results is hard. The most basic metric we can think of is to **measure the execution time but just changing the hardware will produce completely different data points**. Running a program on a toaster will be significantly slower than running it on the latest generations of cloud instances. Joking aside, **merely** **using the same machine at different times will produce different results** because other unpredictable background tasks are eating up the CPU time.

One obvious solution to bring consistency in the results is to run the program in a controlled cloud environment where background running processes are very limited. Despite a significant improvement in the quality of the results, **this doesn’t give repeatable measurements either because of the [noisy neighbor issue](https://www.techtarget.com/searchcloudcomputing/definition/noisy-neighbor-cloud-computing-performance)**. Basically, physical cloud machines are shared among customers by splitting them into multiple Virtual Machines. At a software level, those are perfectly isolated but in the end, they still share some common hardware and **isolation can’t be perfect**(eg. memory, high-level CPU caches, network interfaces). Since it’s not possible to predict the workload running along a measurement task, finding the “truth” in this noise becomes a statistical challenge, and **building an extremely accurate metric is nearly impossible**.

<Figure>
  ![Time measurement for a Fibonacci sequence computation(Python runs from
  GitHub Action)](TimeMeasurementFibo.png)
  <Caption>
    Time measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)
  </Caption>
</Figure>

When it comes to consistency, working with execution time measurement seems doomed to failure. So what if we instead decided to dissect exactly what a virtual machine does to run our program? In this case, running the same measurement again and again on various machines produces the exact same results since **the hosted virtual machine always starts with a predefined state and emulates the same hardware**. We don’t mind the noisy neighbor issue either because **whatever is running along our measurement doesn’t change the instructions executed to run our program** in its sandboxed environment. Furthermore, based on those micro execution steps, it is possible to aggregate **a time equivalent metric that will be consistent, accurate (detect less than 1% performance changes), and hardware agnostic**: **this is how CodSpeed works!**

<Figure>

![CodSpeed Measurement for a Fibonacci sequence computation(Python runs from GitHub Action)](CodSpeedMeasurementFibo.png)

  <Caption>
    CodSpeed Measurement for a Fibonacci sequence computation(Python runs from GitHub Action)
  </Caption>
</Figure>

### Shifting left with CodSpeed

CodSpeed brings this consistent measurement to the Continuous Integration environments, enabling performance checks to be included in the software development lifecycle as early as possible.

On every new feature, the performance is measured and reported directly in the repository provider as a Pull Request comment. Optionally, status checks can also be enabled to enforce performance requirements to be satisfied before merging the new delivery.

<Figure>![Github Integration.png](Github_Integration.png)</Figure>

Furthermore, there is a whole platform available. Breaking down the performance per branches, commits, or benchmarks and giving you an overview of the upcoming performance changes. We will give you more details about this in a future blog post!

But the beta is already opened! 🎉

Here are some of the open-source repositories already using CodSpeed:

- [`pydantic-core`](https://github.com/pydantic/pydantic-core): The core validation logic for pydantic, a Python data parsing and validation library.
- [`pathfinding`](https://github.com/samueltardieu/pathfinding): A pathfinding library for Rust.
- [`swarmion`](https://github.com/swarmion/swarmion): A set of tools to build and deploy type-safe serverless microservices with Typescript.

If you want to learn more, [check out the product](https://codspeed.io) and [follow us on Twitter](https://twitter.com/codspeedhq) to stay updated!


The CodSpeed Blog

Using Vitest bench to track performance regressions in your CI

Pinpoint performance regressions with CI-Integrated differential profiling

Why Rust doesn't need a standard div_rem: An LLVM tale

Introducing CodSpeed: Continuous Benchmarking

Not a regression if you catch it before production

The CodSpeed Blog

Using Vitest bench to track performance regressions in your CI

Pinpoint performance regressions with CI-Integrated differential profiling

Why Rust doesn't need a standard div_rem: An LLVM tale

Introducing CodSpeed: Continuous Benchmarking

.css-13o5clh{display:inline;background-image:linear-gradient(to bottom right, #FF8C4A 0%, #FF6661 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Not a regression if you catch it before production

Not a regression if you catch it before production