
The Rust 1.78.0 version was released on May 2, 2024. The [release announcement](https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html#compatibility-notes) mentions that the bundled LLVM version is upgraded to 18, completing the announced `u128`/`i128` [change](https://blog.rust-lang.org/2024/03/30/i128-layout-update.html) for x86 architectures. It is mentioned that if your code uses 128-bit integers, **you may notice runtime performance improvement**. Let's dive into how this can happen.

### The alignment issue with `u128`/`i128`

> The _alignment_ of a value specifies what addresses are valid to store the value at. A value of alignment `n` must only be stored at an address that is a multiple of `n` (from the [the Rust Reference](https://doc.rust-lang.org/reference/type-layout.html#size-and-alignment))

<Figure maxW="600px" mx="auto">
  ![How alignment works](alignment.excalidraw.png)
</Figure>

Normally, the memory size of a value should always be a multiple of its alignment to prevent misalignment issues.

However, on x86 architectures, there is an inconsistency with the alignment of 128-bit integers between rustc (the rust compiler) and C conventions:

<Table maxW="500px">
  <Thead>
    <Tr>
      <Th>Data type</Th>
      <Th>Memory size</Th>
      <Th>Alignment</Th>
    </Tr>
  </Thead>
  <Tbody>
    <Tr>
      <Td>i128 (rustc 1.77.0)</Td>
      <Td>16 bytes</Td>
      <Td>8 bytes</Td>
    </Tr>
    <Tr>
      <Td>__int128 (clang 17.0.1)</Td>
      <Td>16 bytes</Td>
      <Td>16 bytes</Td>
    </Tr>
  </Tbody>
</Table>

In Rust < 1.77, 128-bit integers are **8-byte aligned,** whereas the corresponding C types are **16-byte aligned**. But they all have a size of **16 bytes**.

Let's see how this misalignment can degrade performance.

### Why memory misalignment can lead to worse performance?

Having a memory size greater than its alignment can lead to cases where the value is stored at an address right at the edge of a memory cache line (which is usually **64 bytes**), which will cause the value to be stored on **two cache lines instead of one**.

Let's create an example where this happens with Rust 1.76.0 on an x86_64 machine. Let's look at the following code:

```rust
use type_layout::TypeLayout;

const GAP: usize = 128;

#[derive(TypeLayout, Copy, Clone)]
#[repr(C)]
struct StockAlignment {
    _offset: [u8; GAP - 8], // 1-byte aligned
    data: u128,             // 8-byte aligned
}

pub fn main() {
    println!("Alignment of u128: {}", align_of::<u128>());
    println!("{}", StockAlignment::type_layout());
}
```

Here, we created a `StockAlignment` struct that contains an array `_offset` of `u8` and an `u128` integer `data`. We use the [`type_layout` crate](https://docs.rs/type-layout/latest/type_layout/) to be able to see what the size and alignment of the struct will be.

After we execute the code, we have the following:

```text
Alignment of u128: 8
StockAlignment (size 136, alignment 8)
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | data      | 16   |
```

That means that an instance of `StockAlignment` will span three cache lines of 64 bytes, like so:

<Figure maxW="500px" mx="auto">

![Stock alignment with Rust 1.76.0](stock-alignment-1.76.excalidraw.png)

  <Caption>Each square represents 8 bytes</Caption>
</Figure>

The misalignment problem is visible here. When `data` is accessed, **two cache lines of 64 bytes are read** from memory. This is effectively loading twice as much data as needed, since `data` is only 16 bytes, a **single cache line could have been read** if it was properly aligned.

Let's create a new struct where we enforce `data` and its containing struct to be correctly (16-bit) aligned. With the proper alignment, this is what we expect the memory layout will look like:

<Figure maxW="500px" mx="auto">

![Enforced alignment with Rust 1.76.0](enforced-alignment.excalidraw.png)

  <Caption>Enforced alignment with Rust 1.76.0</Caption>
</Figure>

With that alignment, accessing `data` will only load a **single cache line**. To achieve it, we can write the following code:

```rust
use type_layout::TypeLayout;

const GAP: usize = 128;

#[derive(Copy, Clone)]
#[repr(C, align(16))]
struct AlignedU128(u128);

#[derive(TypeLayout, Copy, Clone)]
#[repr(C)]
struct EnforcedAlignment {
    _offset: [u8; GAP - 8],
    data: AlignedU128,
}

pub fn main() {
    println!("{}", EnforcedAlignment::type_layout());
}
```

Here we created a struct `AlignedU128`, that only stores a single `u128` and enforce it to be 16-byte aligned. We then use this as the type of `data` in a new `EnforcedAlignment` struct.

After we execute the code, we have the following:

```text
EnforcedAlignment (size 144, alignment 16)
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | [padding] | 8    |
| 128    | data      | 16   |
```

So we optimized `EnforcedAlignment` to access `data` more efficiently, but it comes at a cost in memory since it costs 8 more bytes than `StockAlignment`.

### Comparing the performance between the two alignments

To measure the performance of loading `data` with different alignments, we created performance tests with the [criterion.rs benchmarking library](https://github.com/bheisler/criterion.rs).

Before running the benchmarks, we allocate two arrays of `8192` elements, one filled with `StockAlignment` instances, and the other one with `EnforcedAlignement`. We then create two benchmarks where we access the `data` property of all the elements of the array:

```rust
let c: &mut Criterion;
let stock_align: Vec<StockAlignment>;

c.bench_function("stock align bench", |b| {
    b.iter_batched(
        cache_fuzzer,
        |_| {
            let mut sum = 0;
            for i in 0..N {
                sum += stock_aligned[i].data;
            }
            sum
        },
        criterion::BatchSize::PerIteration,
    );
});
```

For the other struct, we have the same benchmark code.

<Alert>
  The [`iter_batched`
  function](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html#iter_batchediter_batched_ref)
  allows to pass a setup function as the first argument, with the actual
  benchmark being the second argument. The `cache_fuzzer` function is used to
  clear the cache before each benchmark to avoid the effect of cache hits on the
  benchmark results. This ensures that previous iterations have close to no
  impact on the current one.
</Alert>

To have stable and reproducible results, we use the **CodSpeed runner, providing a consistent execution environment**.

<PerfDiff
  base={280.4e-6}
  head={250.5e-6}
  name="ordered access"
  description="Changing from stock to enforced alignment"
/>

This notable **performance improvement when using the properly aligned struct** confirms our initial idea: optimizing the alignment effectively **reduces the number of cache misses** and thus RAM accesses, ending up with a **positive impact on performance**.

Here we only tested when accessing elements of the array in an ordered way. Under the hood, there could be some optimizations loading more memory than requested from contiguous blocks in our measurements. For example, loading more memory than requested from contiguous blocks. To test in different situations, we added two more groups of benchmarks:

- `reversed access`, to simulate access in a reversed order
- `randomized access`, to simulate access in a randomized order

We get the following results:

<PerfDiff
  base={279.8e-6}
  head={249.9e-6}
  name="reversed access"
  description="Changing from stock to enforced alignment"
/>

<PerfDiff
  base={241.5e-6}
  head={219.4e-6}
  name="randomized access"
  description="Changing from stock to enforced alignment"
/>

The performance gains are consistent with the first we observed and thus the order in which we access the data does not affect the performance much.

<Alert>
  Using the **CodSpeed runner will execute the benchmark code on a virtualized
  CPU** in a controlled CI environment, measuring the CPU instructions and
  turning them back into a human-readable time measurement. Learn more about the
  [CodSpeed performance metric on our
  docs](https://docs.codspeed.io/features/understanding-the-metrics/#execution-speed-measurement).
</Alert>

### Real-world check with Rust 1.78.0

Now that we identified and shown the problem with Rust 1.76.0, let's bump Rust to 1.78.0 and see what happens.

```diff
// rust-toolchain.toml
[toolchain]
-channel = "1.76.0"
+channel = "1.78.0"
```

Let's check the alignment of our previous misaligned struct:

```text
Alignment of u128: 16                   <-- was 8 before
StockAlignment (size 144, alignment 16) <-- size was 136 before, alignment was 8
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | [padding] | 8    |           <-- new
| 128    | data      | 16   |           <-- properly aligned now
```

The inconsistency has indeed been solved. The performance of `stock align` increased by **10%** and is now equal to `enforced align` since the alignments are the same.

<Figure maxW="400px" mx="auto">

[![CodSpeed report of the bump to 1.78](rust-align-128-1.78.png)](https://codspeed.io/art049/rust-align-128/branches/upgrade-rust-1-78)

  <Caption>Click on the image to check out the performance report of the Rust bump to 1.78.0</Caption>
</Figure>

### A note on 1.77

Although it was not announced in the [release blog post](https://blog.rust-lang.org/2024/03/21/Rust-1.77.0.html), the [128-bit integers misalignment fix](https://github.com/rust-lang/rust/pull/116672/) was already included as a part of the [1.77.0 release](https://github.com/rust-lang/rust/releases/tag/1.77.0). The reason it was announced later is that there was a related bug linked to 128-bit integers arguments which was only solved with the [update to LLVM 18](https://github.com/rust-lang/rust/pull/120055/) completed with the 1.78.0 release.

[Check out the CodSpeed report of the pull request](https://codspeed.io/art049/rust-align-128/branches/upgrade-rust), where the performance gain is already visible.

### Performance gains from bumping to 1.78.0 in the ecosystem

We have seen a number of repositories gaining performance increases when upgrading the toolchain from **1.77.x** **to 1.78.0**, with **performance gains up to 21%** depending on the benchmark. Those performance changes are not solely due to the alignment fix, but most probably related to optimizations released with the new LLVM version.

<Figure maxW="300px" mx="auto" my={4}>

[![CodSpeed report of the bump of rust to 1.78.0 on oxc](oxc-rust-1.78.png)](https://codspeed.io/oxc-project/oxc/branches/renovate/rust-toolchain)

  <Caption>oxc-project/oxc</Caption>
</Figure>

<Figure maxW="300px" mx="auto" mt={0} mb={4}>

[![CodSpeed report of the bump of rust to 1.78.0 on uv](uv-rust-1.78.png)](https://codspeed.io/astral-sh/uv/branches/samypr100:rust-update-178)

  <Caption>astral-sh/uv</Caption>
</Figure>

### Takeaways

Making sure our data structures have **a proper memory alignment** does **improves performance by reducing cache misses** and **making memory accesses faster**. However, it comes with a **higher memory footprint** implied by the padding added within our memory layouts.

These small performance changes can be hard and tedious to track, but this is where we can **leverage performance testing continuously in CI environments**, enabling informed optimization decisions.

[Here is the repository](https://github.com/art049/rust-align-128) containing the code shown in this article and you can also find its [CodSpeed performance dashboard here](https://codspeed.io/art049/rust-align-128).

### Resources

- [Rust 1.78.0 release blog](https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html#compatibility-notes)
- [u128/i128 layout update blog](https://blog.rust-lang.org/2024/03/30/i128-layout-update.html)
- [Memory Alignment and Performance](https://fylux.github.io/2017/07/11/Memory_Alignment/)
- [Size and Alignment docs in Rust Reference](https://doc.rust-lang.org/reference/type-layout.html#size-and-alignment)
- [type_layout crate](https://docs.rs/type-layout/latest/type_layout/)
- [Repository containing the code shown in the blog](https://github.com/art049/rust-align-128)
- [Getting started with Rust on CodSpeed](https://docs.codspeed.io/benchmarks/rust)



While unit tests have become a standard for ensuring that the code performs as expected, they focus on checking the logic, not the performance. Yet, **performance regressions can be just as dangerous as functional bugs, putting your whole software at risk**. This is why performance should be checked early while testing the code, the icing on the cake being to check it in CI environments.

## Effortless performance testing with pytest

**Benchmarking** is a way of testing a block of code's performance. It is like a test case but for performance. It will execute the code and measure how long it takes to run.

Let's see how to implement that with `pytest`. First, install the `pytest-codspeed` library to enable the benchmark marker and fixture:

```text
pip install pytest-codspeed
```

Then, you're ready to use the `@pytest.mark.benchmark` marker for measuring performance. You can use it directly on a single test:

```diff
+ import pytest
+ @pytest.mark.benchmark
  def test_my_fn():
      inputs = gen_inputs()
      results = my_fn(inputs)
      assert results == "expected_result"
```

But you can also apply it at the module level by using the global `pytestmark` variable, effectively enabling performance testing on all the tests contained within it:

```python
import pytest
pytestmark = pytest.mark.benchmark

# The rest of the test cases are now performance tests as well
```

You can then run the performance tests locally to ensure that everything works:

```bash
pytest tests/ --codspeed
```

## Rely on your CI to avoid Spray and Pray

While being a good starting point, isolated runs do not work for long-term performance tracking and the risk of missing important performance changes is just too high. Regressions will surface when you less expect them and it's essential to **automate these checks into your CI pipeline**. This ensures that any performance degradation is caught automatically during the development cycle and builds a history of your codebase performance.

Using the CodSpeed test runner helps a lot to make the measurement extremely steady. Our runner relies on CPU simulation, enabling us to separate the noisy neighbours (other VMs, workloads, users) from the precious workload you want to measure.

A typical setup with the runner in GitHub Actions would be as simple as:

```yaml
- uses: CodSpeedHQ/action@v3
  with:
    run: pytest tests/ --codspeed
```

This setup not only runs your tests but also uploads the results to CodSpeed, where you can track performance over time.

<Figure maxW="600px" mx="auto">

![Sample pull request report on CodSpeed](sample-pull-request-report.png)

  <Caption>Sample pull request report on CodSpeed</Caption>
</Figure>

## Measuring only what matters

Sometimes, you want more granularity in what is measured. For example, you do not want to measure the time it took to generate the inputs for calling our function or the assertions after getting the result; and instead **focus only on the actual function call**.

We can modify the unit test:

```python
def test_my_fn(benchmark):
    inputs = gen_inputs()
    results = benchmark(my_fn, inputs)
    assert results == "expected_result"
```

This test uses the `benchmark` fixture to only measure the execution time of `my_fn`. The fixture makes it easy to focus on what matters—how long it takes your function to execute under test conditions.

<Alert>
  Using the `benchmark` fixture will automatically mark the test as a benchmark,
  without having to use the `pytest.mark.benchmark` marker.
</Alert>

## Optimizing hot paths with Differential Profiling

When you encounter a performance regression, your next mission(if you accept it🤵) is often to investigate and find what/who/when/why was this issue introduced in the first place.

This is where differential profiling comes in handy. This allows to compare two execution profiles to find exactly what changed between two separate measurements.

The good news is **CodSpeed automatically profiles your benchmark's code while measuring performance**. So if you spot a regression, you'll have all the data to investigate:

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-complex" />
  <Caption>
    Sample flame graph with regressions, improvements, and added code
  </Caption>
</Figure>

## Takeaways

Integrating performance testing into your development process with tools like `pytest` and CodSpeed fosters a culture of continuous improvement. It ensures that performance considerations are never an afterthought but a key component of your software development lifecycle from the ground up.

To see CodSpeed in action, you can check out open-source repositories using the tool in the [explore page](https://codspeed.io/explore). A lot of them are actually using the `pytest` integration we just talked about, like [pydantic](https://codspeed.io/pydantic/pydantic-core) and [polars](https://codspeed.io/pola-rs/polars)

Last but certainly not least, shout out to [patrick91](https://twitter.com/patrick91), who pioneered this use case and whose contributions have made it significantly easier for developers to incorporate benchmarking into their existing unit tests.

## Resources

- [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed): the plugin for `pytest`
- [@codspeed/action](https://github.com/CodSpeedHQ/action): the GitHub Action to run benchmarks and generate flame graphs
- [@codspeed/runner](https://github.com/CodSpeedHQ/runner): the generic CodSpeed runner working with various CI providers
- [CodSpeed Docs: Python Integration](https://docs.codspeed.io/benchmarks/python)
- [Pinpoint performance regressions with CI-Integrated differential profiling](https://codspeed.io/blog/pinpoint-performance-regressions-with-ci-integrated-differential-profiling)



One of the easiest ways to track performance regressions in your JavaScript applications is to use Vitest's `bench` API.
It lets you write benchmarks as easily as unit tests:

```ts
import { bench, describe, test } from "vitest";

describe("fibonacci", () => {
  test("should return 55 when called with 10", () => {
    expect(fibonacci(10)).toBe(55);
  });

  bench("fibo 10", () => {
    fibonacci(10);
  });
});
```

Then you can run the benchmarks with `vitest bench`:

```bash
pnpm vitest bench
```

The problem that we now face is that the results are **inconsistent between runs**. Running the command multiple times in a CI/CD runner or on your machine will yield different results.
Thus it will be hard to spot performance regressions with confidence.

This is where using CodSpeed comes in handy. CodSpeed provides a way to **run benchmarks in a consistent manner**, and thus **detect performance regressions with confidence**.

We are excited to announce that CodSpeed now has first-class support for Vitest!

## How to use Vitest with CodSpeed

Install the `@codspeed/vitest-plugin` package and `vitest` as dev dependencies:

```bash
pnpm add -D @codspeed/vitest-plugin vitest
```

<Alert status="info">
  The CodSpeed plugin is only compatible with
  [vitest@1.0.0](https://www.npmjs.com/package/vitest/v/1.0.0) and above.
</Alert>

Create or update the `vitest.config.ts` file at the root of your project:

```ts title="vitest.config.ts"
import { defineConfig } from "vitest/config";
import codspeedPlugin from "@codspeed/vitest-plugin";

export default defineConfig({
  plugins: [codspeedPlugin()],
  // ...
});
```

<Alert>
  If you are on a project using `vite` and `vitest`, you can also add the plugin
  to the `vite.config.ts` file.
</Alert>

Let's create a fibonacci function and create some benchmarks on it.
Create a new file `src/fibonacci.ts`:

```ts
export function fibonacci(n: number): number {
  if (n <= 1) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
}
```

Create a new file `src/fibonacci.bench.ts`:

```ts
import { bench, describe } from "vitest";
import { fibonacci } from "./fibonacci";

describe("fibonacci", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });

  bench("fibo 15", () => {
    fibonacci(15);
  });
});
```

Now we can run the benchmarks with `vitest bench`:

```bash
$ vitest bench

[CodSpeed] bench detected but no instrumentation found, falling back to default vitest runner
 RUN  v1.0.3 ...

 ✓ src/fibonacci.bench.ts (2) 1521ms
   ✓ fibonacci (2) 1520ms
     name                hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · fibo 10   1,760,687.42  0.0005  1.0279  0.0006  0.0006  0.0007  0.0007  0.0016  ±0.50%   880344   fastest
   · fibo 15     173,932.15  0.0052  0.0594  0.0057  0.0059  0.0064  0.0070  0.0147  ±0.08%    86967


 BENCH  Summary

  fibo 10 - src/fibonacci.bench.ts > fibonacci
    10.12x faster than fibo 15
```

We can see that CodSpeed has been detected and that the benchmarks have been run.
For the moment we can see the default output of Vitest's `bench` API.

## Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This allows CodSpeed to detect the CI environment and properly configure the instrumented environment.

### Enable the repository on CodSpeed

The first step is to log in to CodSpeed and import the repository you want to track. You can find more information about this in the [CodSpeed documentation](https://docs.codspeed.io/importing-repositories).

After installing the GitHub app and importing the repository in CodSpeed, you will be prompted a `CODSPEED_TOKEN` secret to copy. This token is used to authenticate the CodSpeed GitHub Actions workflow.

### Add a CodSpeed GitHub Actions workflow

Open a Pull Request on your repo with the previous changes, and the following GitHub Actions workflow `.github/workflows/codspeed.yml`.
It will run the benchmarks and report the results to CodSpeed on every push to the `main` branch and every pull request:

```yaml title=".github/workflows/codspeed.yml"
name: codspeed-benchmarks

on:
  push:
    branches:
      - "main"
  pull_request:

jobs:
  benchmarks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: pnpm/action-setup@v2
      - uses: actions/setup-node@v3
        with:
          cache: pnpm
      - name: Install dependencies
        run: pnpm install
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: pnpm vitest bench
          # token retrieved from the CodSpeed app at the previous step
          token: ${{ secrets.CODSPEED_TOKEN }}
```

If everything went well, the workflow should run and after it finishes you should see a comment from the CodSpeed GitHub app on the Pull Request with the run report.

### Visualizing the benchmarks on the CodSpeed app

By clicking on the CodSpeed comment in the Pull Request, you will be redirected to the CodSpeed report of the PR.
You will see a list of the benchmarks, and when clicking on the `fibo 10` benchmark its flame graph will appear:

<Figure>
  <FlameGraph height={150} flameGraphsPath="fibo10-vitest" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

And that's it for the installation of CodSpeed with Vitest, now we can explore a bit more what CodSpeed offers.

## Visualizing a performance change

One of the main features of CodSpeed is to detect and visualize the performance changes of a pull request.

Let's now say that we are on a more complex project where we have a `parsePr` function that parses a GitHub pull request.
We created a pull request with some performance improvements, and the following is in the CodSpeed report:

<PerfDiff
  base={1.5e-3}
  head={1.2e-3}
  name="parsePr::short body"
  description="Improve parsing performance"
/>
<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-vitest" />
  <Caption>
    The Diff graph shows the resulting graph of the difference between the head
    and base runs, allowing us to pinpoint where the performance changes are
    inside the code
  </Caption>
</Figure>

We observe a 25% improvement in the overall benchmark, driven by enhancements in the `parsePr` and `parseIssueFixed` functions.
Although the `logMetrics` function has experienced regression, its impact on the overall benchmark is less pronounced compared to the observed improvements.
Additionally, a new function, `modifyTitle`, is introduced and denoted by the distinctive blue color.

Thanks to the differential flame graph, it is now really easy to understand at a glance what code changes impact the performance.

## Wrapping up

And there you have it—CodSpeed now supports Vitest to make performance testing a breeze for your JavaScript and TypeScript projects! With an easy-to-use bench API and the magic touch of CodSpeed's consistency, tracking regressions is as simple as adding a unit test.

Get ready to supercharge your CI with performance reports, flame graphs, and more!

---

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles and product updates.

## Resources

- [Vitest integration](https://docs.codspeed.io/benchmarks/nodejs/vitest) in the CodSpeed documentation
- [@codspeed/vitest-plugin](https://github.com/CodSpeedHQ/codspeed-node/tree/main/packages/vitest-plugin) on GitHub
- [Trace Generation](https://docs.codspeed.io/features/trace-generation) in the CodSpeed documentation
- [Vitest bench API](https://vitest.dev/api/#bench)



Flame Graphs are a visualization tool that helps you understand how your software is performing.
They display the call stack of a program and the time spent in each function.
They are a powerful tool to quickly identify performance bottlenecks.

Differential Flame Graphs combine two flame graphs to highlight the differences between them. They allow you
to easily spot performance regressions and improvements and gain invaluable insights into your software's performance.

Below you can find an example of normal and differential flame graphs following a change in a codebase.
At a glance, we can understand where the code has become slower. Here it is the `parse_issue_fixed` function as it
has the brightest red color.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

Let's explore how to read flame graphs and how CodSpeed automates flame graph generation in your CI pipeline.

<Alert>
  CodSpeed supports Rust, Node.js, Python and generates flame graphs out of the
  box. More languages are coming soon!
</Alert>

## Spotting a performance regression

Let's dive into the previous example.

We have a function `parse_issue_fixed` that parses a GitHub pull request body and extracts the issue
number that it fixes. Given `body = "fixes #123"`, the function returns `123`. Here `body` can be
a multiline string, and the issue number can be anywhere in the string.

<Alert>
  On GitHub, adding the string `fixes #123` in a pull request body will
  automatically close the issue #123 when the pull request is merged.
</Alert>

We notice that the code is quite long and not easily understandable. We refactor it to use a regular expression instead.
This gives us the following diff:

```diff
+import re

+FIXES_REGEX = re.compile(r"fixes #(\d+)")
+
+
 def parse_issue_fixed(body: str) -> int | None:
-    prefix = "fixes #"
-    index = body.find(prefix)
-    if index == -1:
-        return None
-
-    start = index + len(prefix)
-    end = start
-    while end < len(body) and body[end].isdigit():
-        end += 1
-    return int(body[start:end])
+    match = FIXES_REGEX.search(body)
+    return int(match.group(1)) if match else None
```

Great, we have a much shorter and more readable function. But did we introduce a performance regression?

We already have a benchmark in place for the function `parse_pr`, a higher-level function that
parses a GitHub pull request body and calls multiple functions to retrieve information,
including `parse_issue_fixed`. The input is a multiline string of approximately 50kB.
We chose a large input to better understand the performance characteristics of the function and make sure the function performs well when parsing large pull request bodies.

Let's check out the flame graphs for the benchmark to analyze the performance impact of our change.
The **Base** flame graph is the one before the change, the **Head** flame graph is the one after
the change, and finally, the **Diff** flame graph is the difference between the two.

<PerfDiff
  base={1.2e-3}
  head={1.525e-3}
  name="test_parse_pr"
  description="Refactor with regular expression"
/>
<div style={{ marginBottom: "16px" }}>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
</div>

Here we can see that the `parse_issue_fixed` function is bright red, thus denoting that it
was slower after the change and had the biggest performance impact on the benchmark.
When hovering over the `parse_issue_fixed` frame, we can see that the performance
impact is a _-21.5% regression_ on the overall benchmark.

So in our use case, using a regular expression will result in slower executions compared
to using `str.find`.

So yes, our change for the sake of readability introduced a performance regression.
We can now make an informed decision on whether we want to keep the change or not.

Now that we understand how a differential flame graph works and how useful it is,
let's explore how to generate one.

## Generating flame graphs manually is unreliable and time-consuming

The first thing to note is that manually generated flame graphs on a dev machine are **not consistent**.

Running the same script multiple times with the same input will result in different flame graphs.
Indeed, other processes running on the machine will impact the execution time of the script and its different functions.
This make it hard to spot performance regressions with confidence.

The second thing to note is that generating flame graphs manually is **tedious**.

The first tool to generate flame graphs is the eponym FlameGraph by Brendan Gregg written in Perl.
Since then, many other tools have been created in and for other languages. The general steps are the same:

- Run your script with a profiler and output the profile in a file
- Run the tool on the profile to generate a flame graph svg
- Open the generated svg in your browser and explore the data

This is already a quite heavy process: you have to create a script to isolate the part of your program
you want to generate a flame graph for, then run a bunch of commands and finally open
a file in your browser.

This is why we decided to automate the generation of flame graphs in your CI pipeline with CodSpeed,
and find a way to make sure that the flame graphs are consistent.

## Automating flame graphs with CodSpeed

Since we already had wrappers of benchmarking libraries in different languages, we decided to augment them with
profiling capabilities.

### Steps to follow to get flame graphs in your CI pipeline

Write benchmarks with one of our wrappers in Rust, Node.js, or Python.

For example, in Python, use our `pytest` extension `pytest-codspeed` (which API is compatible with `pytest-benchmark`)
and add a test marked as a benchmark:

```python
import pytest
from parse_pr import parse_pr

@pytest.mark.benchmark
def test_parse_pr():
    parse_pr(body=very_long_body_string, pr_number=126, title="Refactor some code")
```

Run the benchmarks in the `@codspeed/action` GitHub Action:

```yml
- name: Run benchmarks
  uses: CodSpeedHQ/action@v3
  with:
    run: pytest tests/ --codspeed
```

The action will run the benchmarks on a "virtualized" CPU with Valgrind.
Instead of measuring the execution time, it measures the number of CPU cycles and memory accesses.
Each benchmark is only ran once and it is enough to get **consistent data**.

This data is then sent to CodSpeed's servers and flame graphs are generated.
A comment is added to the GitHub Pull Request, with a link to the CodSpeed app, where you can browse
the benchmarks and their flame graphs.

And voilà! You now have flame graphs in your CI pipeline, with the benefits of consistency and automation.
Moreover, now that are using CodSpeed, **performance regressions will automatically be detected and reported**
in your pull requests 🎉

<Alert>
  Head out to our [documentation](https://docs.codspeed.io) to view integrations
  for the different languages.
</Alert>

## Going further

Flame graphs are not only useful for spotting performance regressions but they can also be used to
understand the performance impact of a change. Being an improvement or new calls to a function.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-complex" />
  <Caption>
    A more complex flame graph with regressions, improvements, and added code
  </Caption>
</Figure>

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles.

## Resources

- [Trace Generation](https://docs.codspeed.io/features/trace-generation) in the CodSpeed documentation
- [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed), plugin for pytest
- [@codspeed/action](https://github.com/CodSpeedHQ/action), GitHub Action to run benchmarks and generate flame graphs
- [Flame Graphs](https://www.brendangregg.com/flamegraphs.html) by Brendan Gregg
- [Differential Flame Graphs](https://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html) by Brendan Gregg
- [Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
- [Valgrind](https://valgrind.org/)
- [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark)



During the last advent of code, while starting with Rust, I discovered that the Rust standard library was missing the usual `div_mod`(a.k.a. `div_rem`) function that would simultaneously return the quotient and remainder. I got confused since I already used the operator in C++ with [std::div](https://en.cppreference.com/w/cpp/numeric/math/div) or Python with [divmod](https://docs.python.org/3/library/functions.html#divmod). It seemed a basic way of improving performance since both values would come out with the same `DIV` assembly instruction, nicely avoiding extra CPU cycles.

Looking it up, I found [an existing RFC](https://github.com/rust-lang/rfcs/issues/1887) created in 2017, but it hasn't received much attention since.

So, do we need a `div_rem` implementation in the standard library? I decided to investigate.

## A naive approach

Let's naively implement the function:

```rust
pub fn naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    (a / b, a % b)
}
```

To dissect what is executed we're gonna analyze the compilation steps.

<Figure>

![Rust Compilation Chain](compilation-chain.excalidraw.png)

  <Caption>The compilation chain of our Rust program</Caption>
</Figure>

First, let's have a look at the LLVM Intermediate Representation(IR). If you're not
familiar with it, is an extremely low-level programming language close to assembly but
without being tied to specific hardware. It's extremely useful for compiler writers
since it's a common language that can be used to target multiple architectures. It's
also a great tool to understand what the compiler is doing and how it "interpret" our
code.

For the previous code, here is the annotated LLVM IR we can obtain:

```llvm
; Define a function called 'example::naive_div_rem
define { i32, i32 } @example::naive_div_rem(i32 %a, i32 %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for our return tuple
  %_4 = icmp eq i32 %b, 0; Check if divisor %b is zero
  br i1 %_4, label %panic, label %bb1; If %b is zero, panic, else continue
bb1:
  %_3 = udiv i32 %a, %b; Perform unsigned division(only get the quotient)
  %_6 = icmp eq i32 %b, 0; Check again for the divisor being zero (for the modulo)
  br i1 %_6, label %panic1, label %bb2; If %b is zero, panic, else continue
bb2:
  %_5 = urem i32 %a, %b; Compute the unsigned remainder
  ; ...
  ; Stores quotient and the remainder in a tuple, load them back,
  ; and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %8
; ... omitted panic handlers ...
}
```

We can witness that this code contains redundant checks for the division and modulo by 0. Also, we can note two operations to get our quotient and remainder: `udiv` and
`urem`. This is expected at this stage since LLVM doesn't have an instruction combining
them but hopefully, the compiler will be able to optimize this while generating the assembly.

Let's have a look at the next compilation step so we can dive into the assembly code generated from the IR(`x86_64` with optimizations enabled):

```asm
example::naive_div_rem:
    test    esi, esi  ; Test if esi (divisor) is zero
    je      .LBB0_2   ; Jump to panic block if esi is zero
    mov     eax, edi  ; Move the dividend into eax
    xor     edx, edx  ; Zero out edx to prepare for the division
    div     esi       ; Unsigned division, quotient goes into eax, remainder into edx
    ret
.LBB0_2:
    ; ... "attempt to divide by zero" panic handler ...
```

And voila! We can see that the compiler was able to optimize our code and fused the two
operations into a single `div` instruction. So having a dedicated implementation
wouldn't be that helpful for performance while using `rustc` since this optimization
should almost always be done by the IR compiler.

You might have noticed quite a few things in the assembly code, let's have a look at them by creating our own assembly version.

## An inline assembly version

```rust
use std::arch::asm;

pub fn asm_div_rem(a: u32, b: u32) -> (u32, u32) {
    let mut tmp: u32 = a;
    let mut remainder: u32 = 0;
    unsafe {
        asm!(
            "div {divisor}",
            inout("eax") tmp,
            inout("edx") remainder,
            divisor = in(reg) b,
            options(pure, nomem, nostack),
        );
    }
    (tmp, remainder)
}
```

A few things to note in this implementation:

- The `div` instruction divides the `eax` register by its operand value. It will store the quotient in `eax` and the remainder in `edx`.
- `tmp` holds the dividend before our assembly is executed. It will contain the quotient after. This is because its value is tied to the `eax` register and it will be used both for input and output thanks to the `inout` operand.
- `remainder` is also passed with the `inout` operand, allowing us to clear the `edx` register before the operation is done since `remainder` is 0 before the assembly is executed. At the end, the register will hold the remainder value that we return in the output tuple.
- The divisor is passed through any general purpose register as a simple input value that we pass directly to the `div` instruction.
- Finally, we pass a few flags that will help the compiler to know how our inline assembly will behave with `options`. Here: our block has no side effect, we're neither reading nor writing to the memory and we don't push anything to the stack.

Compiling this, we can see that it ends up a bit more concise than the previous compiled assembly we built:

```asm
example::asm_div_rem:
    mov     eax, edi ; Move the value of the dividend from EDI to EAX.
    xor     edx, edx ; Zero out EDX to prepare for the division.
    div     rsi      ; Signed divide the dividend in EAX by the value in RSI('b').
    ret              ; Return from the function. quotient -> EAX, remainder -> EDX.
```

It should be faster but we're cheating a bit on rust here
since we're not taking into account the panic check that was done in the previous version.

<PerfDiff
  base={32.7e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to an inline assembly version"
/>

It's faster! The overhead of those panic checks is not that dramatic in the end. Branch
prediction is probably contributing a lot to this. It was eluded in the first IR
snippets but in there, the rust compiler hints the rest of the pipeline that those
checks are likely to succeed. The compiler's ability to optimize and hint that these
checks are more likely to pass than fail can lead to surprisingly efficient code, even
when it appears to be doing "much more" than the assembly version.

## The cost of panic handling: an unchecked version

Now that we built our unchecked version in assembly, we can get quite close to it by using unchecked div and rem from `std::intrisics`:

```rust
#![feature(core_intrinsics)]
pub unsafe fn unchecked_naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    let quotient = std::intrinsics::unchecked_div(a, b);
    let remainder = std::intrinsics::unchecked_rem(a, b);
    (quotient, remainder)
}
```

And its intermediate representation:

```llvm
; Define a function named 'example::unchecked_naive_div_rem' that takes two arguments
define { i32, i32 } @example::unchecked_naive_div_rem(i32 noundef %a, i32 noundef %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for a tuple to store the quotient and remainder
  %quotient = udiv i32 %a, %b; Perform unsigned division of %a by %b
  %remainder = urem i32 %a, %b; Compute the unsigned remainder of %a / %b
  ; ...
  ; Stores quotient and remainder in a tuple, loads them back, and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %1
}
```

As you can see this IR doesn't contain anything about overflow checks and the performance should be close to or even better than our assembly version.

<PerfDiff
  base={30.5e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to the unchecked version"
/>

That's exactly the same time our assembly version takes. So let's have a look at the assembly:

```asm
example::unchecked_naive_div_rem:
    mov     eax, edi
    xor     edx, edx
    div     esi
    ret
```

And it's the same as our assembly version! So bothering writing our assembly version is not really worth it in the end.

To conclude, there is no performance drawback of not having a single `div_rem` function
since the compiler optimizations will take care of this burden for us. It might seem odd
to write those two operations but unless very specific optimization killer cases they
will be fused as a single assembly instruction.

If you enjoyed this article, you can follow us on [Twitter](https://twitter.com/codspeedhq) to get notified when we publish new articles.

### Resources

- [Compiler Explorer](https://godbolt.org/): an amazing tool to visualize the assembly and IR generated by the compiler
- [Understanding LLVM IR](https://mukulrathi.com/create-your-own-programming-language/llvm-ir-cpp-api-tutorial/#understanding-llvm-ir)
- [The faker's guide to reading (x86) assembly](https://www.timdbg.com/posts/fakers-guide-to-assembly/)
- [Guide to x86 Assembly](https://www.cs.virginia.edu/~evans/cs216/guides/x86.html)



### Why does Continuous Performance analysis matter?

**Performance is the cornerstone of many problems in software development.** Application reactivity, infrastructure costs, energy consumption, and carbon footprint all depend on the performance of the underlying software. **By performance, we often mean execution speed** but more generally, it refers to the efficiency or throughput of a system.

Today, most of the performance monitoring is done through **Application Performance Monitoring** **solutions**(often referred to as APM), provided by companies such as Datadog, Sentry, Blackfire, and many others. These platforms bring **several interesting insights about the production environment's health**: client-side UX monitoring, endpoint latency checks, and even continuous production profiling.

However, these solutions are monitoring tools; **they are built to check if everything is okay in production,** and not really to run experiments. They need real users serving as guinea pigs to experience poor performance to be able to report anomalies.

<Figure>

![Performance feedback in the **current** Software Development Life-Cycle](SDLC_Traditional.png)

  <Caption>Performance feedback in the <b>current</b> Software Development Life-Cycle</Caption>
</Figure>

So as a developer, to understand the performance impact of my changes, I need to wait for end users to try out my changes in production!? Then, if something is wrong, maybe I’ll try to improve it, someday…

In an ideal world, **performance checks should be included way earlier in the development lifecycle**; just as an additional testing flavor, **nurturing continuous improvement**.

<Figure>

![Performance feedback in an **ideal** Software Development Life-Cycle](SDLC_Ideal.png)

  <Caption>
    Performance feedback in an <b>ideal</b> Software Development Life-Cycle
  </Caption>
</Figure>

This much shorter feedback loop would **provide visibility with consistent metrics to the teams while they are building** and not once everything is already shipped in production environments. Besides, **guesstimating performance is hard and often plain wrong**. Accurate performance reports help in those cases and can serve as an educational tool for software developers.

### Building a consistent performance metric

Measuring software performance in various environments to gather reproducible results is hard. The most basic metric we can think of is to **measure the execution time but just changing the hardware will produce completely different data points**. Running a program on a toaster will be significantly slower than running it on the latest generations of cloud instances. Joking aside, **merely** **using the same machine at different times will produce different results** because other unpredictable background tasks are eating up the CPU time.

One obvious solution to bring consistency in the results is to run the program in a controlled cloud environment where background running processes are very limited. Despite a significant improvement in the quality of the results, **this doesn’t give repeatable measurements either because of the [noisy neighbor issue](https://www.techtarget.com/searchcloudcomputing/definition/noisy-neighbor-cloud-computing-performance)**. Basically, physical cloud machines are shared among customers by splitting them into multiple Virtual Machines. At a software level, those are perfectly isolated but in the end, they still share some common hardware and **isolation can’t be perfect**(eg. memory, high-level CPU caches, network interfaces). Since it’s not possible to predict the workload running along a measurement task, finding the “truth” in this noise becomes a statistical challenge, and **building an extremely accurate metric is nearly impossible**.

<Figure>
  ![Time measurement for a Fibonacci sequence computation(Python runs from
  GitHub Action)](TimeMeasurementFibo.png)
  <Caption>
    Time measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)
  </Caption>
</Figure>

When it comes to consistency, working with execution time measurement seems doomed to failure. So what if we instead decided to dissect exactly what a virtual machine does to run our program? In this case, running the same measurement again and again on various machines produces the exact same results since **the hosted virtual machine always starts with a predefined state and emulates the same hardware**. We don’t mind the noisy neighbor issue either because **whatever is running along our measurement doesn’t change the instructions executed to run our program** in its sandboxed environment. Furthermore, based on those micro execution steps, it is possible to aggregate **a time equivalent metric that will be consistent, accurate (detect less than 1% performance changes), and hardware agnostic**: **this is how CodSpeed works!**

<Figure>

![CodSpeed Measurement for a Fibonacci sequence computation(Python runs from GitHub Action)](CodSpeedMeasurementFibo.png)

  <Caption>
    CodSpeed Measurement for a Fibonacci sequence computation(Python runs from GitHub Action)
  </Caption>
</Figure>

### Shifting left with CodSpeed

CodSpeed brings this consistent measurement to the Continuous Integration environments, enabling performance checks to be included in the software development lifecycle as early as possible.

On every new feature, the performance is measured and reported directly in the repository provider as a Pull Request comment. Optionally, status checks can also be enabled to enforce performance requirements to be satisfied before merging the new delivery.

<Figure>![Github Integration.png](Github_Integration.png)</Figure>

Furthermore, there is a whole platform available. Breaking down the performance per branches, commits, or benchmarks and giving you an overview of the upcoming performance changes. We will give you more details about this in a future blog post!

But the beta is already opened! 🎉

Here are some of the open-source repositories already using CodSpeed:

- [`pydantic-core`](https://github.com/pydantic/pydantic-core): The core validation logic for pydantic, a Python data parsing and validation library.
- [`pathfinding`](https://github.com/samueltardieu/pathfinding): A pathfinding library for Rust.
- [`swarmion`](https://github.com/swarmion/swarmion): A set of tools to build and deploy type-safe serverless microservices with Typescript.

If you want to learn more, [check out the product](https://codspeed.io) and [follow us on Twitter](https://twitter.com/codspeedhq) to stay updated!


The CodSpeed Blog

Rust 1.78: Performance Impact of the 128-bit Memory Alignment Fix

One pytest Marker to Track the Performance of Your Tests

Using Vitest bench to track performance regressions in your CI

Pinpoint performance regressions with CI-Integrated differential profiling

Why Rust doesn't need a standard div_rem: An LLVM tale

Introducing CodSpeed: Continuous Benchmarking

Not a regression if you catch it before production