Flame Graphs are a visualization tool that helps you understand how your software is performing. They display the call stack of a program and the time spent in each function. They are a powerful tool to quickly identify performance bottlenecks.
Differential Flame Graphs combine two flame graphs to highlight the differences between them. They allow you to easily spot performance regressions and improvements and gain invaluable insights into your software's performance.
Below you can find an example of normal and differential flame graphs following a change in a codebase. At a glance, we can understand where the code has become slower. Here it is the function as it has the brightest red color.
Let's explore how to read flame graphs and how CodSpeed automates flame graph generation in your CI pipeline.
CodSpeed supports Rust, Node.js, Python and generates flame graphs out of the box. More languages are coming soon!
Let's dive into the previous example.
We have a function that parses a GitHub pull request body and extracts the issue number that it fixes. Given , the function returns . Here can be a multiline string, and the issue number can be anywhere in the string.
On GitHub, adding the string in a pull request body will automatically close the issue #123 when the pull request is merged.
We notice that the code is quite long and not easily understandable. We refactor it to use a regular expression instead. This gives us the following diff:
Great, we have a much shorter and more readable function. But did we introduce a performance regression?
We already have a benchmark in place for the function , a higher-level function that parses a GitHub pull request body and calls multiple functions to retrieve information, including . The input is a multiline string of approximately 50kB. We chose a large input to better understand the performance characteristics of the function and make sure the function performs well when parsing large pull request bodies.
Let's check out the flame graphs for the benchmark to analyze the performance impact of our change. The Base flame graph is the one before the change, the Head flame graph is the one after the change, and finally, the Diff flame graph is the difference between the two.
Here we can see that the function is bright red, thus denoting that it was slower after the change and had the biggest performance impact on the benchmark. When hovering over the frame, we can see that the performance impact is a -21.5% regression on the overall benchmark.
So in our use case, using a regular expression will result in slower executions compared to using .
So yes, our change for the sake of readability introduced a performance regression. We can now make an informed decision on whether we want to keep the change or not.
Now that we understand how a differential flame graph works and how useful it is, let's explore how to generate one.
The first thing to note is that manually generated flame graphs on a dev machine are not consistent.
Running the same script multiple times with the same input will result in different flame graphs. Indeed, other processes running on the machine will impact the execution time of the script and its different functions. This make it hard to spot performance regressions with confidence.
The second thing to note is that generating flame graphs manually is tedious.
The first tool to generate flame graphs is the eponym FlameGraph by Brendan Gregg written in Perl. Since then, many other tools have been created in and for other languages. The general steps are the same:
This is already a quite heavy process: you have to create a script to isolate the part of your program you want to generate a flame graph for, then run a bunch of commands and finally open a file in your browser.
This is why we decided to automate the generation of flame graphs in your CI pipeline with CodSpeed, and find a way to make sure that the flame graphs are consistent.
Since we already had wrappers of benchmarking libraries in different languages, we decided to augment them with profiling capabilities.
Write benchmarks with one of our wrappers in Rust, Node.js, or Python.
For example, in Python, use our extension (which API is compatible with ) and add a test marked as a benchmark:
Run the benchmarks in the GitHub Action:
The action will run the benchmarks on a "virtualized" CPU with Valgrind. Instead of measuring the execution time, it measures the number of CPU cycles and memory accesses. Each benchmark is only ran once and it is enough to get consistent data.
This data is then sent to CodSpeed's servers and flame graphs are generated. A comment is added to the GitHub Pull Request, with a link to the CodSpeed app, where you can browse the benchmarks and their flame graphs.
And voilà! You now have flame graphs in your CI pipeline, with the benefits of consistency and automation. Moreover, now that are using CodSpeed, performance regressions will automatically be detected and reported in your pull requests 🎉
Head out to our documentation to view integrations for the different languages.
Flame graphs are not only useful for spotting performance regressions but they can also be used to understand the performance impact of a change. Being an improvement or new calls to a function.
If you enjoyed this article, you can follow us on Twitter to get notified when we publish new articles.