Benchmark performance impact
The performance impact denotes an improvement or regression in performance of a benchmark. It is calculated by comparing the benchmark time measurement of the head commit with the time measurement of the base commit. A negative performance metric means that the benchmark is slower than the previous commit. The closer its value is to-1, the slower it is.
A positive performance metric means
that the benchmark is faster than the previous commit. Its value can go up to
to denote massive speed improvements.
Naturally, when the benchmark is as fast as the previous commit, the performance
metric is .
Regression threshold
On the settings page of a project, you can set a threshold for a regression to be considered a regression. By default, this value is set to10% (which is
equivalent to 0.1). The value can be set from 0% to 50% by an admin of the
project. More information about setting the threshold can be found in the
customization documentation.
Commit performance impact
To get the overall performance impact of a commit, we aggregate all the benchmark impacts.n being the number of benchmarks.
Regression threshold exceeded
If there is a regression above the threshold, the overall commit impact will be the biggest regression impact.For example, with impacts:[0.1, 0, -0.3]and a threshold of0.25, the overall commit impact will be-0.3.
Improvement threshold exceeded
If there is an improvement above the threshold, the overall commit impact will be the maximum improvement impact.For example, with impacts:[0.1, 0.3, -0.2]and a threshold of0.25, the overall commit impact will be0.3.
No threshold exceeded
Finally, in the remaining cases, a geometric mean is calculated from all the benchmark performance impacts.For example, with impacts:[0.1, 0.3, -0.2]and a threshold of0.5, the overall commit impact will be approximately0.0459.
A geometric mean will give more relevant results than an arithmetic mean for
this kind of measure as it will be less sensitive to outliers.
Baseline report selection
To create a performance impact, we need to compare the execution speed of the benchmarks against a baseline of those benchmarks’ execution speed. Depending on the context of the run, the baseline report can be different.Pull Request
When triggering a CodSpeed run on a pull request between ahead branch and a
base branch, the baseline report will be the report of the latest commit of
the base branch with a CodSpeed run.
Checked-out commit of a Pull Request in GitHub ActionsBy default, when using
the
action/checkout in GitHub Actions,
the checked-out commit of a pull request will be the merge
commit of the pull request. This means that GitHub will create a
merge commit M between the base branch and the head branch, and use it
as the checked-out commit.
HEAD commit C of the base
branch and the merge commit M. The report will thus only include the changes
of commits B1 and B2.
Every time a new commit is pushed on the
feat-branch, the baseline report will
be updated to the latest commit of the base branch with a CodSpeed run.This pull request base selection algorithm will only work on
pull_request event for GitHub Actions and merge_request_event for GitLab CI.
If you run CodSpeed on a different event, the branch base selection algorithm defined below will be used instead.Branch
When triggering a CodSpeed run following a push on a branch, the baseline report will be the report of the closest commit of the branch.
In this example, a report already exists for the commit B of the
main
branch. A new commit C is pushed on the main branch. The baseline report
for C will be the report of the commit B.Performance impact Gauge
The performance impact gauge is a visual representation of the performance impact, displayed in multiple places of the CodSpeed UI. Some examples of the gauge with their corresponding impact values:
To make it easier to spot regressions and improvements, the mapping between the
actual performance impact value and the gauge is not linear.