Ignoring a benchmark should be a last resort when the source of flakiness
cannot be immediately addressed. Some reasons to ignore a benchmark include:
The benchmark is I/O bound (network access, file-system access, etc.), leading
to inconsistent results
The benchmarked source code has a non-deterministic behavior (random number
generation, memory allocation, etc.)
The benchmark is consistently flaky for no apparent reason
Remember, the goal of CodSpeed is to provide accurate and reliable
measurements. Ignoring a benchmark should be a temporary measure until the
source of flakiness can be addressed.
An ignored benchmark will be excluded from the overall measure of a report.However, it will still be accessible at the bottom of the report, in the
Ignored section. This allows you to keep track of ignored benchmarks, and keep
access to their
execution profiles and flamegraphs and
re-enable them once the source of flakiness has been addressed.Ignoring or un-ignoring a benchmark will regenerate and potentially modify the
reports of the latest commits of the default branch and branches with opened
pull requests.
Ignoring Permissions: When working with an organization’s repository, only
the admins are allowed to ignore or un-ignore benchmarks.
To ignore a benchmark, follow these steps:
Navigate to the benchmark’s dashboard. You can do this by clicking on the
button in the report or directly through the benchmarks page
Once on the benchmark’s dashboard, click on the Ignore button
Enter a reason for ignoring the benchmark. This information will be used by
CodSpeed to prioritize areas where we can reduce the flakiness of benchmarks.