Skip to main content
Macro runners are bare-metal machines managed by CodSpeed, for walltime measurements with low variance. A CircleCI executor shares its host with other jobs, so the walltime of a benchmark moves from one run to the next for reasons unrelated to your code. A macro runner is dedicated to your job for its whole duration. On CircleCI they are exposed as self-hosted runner resource classes. CodSpeed creates and owns those resource classes in your CircleCI organization, watches them for queued benchmark jobs, and brings up a runner for each one. Your side of it is one token and one resource_class line.
Macro runners on CircleCI are enabled per organization. Contact us to have them enabled on yours.

​
Prerequisites

  • A CodSpeed organization with macro runners enabled. Macro runners are not available on personal accounts.
  • The CircleCI integration set up.
  • A CircleCI user who is an admin of the CircleCI organization connected to your GitHub organization.
  • CircleCI’s self-hosted runner terms accepted for that organization, under Organization Settings in CircleCI. CodSpeed cannot accept them for you.

​
1. Create a CircleCI personal API token

In CircleCI, go to User Settings → Personal API Tokens and create a token. CodSpeed uses it for two things: creating the resource classes your benchmark jobs target, and reading how many benchmark jobs are waiting for a runner. That unclaimed task count is the only signal CircleCI exposes for a job queued on a self-hosted resource class, and only a personal API token can read it.
A CircleCI personal API token reaches everything its user reaches, in every organization that user belongs to. Create the token with a service user that belongs to your CircleCI organization only, rather than with a personal account.
CircleCI personal API tokens expire. Set the expiry to the maximum, one year: CodSpeed lets you know when the token has expired, and the benchmark jobs queued meanwhile wait in CircleCI until you save a new one.

​
2. Save the token in CodSpeed

In your organization settings, open the Capabilities tab, paste the token in the CircleCI personal API token card and save. On save, CodSpeed creates a runner namespace for your CircleCI organization if it does not have one yet, then one resource class per available runner.

​
3. Available Macro Runners on CircleCI

<namespace> is your organization name, lowercased. A CircleCI organization owns a single namespace, so an organization that already has one, for example from publishing an orb, keeps it and the resource classes are created in it.

​
4. Point the benchmark job at the resource class

Copy the resource class of the runner you want and set it on the benchmark job, with the walltime instrument:
.circleci/codspeed.yml
A job on a self-hosted resource class runs with machine: true, directly on the machine behind the resource class, which sets the OS and the installed software: docker executors and CircleCI’s convenience images are not available there. Everything else about the job is unchanged, including how it authenticates with CodSpeed.

​
5. Open a pull request

Open a pull request from the branch carrying the change. The benchmark job is queued in CircleCI, CodSpeed hands it a macro runner, and the pull request receives its performance report. The walltime measurements of a run are comparable to the baseline recorded on your default branch only if both ran on macro runners, so merge the change to record a first baseline on a macro runner.

​
Behavior on CircleCI

  • A runner runs one benchmark job at a time. Benchmarks split across several jobs need one runner each, and the jobs that find no free runner stay queued until one frees up.
  • Macro runner minutes are billed to the organization, whichever of its repositories the job belongs to: a resource class is org-scoped in CircleCI, so every repository of the organization can use it. See the pricing of the runner.

​
Troubleshooting

CircleCI queues a job whose resource class has no runner, without failing it, so a job that never starts points at the resource class rather than at the job itself. In order:
  • Check that resource_class matches, character for character, one of the resource classes listed in the CodSpeed settings card.
  • Check that the token is still valid. CircleCI personal API tokens expire, and a revoked or expired token stops CodSpeed from seeing the job: paste a new one in the card and save.
CodSpeed reports the step that failed, followed by CircleCI’s own message. The usual causes are:
  • The token’s user is not a member of a CircleCI organization named after your GitHub organization, or is not one of its admins.
  • The self-hosted runner terms have not been accepted for that organization.
Removing them stops the benchmark jobs targeting those resource classes from running on macro runners. The resource classes themselves are left in place, so the configuration of your pipelines stays valid and their benchmark jobs queue, until you save a token again or point them at another executor.

​
Next Steps

Macro Runners

The runners, and what they are for

Walltime Instrument

Measure and interpret walltime benchmarks

Configure CircleCI

Authentication, parallel jobs and advanced options

Performance Checks

Catch regressions on every pull request