To reduce CI queue time, first separate waiting from execution. A ten-minute pipeline may spend six minutes eligible but unassigned, two minutes provisioning a runner, and two minutes running. Making tests faster affects only the last interval. Adding runners may not help if jobs are blocked by repository concurrency, require a scarce label, wait for an approval, or arrive in obsolete waves after every push. Queue delay is a system outcome produced by workload shape, scheduling policy, and capacity response.
The goal is not maximum runner utilization. A fleet held near 100 percent leaves no room for bursts and turns small demand changes into long waits. The goal is a documented service level for feedback, such as 95 percent of pull-request critical-path jobs starting within two minutes during working hours, achieved at acceptable cost. Measure by runner pool and job class, because an overall average can hide a saturated macOS or private-network pool behind abundant general Linux capacity.
Build a CI queue measurement model
Capture workflow created, job queued, job eligible, runner assigned, runner ready, job started, job completed, and cancellation timestamps. Provider terminology differs, so define each boundary. Queue time should exclude intentional approval and dependency wait when diagnosing runner capacity, while end-to-end feedback still includes them. Record repository, workflow, job, runner labels, pool, priority, event type, branch, result, attempt, commit, and whether the run was superseded. Use bounded identifiers to avoid telemetry cardinality problems.
| Interval | Meaning | Likely cause | Primary owner |
|---|---|---|---|
| Created to eligible | DAG, approval, or concurrency wait | Serial dependencies or gate policy | Pipeline owner |
| Eligible to assigned | True scheduler queue | No compatible free runner | CI platform |
| Assigned to ready | Provisioning and registration | Image pull, VM boot, autoscaler lag | Runner platform |
| Ready to started | Executor setup | Checkout, container, tool setup | Build engineering |
| Started to completed | Job service time | Task scope, cache, tests | Application and build teams |
Analyze arrivals, service time, and bursts
Chart eligible job arrivals per minute, concurrent demand, service-time percentiles, and queue percentiles over the same clock. Use five- or ten-minute buckets so a morning merge burst is not flattened into a daily average. Distinguish ordinary pull requests, mainline, scheduled security scans, releases, and manual reruns. A long service-time tail pins runners and deserves separate treatment. Calculate offered work roughly as arrivals multiplied by average service time, but use simulation or historical replay for capacity decisions because job eligibility, pools, and heavy-tailed duration violate simple averages.
Inspect synchronized triggers. A commit can launch duplicate workflows, matrix jobs can expand before cheap validation, and scheduled work can start at the top of the hour with every other repository. Move schedules, combine compatible checks, and gate expensive fan-out behind fast failures. DORA's continuous integration guidance emphasizes small batches and frequent integration; reducing commit size and keeping branches short also reduces giant validation waves and the costly rework that queues can magnify.
Remove obsolete and duplicate work
Cancel superseded pull-request runs when only the latest commit matters. GitHub concurrency groups can restrict related runs and cancel in-progress work, but group keys require care: a repository-wide key can serialize unrelated branches, while a key missing the workflow name can make workflows cancel one another. Release, migration, and production deployment jobs may require ordered queuing rather than cancellation. Write semantics for every group instead of applying one organization-wide snippet.
Use affected analysis and content-addressed caches to reduce jobs before they enter the queue. Collapse repeated setup into runner images or caches while preserving reproducibility. Avoid splitting a two-minute test into forty jobs if each requires a minute of provisioning; parallelism can increase queue pressure and total cost while barely reducing the critical path. Combine tiny tasks, split only long divisible tasks, and place quick deterministic checks early. Re-run failed shards rather than the entire matrix when independence and reporting support it.
Fix runner routing and pool fragmentation
A job can wait while runners sit idle because no idle runner satisfies its labels, group access, architecture, operating system, network, or size. Build a compatibility matrix from queued jobs to pools and alert on idle-while-eligible-waits. Remove unnecessary labels and standardize common images. Keep genuinely privileged, GPU, mobile, or private-network work isolated, but avoid one bespoke pool per team. Use fallback routing only when the alternate environment is equivalent and does not weaken data or network controls.
| Observed pattern | Likely intervention | Tradeoff | Proof |
|---|---|---|---|
| Old commits consume runners | Cancel superseded runs | No historical result for every commit | Latest feedback improves |
| Idle runners beside queued jobs | Fix labels and access | Less bespoke isolation | Compatibility wait disappears |
| Provisioning dominates | Warm minimum or faster image | Idle cost or image maintenance | Startup percentile falls |
| One long job blocks pool | Right-size or split task | More orchestration | Service-time tail shrinks |
| All pools saturated in burst | Autoscale or add capacity | Higher spend | Queue SLO met in replay |
| Serial DAG dominates | Reshape dependencies | Pipeline redesign | Created-to-eligible falls |
Tune autoscaling and warm capacity
Autoscaling reacts after demand is visible, so startup time creates a lower bound on queue delay. Measure webhook or listener delay, infrastructure allocation, image pull, runner registration, and first-job setup. GitHub's runner scale set guidance exposes minimum and maximum runner settings plus startup and execution metrics. Maintain a small warm minimum for latency-sensitive pools, pre-pull large images, and use immutable images with essential tools. Too much baked state increases patching and drift, so keep the image purposeful.
Scale on queued eligible work by pool, not host CPU alone. Include service time so ten one-minute jobs and ten forty-minute jobs do not produce the same target. Set a maximum to contain runaway workflows, but alert before it becomes a hidden queue ceiling. Define drain behavior for maintenance and controller failure. GitLab's runner fleet guidance includes queue-duration and fleet metrics; equivalent telemetry should support every provider. Test scale-from-zero separately because it has the greatest cold-start penalty.
Protect the critical feedback path
Not all jobs have equal delay cost. Pull-request compile and unit checks unblock active developers; nightly compatibility tests and repository maintenance may tolerate later completion. Create a small number of service classes with quotas or reserved capacity, preventing a release storm or scheduled scan from starving interactive feedback. Avoid unlimited priority because low-priority work must eventually run. Aging, fair-share scheduling, repository caps, and separate batch windows can preserve both responsiveness and completion.
Trace the pull-request critical path through dependencies and runner waits. Starting every job sooner does not matter if one serial integration job determines completion. Give that path compatible warm capacity and fail-fast checks, then move non-blocking evidence outside the merge gate when risk permits. Publish queue and end-to-end SLOs by service class. Teams should see whether a delay came from their workflow, a gate, a runner pool, or platform-wide demand, reducing unproductive requests to simply add machines.
Decide when to buy more runners
Add capacity when eligible demand genuinely exceeds compatible supply after obsolete work, routing errors, and provisioning delays are addressed. Replay representative peak windows with proposed capacity and estimate queue percentile, idle time, and cost. Include licenses, cluster overhead, image maintenance, patching, logs, and support, not only compute. A modest warm reserve can be cheaper than engineering a complex scheduler; conversely, doubling a fragmented fleet may leave the same jobs waiting. Make the decision per pool and revisit it as workload changes.
Quantify delay cost carefully. Multiplying queued developer minutes by salary produces a dramatic number but assumes every wait blocks productive work. Combine surveys or workflow telemetry with queue data to estimate the fraction of delays on the critical feedback path, context-switch frequency, and merge impact. Compare that cost with incremental runner expense and the engineering cost of optimization. Use a range, not a false-precision total. The same model helps justify a small expensive macOS reserve while declining excess general capacity that would sit idle without improving completion time.
CI queue time key takeaways
- Instrument created, eligible, assigned, ready, started, and completed timestamps.
- Analyze arrivals and service-time tails by compatible runner pool and job class.
- Cancel superseded work and remove duplicate fan-out before increasing supply.
- Detect routing fragmentation whenever jobs wait beside idle but incompatible runners.
- Tune autoscaling from queued work and startup latency, with deliberate warm minimums.
- Protect the pull-request critical path and buy capacity only after historical replay supports it.
CI queue time FAQ
What runner utilization is healthy? There is no universal percentage. Sustained near-maximum utilization is risky for bursty interactive CI. Choose enough headroom to meet the queue SLO and measure idle cost against developer delay.
Should every new commit cancel the previous run? For pull-request validation, often yes. Do not cancel ordered deployments, migrations, evidence-producing release jobs, or workflows where every commit requires a result.
Is scale-to-zero suitable for CI? It can be for infrequent batch pools with tolerant latency. Interactive pools often need warm capacity because image provisioning and registration delay become the user's queue time.
Conclusion
Queue time becomes tractable when CI is modeled as demand flowing through eligibility, scheduling, provisioning, and execution. Precise timestamps reveal whether to reshape workflows, cancel stale runs, repair routing, warm a pool, tune scaling, or increase capacity. Optimize for a feedback service level rather than a utilization target, and runner spending can follow evidence instead of frustration.