Multi-Window Burn-Rate Alerts That Page Before the Budget Is Gone

Multi-window burn rate alerts page when a service is consuming a meaningful share of its error budget quickly and is still doing so now. They avoid two failures of simple threshold alerts: paging on brief harmless noise and discovering a sustained degradation only after the budget is exhausted. The design pairs a longer window that establishes significance with a shorter window that confirms recency, and applies several burn thresholds so an acute outage and a quieter multi-hour problem receive appropriate urgency.

This method begins with a valid service-level indicator. If the numerator includes expected client errors, the denominator omits valid demand, or telemetry disappears during an outage, precise alert mathematics will still produce bad pages. Implementing SLOs recommends SLIs based on good events divided by valid total events and measured as close to the user's experience as practical. Establish that measurement and an agreed objective before translating the remaining reliability budget into operational notifications.

Derive burn rate from the error budget

For an objective S, the allowed bad-event ratio is 1 minus S. Burn rate is observed bad-event ratio divided by that allowance. A 99.9 percent SLO permits 0.1 percent bad events; an observed 1 percent bad ratio burns at 10x. Burn rate 1 would consume exactly the full budget if sustained throughout the compliance period. This normalization lets one alert pattern apply across services with different objectives, although extreme objectives and sparse traffic still require product-aware treatment.

The first pair consumes about 2 percent of a 30-day budget over one hour; the second about 5 percent over six hours. These are starting points from Google's Alerting on SLOs, not universal constants. Derive thresholds from the fraction of budget that justifies interrupting a human and the response time available. For a period P, window W, and chosen budget fraction F, burn threshold is approximately F multiplied by P divided by W. Keep units consistent and document rounding.

Pair long and short windows with AND

Within each severity pair, require the burn threshold over both the long and short windows. The long window prevents a momentary spike from paging; the short window lets the alert reset soon after recovery instead of remaining active until the long average clears. Combine the fast and sustained page pairs with OR. The short window is often about one-twelfth of the long window, such as five minutes with one hour or thirty minutes with six hours. It must still contain enough events to produce a meaningful ratio.

Do not replace a long-window rate with a short rate plus a long for duration. A fluctuating incident can reset the timer repeatedly while spending substantial budget, and a total outage waits the same duration as a minor excess. Prometheus alerting rules support for, but burn-rate windows already encode duration and severity. A small for may absorb evaluation jitter only when backtesting demonstrates value; it should not become the primary significance control.

Implement recording rules before alert expressions

Create recording rules for bad-event and total-event rates at every required window, aggregated by the service and SLO dimensions used for action. Divide aggregated rates, not averages of instance ratios, so low-volume instances do not receive equal weight. Use the same event classification across windows. Record objective metadata and derive burn expressions from one source of truth rather than copying 0.001 into many files. Evaluate recording groups frequently enough for the five-minute window and monitor rule failures and lateness.

Handle low traffic and missing data explicitly

For ten requests per hour, one failure can represent a huge burn rate and may or may not justify waking someone. Decide from user harm. A high-value, non-retryable transaction may merit immediate investigation, though alerting arrives after the failure. Otherwise, add synthetic transactions, aggregate related request types with a shared user outcome, lower an unjustified objective through product agreement, or require a minimum event count while retaining a separate complete-outage probe. Never hide sparse failures solely to improve pager statistics.

Missing numerator and denominator series are not zero errors. Define telemetry absence separately, using expected traffic, black-box probes, scrape health, or heartbeat signals. If traffic legitimately stops, suppress the ratio without marking the service healthy. If an outage prevents instrumentation from emitting completed requests, an external SLI or synthetic check must preserve detection. Test NaN, zero denominator, delayed ingestion, partial regions, and label churn. Monitoring needs its own reliability target because a budget alert cannot defend an SLO when the evidence path is blind.

Route pages and tickets by response urgency

A page means an on-call engineer can take immediate action to protect users or budget. Include service, SLO, current burn over each window, budget remaining, affected region or operation, dashboard, recent changes, runbook, and rollback path. Deduplicate the fast and sustained page pairs into one incident and inhibit symptom alerts that provide no additional action. Slow burns with days of response time belong in a ticket routed to the owning team, with due date based on projected exhaustion.

Follow Prometheus alerting practices by alerting on actionable symptoms and providing context. Keep cause-oriented signals such as CPU or dependency saturation on the dashboard or as diagnostic alerts unless they require separate action. If the same incident burns several SLIs, group where one responder and mitigation apply, but preserve distinct customer outcomes in the evidence. Avoid paging both the service and every dependency automatically; dependency ownership and escalation should follow confirmed impact.

Validate rules with replay and failure injection

Unit-test PromQL with synthetic counters for steady health, complete outage, brief spike, intermittent failure, slow burn, recovery, counter reset, zero traffic, and missing series. Confirm firing and reset times analytically. Replay historical telemetry from significant incidents and noisy periods, then compare pages with decisions engineers actually needed. Run rules in shadow mode and inspect precision, recall, detection time, reset time, duplicate notifications, and operator action before replacing existing alerts.

Inject controlled failures in a test service or narrow production cohort to validate the complete path from bad event through recording rule, alert manager, notification, runbook, and acknowledgement. Review thresholds after objective changes, traffic-model changes, major retries or fallbacks, and incident learnings. Standardize templates across similar service classes to reduce configuration toil, but allow reviewed exceptions for sparse, asynchronous, or unusually high-consequence systems. Every exception should explain which assumption of the template does not hold.

Connect alerts to the error-budget policy

Burn alerts protect the current period; an error-budget policy governs behavior when reliability is threatened. Define when releases pause, risky changes require additional review, reliability work gains priority, and normal delivery resumes. Track budget consumption by cause and customer journey rather than using the SLO as an individual performance score. A page without authority to roll back or reduce load creates stress but not protection. Give responders clear mitigation rights and define how product and engineering resolve sustained tradeoffs.

Burn-rate alerting key takeaways

• Calculate burn from a valid bad-events-over-total-events SLI and agreed objective.
• Require long and short windows for each threshold, then OR fast and sustained pairs.
• Aggregate event rates before division and centralize objective metadata.
• Treat low traffic, zero denominators, and missing telemetry as explicit product and monitoring cases.
• Page only when immediate action can protect users or budget; route slower threats to tickets.
• Backtest, shadow, inject failures, and connect notifications to a real error-budget policy.

Multi-window burn rate alerts FAQ

Are 14.4x and 6x always correct? No. They are useful starting points for a 30-day period and specific budget-spend choices. Derive thresholds from your compliance period, desired budget fraction, and response urgency.

Why use a short window if the long one already proves significance? It confirms the service is still burning and lets the alert reset quickly after recovery. Both windows must exceed the same threshold within a pair.

Should latency have its own burn alert? Yes when latency is a user-centered SLI. Classify requests beyond the latency target as bad events, taking care with histograms, routes, and enough observations.

Conclusion

Multi-window burn alerts translate an abstract reliability target into response at the right speed. Their value comes from sound event accounting, paired windows, explicit sparse-data behavior, tested rule joins, actionable routing, and authority to mitigate. Start from the budget and customer harm, validate against real failures, and the pager can warn before the organization spends reliability it intended to preserve.