SLOs for asynchronous systems should measure whether accepted work produces the promised outcome correctly and within a useful time, not merely whether a broker accepted a message. A 202 Accepted response can remain healthy while consumers are stopped, a workflow is retrying for hours, one tenant monopolizes a partition, or poison work has moved to a dead-letter queue. Infrastructure availability and queue depth help diagnose these conditions, but the user experiences completion, delay, duplication, omission, or an explicit terminal failure.
Begin with a unit of work and a lifecycle. It may be an order, document conversion, notification, reconciliation, media job, or workflow instance. Give it a stable ID and timestamps for accepted, eligible, started, materially completed, and terminally failed. Preserve its priority, tenant class, and deadline without using high-cardinality identifiers in metrics. The SLI counts work units over an agreed observation period; traces and logs retain individual evidence for diagnosis and audit.
Define the asynchronous user promise
Write the promise in product language: 99 percent of standard exports accepted during supported hours become downloadable within ten minutes; 99.9 percent of payment events reach a correct terminal state within two minutes; urgent notifications accepted by the service are handed to the downstream provider within thirty seconds. State when the clock starts, what pauses it, what counts as complete, and which exclusions are legitimate. If users choose a future schedule, measure from eligibility time, not submission. If they cancel, classify separately rather than calling cancellation a service failure.
| Timestamp | Meaning | Useful interval | Common ambiguity |
|---|---|---|---|
| Accepted | Service owns the work | End-to-end completion | Request acknowledged before durable write |
| Eligible | Delay or prerequisite has cleared | Scheduling wait | Customer-scheduled work |
| First started | A worker began attempt | Queue wait | Retries obscure first attempt |
| Completed | User-visible outcome committed | Processing and total | Internal step mistaken for completion |
| Terminal failure | No automatic retry remains | Failure ratio | DLQ treated as success |
| Observed by consumer | Outcome can be read or delivered | Publication delay | Write complete but index stale |
Choose completion, timeliness, and correctness SLIs
A completion SLI is good terminal outcomes divided by eligible accepted work whose deadline can be evaluated. A timeliness SLI counts completed work within the promised threshold over the same eligible population. Correctness may require a domain invariant, reconciliation, or synthetic known-answer work. Keep the denominator mature: a job accepted one minute ago cannot be called late against a ten-minute promise. Evaluate cohorts by acceptance or eligibility time and allow their deadline to pass before final accounting, while a near-real-time leading indicator estimates risk.
Google's service level objectives guidance notes that big-data and processing systems care about throughput and end-to-end latency. Measure the entire chain that your service owns, including publish, queue, processing, retries, and user-visible commit. Component metrics remain useful for debugging but do not multiply component objectives into a customer promise. Segment only where users receive different commitments, such as interactive versus bulk priority, rather than giving every internal queue its own SLO.
Instrument work end to end
Write lifecycle events to a durable status store or append-only audit stream using the work ID and attempt ID. Counters should record accepted, eligible, completed on time, completed late, terminal failure, cancelled, and expired units by bounded service class. Histograms capture end-to-end and processing duration. The current OpenTelemetry messaging metrics conventions define messaging.process.duration and attributes for messaging system, destination, operation, and consumer group; these are useful processing diagnostics but do not automatically provide business completion.
Retries must not inflate denominators. One work unit with five attempts remains one accepted unit. Emit attempt metrics separately for diagnostics, preserving error class and backoff. Propagate trace context where practical, but long delays and fan-out may require links rather than one continuously open span. Avoid message payload, customer ID, or work ID in metric labels. For debugging, exemplars or logs can connect an aggregated late-work bucket to sampled work IDs under access controls.
Use queue health as leading evidence
Oldest eligible work age is often more actionable than depth because it maps directly to a completion deadline. Depth depends on arrival rate and can be large while consumers keep pace. Also track arrival and completion rates, time to start, in-flight work, retry rate, consumer saturation, partition skew, and estimated drain time. Alert on the user SLO for paging and use these signals to identify cause or act before breach. Capacity decisions should compare backlog age growth with available headroom and maximum safe concurrency.
| Signal | Strong use | Blind spot | Companion |
|---|---|---|---|
| Oldest message age | Detects waiting deadline risk | Provider semantics may reorder poison work | Work-status age |
| Visible depth | Shows pending volume | No arrival-rate context | Arrival and completion rate |
| In-flight count | Shows active concurrency | May include stuck attempts | Processing duration |
| Dead-letter count | Shows exhausted retries | May reset original age | Original acceptance timestamp |
| Consumer CPU | Shows resource saturation | Not user impact | Completion and age |
| Estimated drain time | Supports scaling decision | Assumes stable rates | Burst scenarios |
Provider metrics require semantic review. Amazon SQS documents ApproximateAgeOfOldestMessage as the age of the oldest unprocessed message, but standard queues can move repeatedly received messages to the back, poison messages can be excluded from that age behavior, and age resets when moved to a dead-letter queue. The SQS metric documentation also notes distributed metrics are approximate. Preserve original acceptance time in application evidence when the promise requires exact end-to-end age.
Define retry, dead-letter, and replay semantics
A retry is neither success nor terminal failure; it is time spent from the same work budget. Set maximum attempts and maximum elapsed time from the user promise, not an arbitrary broker default. Use exponential backoff with jitter for transient failures and stop retrying permanent validation or authorization errors. A dead-letter queue is a quarantine that enables inspection and replay. Entering it usually means the original work missed completion and should count as bad unless the product explicitly promised only durable capture and later manual resolution.
Replay creates a new attempt linked to the original work, not a new denominator that erases the miss. Define whether eventual correction changes reporting; operational SLOs normally preserve the original late or failed event while a recovery metric tracks time to repair. Make handlers idempotent and use stable idempotency keys around side effects. For fan-out, define completion as all required children, a quorum, or a named subset. Partial success should be visible to users and metrics instead of becoming an ambiguous green workflow.
Set objectives and alerts from observed demand
Gather several weeks of acceptance-to-completion distributions by meaningful class, then negotiate thresholds from user need and dependency commitments. Do not set the objective exactly at current performance with no engineering rationale. Use a rolling or calendar period consistent with the product promise. Compute bad units from completed-late, terminal-failed, and still-incomplete-after-deadline cohorts. Burn-rate alerting works when volume is sufficient; low-volume high-value workflows may need per-event pages, synthetic work, or calendar-window reviews.
Page when intervention can protect not-yet-late work: oldest eligible age approaches the deadline while drain time rises, or a fast burn consumes completion budget. Include affected class, age distribution, backlog, arrival and completion rates, failing step, recent changes, and safe controls for scaling, pausing intake, shedding low-priority work, or replaying. A separate ticket can track slow reliability erosion. Implementing SLOs stresses that objectives drive product decisions; use the budget to negotiate capacity and feature tradeoffs, not to blame operators.
Test the SLO and recovery path
Inject a synthetic work item with a known expected result and measure it through the real queue and status path. Test worker loss, broker delay, duplicate delivery, partition starvation, poisoned payload, slow dependency, status-store failure, clock skew, DLQ transfer, and replay. Confirm counters do not duplicate across retries and the deadline cohort closes correctly. Reconcile accepted work with terminal states regularly so silent omissions appear even when no message remains in a queue.
Review objectives when priorities, deadlines, retry policy, providers, or workflow topology change. Track SLO attainment, budget by failure class, p50/p95/p99 completion time, late-work repair time, replay success, unowned DLQ work, and age-measurement disagreements. Retain enough event history to recalculate after metric bugs. An async SLO is trustworthy only when operators can trace aggregate failure back to work lifecycle evidence without exposing customer payloads.
Asynchronous SLO key takeaways
- Define one durable unit of work and the exact accepted, eligible, completed, and terminal states.
- Measure completion and timeliness over mature cohorts, not broker acknowledgements or attempts.
- Use queue age and drain time as leading indicators while paging on customer outcome.
- Count retries as attempts of one work unit and treat dead-lettering as quarantine, not success.
- Preserve original deadlines through replay and define partial-success semantics for fan-out.
- Reconcile accepted and terminal work and exercise failure, duplicate, poison, and recovery paths.
Asynchronous systems SLO FAQ
Is queue depth a good SLI? Usually not by itself. Users care about completion and delay; depth is a diagnostic whose meaning depends on arrival rate, processing rate, priorities, and deadlines.
When does the SLO clock start? When the service accepts ownership of work, or at an explicit later eligibility time for scheduled or dependency-gated work. Document and instrument that boundary.
Should a replayed job become good after it completes? Record recovery, but preserve the original deadline miss in SLO accounting. Otherwise repeated replay can rewrite reliability history.
Conclusion
Async reliability becomes visible when the system follows work from accepted promise to observable outcome. Mature-cohort completion ratios, end-to-end clocks, queue-age leading signals, attempt-aware instrumentation, explicit dead-letter semantics, and reconciliation reveal failures that request metrics miss. Build the objective around what users wait for, and queues and workers become explainable parts of that promise rather than misleading proxies for it.