Data pipeline SLOs answer whether consumers received usable data in time, at expected coverage, and with recoverable errors. Scheduler uptime and green task states are weak substitutes. A job can finish successfully after reading only half its partitions, a streaming application can remain available while its watermark stops, and a warehouse table can update on schedule with duplicated or invalid rows. Reliability begins at the consumer's decision or product behavior and works backward through the data path.
Use a small portfolio of outcomes rather than one overloaded score. Freshness describes delay between the relevant source event or expected period and consumer-ready data. Completeness describes how much expected data arrived and was processed. Correctness or validity checks whether values satisfy domain rules or known outcomes. Recovery measures how quickly the organization restores trustworthy output and reprocesses affected periods after a breach. Availability can matter for interactive query serving, but it does not replace these pipeline-specific dimensions.
Define the data consumer contract
Inventory each material data product, its owner, consumers, decisions, serving location, schedule, priority, and tolerated staleness. Finance close, fraud models, executive dashboards, customer recommendations, and exploratory notebooks do not deserve identical objectives. Define the consumer-visible completion point: a table committed, quality gates passed, catalog version published, cache refreshed, or model loaded. Record time zone, business calendar, late-arrival policy, and maintenance windows. An hourly partition promised by 15 minutes past the hour needs a different denominator from an always-on stream.
| Consumer need | SLI | Good event | Primary evidence |
|---|---|---|---|
| Current dashboard | Freshness | View reads data newer than threshold | Output generation timestamp |
| Complete finance period | Coverage | Expected partitions and records reconciled | Source and sink audit counts |
| Valid operational decision | Correctness | Known invariants and probes pass | Quality assertions |
| Stable streaming feature | Timeliness | Event processed within event-time target | Watermark and sink commit |
| Repair after incident | Recovery | Affected output corrected within target | Backfill completion and validation |
Measure freshness at the consumer boundary
For batch data, measure expected availability time against the timestamp when a complete, validated output version becomes visible. A daily job completing at 02:00 is not fresh if its source extract arrived late and contains data only through the prior morning. Store source cutoff, event-time range, processing completion, and publication time. For streaming, measure event-time lag or the proportion of eligible events reflected in serving state within the target. Processing time alone hides producer delay; event time alone can be unreliable when client clocks are untrusted, so define the authoritative timestamp.
The Google SRE Data Processing Pipelines chapter gives useful freshness forms: a percentage of data processed within a duration, oldest data no older than a duration, or a job completed within a duration. Prefer a ratio that can consume an error budget, weighted by real reads when feasible. A table unused overnight should not necessarily spend the same customer-impact budget as a stale dataset read thousands of times, although read-weighted metrics need a fallback to detect latent staleness before the next important consumer arrives.
Measure completeness and validity independently
Completeness requires an expected total. Reconcile producer acknowledgements with durable ingest, expected partitions with delivered partitions, source control totals with sink totals, or business entities across systems. Account for legitimate filters, deduplication, corrections, and late arrivals. A percentage can hide concentrated loss, so segment by high-value tenant, region, event type, or finance-bearing class where commitments differ. Preserve audit counts before transformations so the pipeline can explain where records were lost or multiplied.
Validity checks schema, nullability, range, uniqueness, referential integrity, distribution, and domain invariants. Not every anomaly is incorrect, and static thresholds can reject real business change. Classify checks as hard publication gates, warning guardrails, or diagnostics, with owner and response. Synthetic golden records can provide known outputs when ground truth is otherwise unavailable. Keep correctness separate from freshness: publishing stale but known-good data may be safer than exposing a fresh corrupt table, and the product should communicate that degradation deliberately.
| Dimension | Failure example | Safe degradation | Repair evidence |
|---|---|---|---|
| Freshness | Hourly output is two hours late | Serve marked prior version | New version published |
| Completeness | One region's partition missing | Block aggregate or label partial | Counts reconcile |
| Validity | Price column violates domain | Quarantine affected partition | Assertions and sample pass |
| Skew | Events land in wrong period | Delay close for late window | Periods reshuffled |
| Recovery | Backfill cannot meet catch-up target | Prioritize critical products | Validated backlog drained |
Instrument lineage and quality evidence
Emit one run identity across orchestrator, compute job, input versions, output dataset version, and quality result. Capture row and file counts, byte volume, event-time bounds, schema version, source cutoff, watermark, and assertions. OpenLineage's data quality metrics facet can carry dataset health metrics such as row count, file count, bytes, update time, and column statistics. Treat these as interoperable evidence, while the SLO service applies consumer contracts and bounded dimensions.
Lineage supports impact analysis: when an upstream source is late or a quality rule fails, identify which data products and commitments depend on it. Keep lineage current through runtime events and compare it with declared ownership. Avoid putting sensitive values or high-cardinality record identifiers into monitoring. Store sampled failing records in a restricted diagnostic location with retention. Reconcile orchestrator success against output publication and consumer probes because a successful task event can precede a failed commit, catalog update, or serving refresh.
Set objectives and alert on budget risk
Baseline performance across business cycles, then negotiate targets from consumer impact and upstream commitments. Express objectives as ratios when possible: 99 percent of hourly partitions published complete within 20 minutes; 99.9 percent of finance events present in the correct daily period; 95 percent of freshness breaches repaired within two hours. Google SRE's example SLO document includes pipeline freshness and completeness examples and illustrates documenting rationale, measurement, and targets rather than treating the number alone as the SLO.
Page on customer-impacting freshness or correctness risk when immediate action can preserve a deadline. Leading alerts may use watermark stall, missing partition, quality-gate failure, or estimated catch-up time, but route them according to urgency. Group alerts by root delay and impacted products so one late source does not page every downstream owner independently. Include last good output, affected period, lineage, expected and actual counts, recent changes, retry state, backlog, recovery estimate, and runbook. Missing SLO telemetry needs a separate alert.
Design backfill and recovery objectives
Recovery begins by stopping further bad publication, preserving last known-good output, and identifying affected partitions and consumers. Backfill should be idempotent, versioned, restartable, and isolated from the live workload. Use checkpoints, deterministic transformations, source snapshots, and output versioning. Prioritize critical periods and consumers when full replay would overwhelm shared capacity. Reserve catch-up headroom; a pipeline provisioned only for normal arrival rate can never reduce backlog after an outage.
Define recovery time from breach detection or from the first bad output to validated consumer-ready correction, and measure detection separately. Completion of a backfill job is not recovery until completeness and validity checks pass, outputs publish atomically, caches or models refresh, and consumers receive disposition. Preserve the original SLO miss even after correction; report repair success and duration alongside it. Rehearse source unavailability, schema rollback, partial-region replay, and an erroneous backfill that must itself be reversed.
Use tool checks without confusing them for the SLO
dbt source freshness compares source recency with configured expectations and can run as a job step. The documentation recommends check frequency appropriate to the SLA, such as checking a one-hour commitment more often than hourly. It is a valuable source-side leading signal, not proof that downstream models are complete, valid, and published. Decide whether stale sources should block transformation or allow a marked prior version based on the cost of stale versus partial data.
Catalog every objective as code with owner, consumer, query, threshold, period, exclusions, source of truth, alert route, runbook, and review date. Unit-test expected periods, daylight-saving transitions, late events, empty but valid partitions, duplicate records, schema evolution, and missing metrics. Recompute historic windows after instrumentation changes before switching. Track product coverage, objective attainment, budget by cause, time to detect, time to validated recovery, backfill capacity, repeated upstream offenders, and consumers still reading known-bad versions.
Data pipeline SLO key takeaways
- Start from a named consumer decision and define the consumer-visible publication point.
- Measure source cutoff or event time through validated output, not scheduler completion alone.
- Reconcile expected and delivered data for completeness and keep validity as a separate dimension.
- Use runtime lineage and dataset quality evidence to find impacted products and failure location.
- Set ratio objectives and alert according to user deadline and available action.
- Treat validated backfill and consumer refresh as recovery, while preserving the original miss.
Data pipeline SLO FAQ
Is successful job completion a freshness SLI? It can be one input for a simple batch, but only if completion reliably means validated data is consumer-visible and covers the intended source period. Verify that contract rather than assuming it.
Should incomplete data be published on time? Usually a known-good stale version is safer, but the correct choice depends on consumer use. Make partial publication explicit, labeled, and governed; never let timeliness hide incompleteness.
Does a completed backfill erase the SLO violation? No. It demonstrates recovery. Preserve the missed freshness or completeness event and report validated repair time separately.
Conclusion
Reliable data is timely enough for its decision, complete enough for its population, valid enough for its use, and repairable within an agreed window. Consumer contracts, end-to-end timestamps, audit reconciliation, quality evidence, lineage-aware alerts, and rehearsed backfills turn that statement into measurable operations. Tool-level freshness checks help, but the SLO belongs at the point where people and products consume the result.