Reliable data pipelines are agreements made executable. A producer supplies a known shape, a pipeline proves what it accepted, transformations preserve meaning, and a consumer can tell whether the result is current enough to use. The hardest failures are rarely dramatic outages; they are silent shifts in grain, duplicated events, late partitions, and corrected history that never reaches the people relying on a metric. A dependable design makes those conditions visible and gives operators a bounded way to respond.
The useful unit is a published data product, not an isolated scheduled job. It has a consumer, an owner, a source authority, a freshness objective, a quality policy, an evidence trail, and a recovery method. This checklist focuses on the decisions that keep those promises intact as source systems change, volumes grow, and teams rotate.
For a broader architectural view, the data pipeline field guide is useful when choosing boundaries and operating responsibilities. The guidance here starts one level closer to production: what must be recorded, tested, observed, approved, and replayed before a dataset is trusted.
Four authoritative references provide practical context. Apache Airflow documentation describes explicit tasks, dependencies, retries, and scheduling; dbt data tests show how data assumptions become executable checks; OpenLineage documentation defines interoperable job and dataset lineage; and BigQuery documentation illustrates the operational concerns of a managed analytical store. These tools address different layers, so a sound pipeline still needs a business contract joining them.
Write the contract before the DAG
Start with the decision the consumer must make and work backward to the data required. Record the dataset grain, primary identity, source of authority, expected delivery window, acceptable lateness, correction policy, and fields that are allowed to be null. A sales fact table might promise one row per order line, while a support event stream might promise one row per state transition. Those are different contracts even when both arrive as JSON. Naming the grain prevents a later join or aggregation from quietly changing what one row means.
A contract also needs a response to disagreement. If the CRM says an account is active while the billing system says it is suspended, the pipeline should name which system wins for which decision and retain the losing value for investigation where policy permits. Define who can change that rule and how consumers will be notified. A contract without an owner is documentation that cannot settle an incident.
| Contract field | Question to answer | Operational consequence |
|---|---|---|
| Grain and key | What does one record represent and how is it identified? | Controls deduplication, joins, updates, and reconciliation |
| Freshness objective | When must the result be usable and what lateness is visible? | Sets alerts, stale states, escalation, and consumer expectations |
| Schema policy | Which additions, removals, and type changes are compatible? | Determines validation, versioning, and approval gates |
| Correction policy | How are late, deleted, or amended records represented? | Defines replay scope, history changes, and notification |
| Accountability | Who owns source quality, pipeline operation, and meaning? | Gives every failure a first responder and a decision maker |
Make schema evolution explicit
Schema change is normal, but unannounced schema change is an outage with a delayed symptom. Separate additive changes from semantic changes. Adding an optional field may be compatible for a tolerant consumer; changing a timestamp from local time to UTC can alter every downstream grouping without changing the field type. Require producers to publish a schema version, compatibility expectation, effective date, and example payload for material changes. Store the received schema with the run so an investigator can compare what arrived with what the code expected.
Use a compatibility matrix rather than a single yes-or-no rule. A nullable field addition can pass ingestion but still need a model review if it affects a metric. A field removal should fail before publication when a consumer depends on it. A type widening may be safe for storage but unsafe for a downstream parser. The pipeline should distinguish structural acceptance from semantic approval, because passing the first does not prove the second.
- Capture the producer schema, payload version, delivery identifier, and effective timestamp for each input.
- Reject missing required fields and incompatible types before they reach curated models.
- Route unknown fields to a reviewed path instead of silently dropping them from evidence.
- Compare field meaning, units, timezone, code sets, and null behaviour during semantic review.
- Give consumers a deprecation window and a tested migration path for removals or renames.
The data contracts guide provides a useful companion for negotiating producer and consumer responsibilities. In implementation, make the contract observable: expose the accepted version, rejected version, and last compatible delivery beside the dataset status. That lets an operator answer whether the failure is bad data, an old consumer, or an unapproved change.
Validate at boundaries and invariants
Validation should happen where an error can still be contained. Check transport integrity and file completeness at ingestion, structural compatibility before parsing, key and relationship rules after conformance, and business invariants before publication. A row count is a signal, not a quality strategy. A complete file can still contain duplicate order IDs, a future accounting date, an invalid currency, or a total that no longer reconciles to the source ledger.
Classify checks by consequence. A critical uniqueness or reconciliation failure should block publication or mark the output unusable. A sudden but explainable volume change may warn and require owner acknowledgement. A new optional attribute can be recorded for review without delaying the core dataset. Store failed records, check definitions, thresholds, and the run identity together. That turns a red alert into evidence a producer can act on rather than a vague request to resend data.
| Check class | Example | Release decision |
|---|---|---|
| Integrity | Checksum, file count, event ID, partition completeness | Reject or quarantine input when evidence is incomplete |
| Structure | Required fields, types, enum values, schema compatibility | Stop parsing or route to a versioned adapter |
| Relationship | Unique keys, valid references, expected parent records | Block affected models and report the population at risk |
| Business invariant | Debits equal credits, inventory is non-negative, totals reconcile | Require owner review before publishing a changed result |
| Freshness | Latest accepted event and publication time against objective | Show current, delayed, or stale state to consumers |
Observe the data, not only the jobs
Job success is only one part of pipeline health. Track source arrival, accepted and rejected records, freshness age, volume shape, validation outcomes, transformation duration, publication state, and consumer impact. Keep the same run identifier across ingestion, modeling, tests, lineage, and publication. With that correlation, a responder can follow one late partition through the system instead of searching separate scheduler, warehouse, and dashboard logs.
Design alerts around decisions and action. An alert should name the affected dataset, missed objective, owning team, last good run, likely scope, and first safe response. Alerting on every retry trains operators to ignore the system; alerting when a critical model is stale without saying which consumers are affected creates a different kind of delay. Give dashboards a visible degraded state so users do not mistake yesterday's result for current truth.
Lineage is most valuable when it answers impact questions quickly. A field-level dependency can reveal which measures may change after a source column is revised, while run metadata shows which version actually produced the published rows. OpenLineage documentation provides a standard vocabulary for this evidence. Pair it with a human owner for each important dataset; a graph can show dependency, but it cannot approve a semantic change or explain a business exception.
Design replay and backfills as releases
Replay is the ability to reproduce a bounded interval from identifiable inputs and versioned logic. Backfill is a replay that intentionally changes historical availability or meaning. Both need idempotent writes, a declared scope, a comparison against the prior result, and a reconciliation owner. Never define a recovery job only as rerun from the beginning; make the input window and output replacement rule explicit so the operator knows what can change.
A safe backfill begins with a dry run that reports affected partitions, row counts, key differences, aggregate deltas, and downstream datasets. Use isolated output or a versioned candidate table for comparison. Approve the result with the owner of the business decision, then publish atomically where possible. Keep the old result long enough to support rollback or investigation, and tell consumers which dates, measures, or records changed.
- Name the source snapshot, input interval, code revision, parameters, and target partitions.
- Make the write idempotent so repeating the operation cannot multiply records.
- Compare old and new counts, keys, totals, and representative records before publication.
- Re-run dependent models when a corrected upstream value changes their derived result.
- Record approval, publication time, affected consumers, and rollback or retention expiry.
Consider a subscription pipeline that receives a corrected cancellation file three days after the original delivery. The recovery path should identify the file version, replace only the affected customer-period rows, recompute retention cohorts, reconcile counts with the billing source, and expose the correction date to analysts. A silent append may make the latest file visible while leaving two contradictory cancellation events in the model.
Assign ownership and recovery actions
Ownership should follow the failure boundary. The source owner is accountable for delivery and declared meaning, the pipeline team is accountable for transport and transformation behaviour, the data steward is accountable for shared definitions, and the consumer owner decides whether a degraded result is usable. One person or team may hold several roles in a small company, but the roles still need to be named. This prevents an operator from being asked to decide whether a business metric is correct.
Write runbooks around state transitions rather than tool commands. For a missing partition, the runbook should say when to wait, when to escalate, whether the last result remains visible, and how to catch up. For a failed assertion, it should identify quarantine location, producer contact, release hold, and safe retry criteria. For a bad publication, it should define who can restore the prior version, how dependent models are handled, and what notice goes to consumers.
| Failure signal | First safe action | Owner and evidence |
|---|---|---|
| No expected delivery | Hold dependent publication and expose delay | Source owner; delivery status and last good input |
| Contract or schema mismatch | Quarantine input and preserve received payload | Producer plus pipeline owner; schema diff and sample rows |
| Quality invariant fails | Stop affected model and identify population at risk | Data steward; failed assertion and run correlation |
| Incorrect result published | Restore or mark version, then scope correction | Consumer owner; impact list and change communication |
Gate every change before publication
A pipeline release includes code, configuration, schemas, check thresholds, adapters, and backfill parameters. Review the proposed change against its contract, run representative historical fixtures, inspect quality results, and trace the likely downstream impact. The release should state whether it is additive, corrective, or semantic. That label changes the required reviewers and the communication plan. A one-line threshold edit can be as consequential as a new transformation when it allows poor data to pass.
Use progressive publication when the dataset is decision-critical. Produce a candidate partition or version, run reconciliation and consumer checks, then switch readers to the approved result. Keep a release record with code revision, contract version, test evidence, owner approval, publication time, and rollback target. BigQuery documentation is a useful reference for managed warehouse capabilities, but the gate remains a product decision: no warehouse feature substitutes for evidence that the result is fit for its use.
The ELT workflow guide can help when deciding where transformation logic belongs. Whichever pattern is selected, require the same release questions: can the new logic be tested against representative history, can the affected output be identified, can the write be repeated safely, and can a consumer understand the resulting change?
Key takeaways
- Define grain, authority, freshness, schema compatibility, correction policy, and accountable owners before implementation.
- Separate structural validation from semantic review so a parsable change cannot silently change a metric.
- Carry run identity, source version, quality results, lineage, and publication state through every layer.
- Make replay and backfill bounded, idempotent, comparable, approved, and visible to affected consumers.
- Gate code, configuration, thresholds, schemas, and historical corrections with evidence and a rollback target.
FAQ
What should a data contract contain?
At minimum, name the dataset grain, identity key, required fields, units and timezone, schema compatibility policy, expected delivery, freshness state, correction behaviour, source authority, and accountable owners. Add examples of valid and invalid records plus the response when the promise is missed. The contract should be precise enough to drive checks and specific enough to settle a disagreement.
When should a pipeline block publication?
Block when the evidence shows that publishing would mislead a decision or make recovery harder, such as a broken key invariant, incompatible schema, failed financial reconciliation, or missing required partition. A warning may be appropriate for an explainable volume shift or optional field issue. The choice belongs in the contract and should identify who can approve an exception.
How much history should a pipeline retain for replay?
Retain enough governed input, metadata, code references, and output versions to cover the correction window and the investigations the business must support. The period varies by dataset and regulation. Apply access controls, minimization, and deletion schedules deliberately; do not keep sensitive raw data indefinitely just because replay is convenient. Document what can be reconstructed after expiry and what requires a source re-export.
Conclusion
A reliable data pipeline earns trust through specific evidence: a contract that names what a record means, validation that catches harmful drift, observability that connects a failure to its impact, and recovery that can change history without concealing the change. Schema evolution, backfills, and incidents are normal parts of the service, so they belong in the design rather than in an improvised response. When owners, tests, lineage, replay boundaries, and release gates travel with the dataset, consumers can use the result with a clear understanding of its currency and limitations. That is the practical standard for dependable data delivery.