The phrase data observability vs data quality suggests a tool contest, but reliable data systems need both explicit expectations and evidence about unfamiliar runtime behavior. Data quality tests encode known, deterministic conditions such as uniqueness, referential integrity, accepted values, or reconciliation. Observability collects and relates signals such as freshness, volume, distribution, schema, lineage, runs, and failures so operators can detect and investigate change. Either can alert after execution; the useful distinction is the knowledge and lifecycle position of the control.
Tests can prevent publication when a known rule fails. Runtime monitors can reveal a novel shift that no author anticipated. Observability without contracts produces noisy suspicion; tests without runtime context miss outages, drift, and unknown interactions. The design question is which failure should be blocked before consumers see it, which should be detected after operation begins, and how both routes create accountable incident evidence.
Separate prevention, detection, and diagnosis
Prevention sits at a trusted change or publication boundary. It evaluates a rule whose failure justifies stopping a merge, deployment, table commit, or data release. Detection observes an operating system and signals that behavior may be abnormal. Diagnosis joins context to explain cause and impact. A row-count assertion can detect a bad batch after it runs and still prevent that candidate from publishing. A freshness monitor may detect an outage but cannot restore data. Classify the decision, not the product category.
For each control, write the asset, condition, evaluation time, consequence, owner, and evidence. A warning with no response objective is telemetry, not an operational control. A blocking test with a high false-positive rate will be bypassed. Assign severity from consumer impact and recovery cost. Keep informational profiling separate from paging so discovery can remain broad without exhausting responders.
| Control role | Best for | Example | Failure consequence |
|---|---|---|---|
| Pre-merge test | Code and contract defects | Required field and type | Reject producer change |
| Pre-publication test | Candidate data defects | Balance to source control | Keep candidate hidden |
| Runtime monitor | Availability and unfamiliar drift | Freshness or volume anomaly | Alert and investigate |
| Consumer check | Fitness for a decision | Feature coverage by cohort | Pause dependent use |
| Reconciliation | End-to-end correctness | Source and ledger totals | Repair and attest |
Use deterministic tests for known obligations
Encode stable rules close to the owning transformation or contract. Common tests cover schema, nullability, uniqueness, relationships, domains, ranges, balances, and row-level business predicates. dbt data tests return failing rows from SQL assertions; Great Expectations organizes explicit expectations and validation definitions. The framework matters less than whether the rule has a documented purpose, scoped data, severity, owner, and remediation. Avoid hundreds of generated checks that nobody can explain.
Run cheap structural checks in CI with fixtures, richer checks on a representative candidate, and full reconciliation at publication or after ingestion. Tests should be deterministic for the same data and configuration. If a threshold changes automatically with recent history, it behaves more like a monitor and needs drift governance. Store failure rows securely and cap samples; quality evidence can contain sensitive data. Version tests alongside the contract they enforce.
Use observability for operating behavior and unknowns
Instrument pipeline runs, datasets, and dependencies with consistent identities. Capture start and completion, status, duration, input and output versions, freshness, volume, schema, distribution summaries, rejected records, and lineage where available. OpenLineage facets provide a standard extension model and a data-quality metrics facet for dataset-level measures. Signals become useful when they share run and dataset context; isolated dashboards force responders to reconstruct the system during an incident.
Anomaly detection can identify deviations from historical patterns, but seasonality, launches, backfills, and changing product mix produce legitimate shifts. Begin with transparent baselines and segment important cohorts. Tune on incident usefulness rather than model sophistication. Every alert should identify the affected asset, expected and observed behavior, confidence or threshold, recent changes, upstream status, likely consumers, owner, and a path to raw evidence.
| Signal | Known-rule test | Runtime monitor | Useful response |
|---|---|---|---|
| Freshness | Must publish by agreed deadline | Delay trend by source | Check run and upstream health |
| Volume | Nonzero or bounded control total | Seasonal anomaly | Compare product events and partitions |
| Schema | Contract compatibility | Unexpected runtime schema | Stop publish or route migration |
| Distribution | Domain and range | Novel cohort shift | Validate semantics and source change |
| Lineage | Required dependency declared | Observed path changed | Assess downstream impact |
Build coverage from failure modes
List consequential failure modes for each critical data product: source missing, partial partition, duplicate keys, stale reference data, schema break, semantic drift, failed delete, late event, transformation defect, unauthorized field, and consumer misuse. Map each to prevention, detection, diagnosis, correction, and owner. Gaps become investment decisions. Multiple tools that all alert on row count are not broad coverage; they are duplicate signals with separate operating costs.
Prioritize by decision impact, not table popularity. A small regulatory export may need stricter reconciliation than a heavily queried exploratory dataset. Establish data service levels for freshness, completeness, and response where meaningful. Keep producer contract controls distinct from consumer fitness checks: a dataset can satisfy its published contract and still be inappropriate for a particular model or report. Consumers should own additional assumptions they introduce.
Price the full operating burden
Tool cost includes agents, metadata ingestion, storage, query load, integration maintenance, rule authoring, model tuning, alert triage, lineage repair, access control, upgrades, and training. Estimate alerts per on-call shift, percentage actionable, time to identify owner, and time to useful diagnosis. A broad platform with poor identity and ownership can increase toil. A focused test framework can be more valuable when the main risk is known contract breakage.
Decide where evidence lives and how long it remains. Quality samples and profile statistics can expose personal or confidential information. Use aggregate metrics where possible, redact payloads, apply dataset access rules, and audit who can inspect failures. Define availability expectations for the observability control plane; if it fails with the pipeline, incident response loses its evidence. Export or retain essential run and alert history for continuity.
Evaluate tools with real incidents
Run a proof using two critical pipelines and several seeded failures: missing partition, schema addition, duplicate replay, stale source, gradual distribution shift, and legitimate launch spike. Measure detection, false positives, owner routing, lineage accuracy, query overhead, and diagnostic time. Test deployment through CI, infrastructure, permissions, and incident workflows. Vendor feature lists do not reveal whether metadata identity matches your catalogs and schedulers.
Prefer interoperability and exit paths. Rules should be versionable, results exportable, dataset identity documented, and alerts routable through standard systems. Open specifications can reduce integration coupling, but adopting a standard does not guarantee complete instrumentation. Price the work to maintain adapters and fill coverage gaps. Choose a product only after deciding the operating model; otherwise software becomes the substitute for ownership.
Implement one reliability loop
Start with a data product tied to an important decision. Define its contract, consumers, failure modes, and publication boundary. Add a small set of blocking deterministic tests, then instrument run, freshness, volume, schema, and lineage signals. Route alerts to the owning team with a response objective. Create a correction path and post-incident review that turns recurring unknown failures into explicit tests or engineering fixes.
Review controls quarterly and after incidents. Remove checks that never influence a decision, tune noisy monitors, and test alert delivery. Track escaped defects, prevention catches, actionable alert rate, time to ownership, time to restore, repeat incidents, and coverage of critical products. Do not combine them into one quality score; separate measures preserve the tradeoffs and point to different interventions.
Exercise the response path, not only the detector. In a controlled game day, delay an upstream partition, introduce a compatible schema addition, replay duplicate keys, and change one important distribution. Confirm that alerts reach the correct owner, evidence access works, consumers can be identified, publication can stop, and corrected data can be released. Record missed signals and ambiguous ownership as reliability defects.
Key takeaways
- Classify controls by decision role: prevent, detect, diagnose, correct, or attest.
- Use deterministic tests for known contracts and runtime signals for system behavior and unfamiliar drift.
- Join quality metrics with run, dataset, lineage, change, and ownership context.
- Evaluate actionable incidents, overhead, privacy, and retained operating work, not feature count.
- Feed recurring runtime failures into tests or engineering changes so the system learns.
Frequently asked questions
Can observability replace data tests?
No. It may execute or display tests, but known obligations still need explicit assertions and enforcement. Historical anomaly models cannot reliably infer every business contract.
Should anomaly alerts block publication?
Only when their precision and consequence justify blocking. Many anomaly signals should begin as warnings with human review. Promote them after measuring false positives and defining a safe override.
Is one data quality score useful?
A summary can aid navigation, but it should not hide freshness, completeness, correctness, coverage, and incident response behind one number. Those dimensions have different owners and remedies.
Conclusion
Data observability and data quality testing solve overlapping but distinct reliability jobs. Explicit tests make known obligations enforceable; observability reveals how the distributed data system behaves and helps responders understand unfamiliar failure. Designing them as one prevention, detection, diagnosis, and learning loop produces stronger coverage with less duplicate noise than buying either label in isolation.