A data contract becomes operational when a producer change can be evaluated before it alters shared data. Documentation published after deployment cannot protect independently released consumers. Data contract compatibility testing puts the current approved contract, proposed contract, serialization rules, semantic assertions, sample records, and consumer evidence into continuous integration. The gate classifies change and either proves compatibility, requires a coordinated migration, or records a bounded exception.
Schema compatibility is necessary but not sufficient. A type can remain identical while its unit, timezone, grain, population, null meaning, or calculation changes. Conversely, some structural changes are safe when consumers ignore them and the format supports evolution. The contract must define both machine-checkable shape and human-owned meaning, then assign which changes can merge automatically and which require evidence from consumers.
Define the contract and its enforcement boundary
Identify the data product, owner, consumers, environment, delivery interface, and lifecycle. The contract can describe a table, event, file, API response, model, or feature set. Include stable identity; schema; key and grain; field descriptions; units; timezone; null semantics; accepted domains; quality objectives; freshness; retention; version; deprecation; and support contact. Separate promises from implementation details so a storage refactor behind an unchanged interface does not create needless consumer work.
Choose the authoritative artifact and how runtime enforcement relates to it. dbt model contracts can enforce model column names and data types, with constraint support depending on platform. A schema registry can enforce serialization compatibility. Neither automatically checks business meaning or every sink. Generate platform-specific schemas from one governed contract where possible, and detect divergence among generated artifacts. Avoid several editable copies that can disagree.
| Contract dimension | Machine-checkable evidence | Human-owned evidence | Failure consequence |
|---|---|---|---|
| Shape | Names, types, nullability | Field purpose | Block incompatible build |
| Identity | Key and partition fields | Entity and grain definition | Require migration design |
| Meaning | Unit/domain assertions | Semantic change rationale | Consumer review |
| Service level | Freshness and completeness metrics | Business impact and objective | Release or incident action |
| Lifecycle | Version and deprecation dates | Support and migration ownership | Prevent premature removal |
Resolve a trusted compatibility baseline
CI must compare against the contract actually approved in the target environment, not only the producer branch. Fetch an immutable registry version or signed repository release and verify identity. Protect baseline updates with authorization and audit history. If multiple production versions are supported, compare the proposal with all of them or use a transitive compatibility policy that truly matches retention and deployment. Comparing only to the latest version can miss an older supported consumer.
Store producer code revision, contract version, serializer configuration, and generation tool versions in the check result. A tool upgrade can change canonical form or type mapping without an intentional contract edit. Fail closed when the baseline cannot be resolved for a material interface; silently using an empty contract turns an outage into permission for a breaking release. Cache only immutable versions and expose their age.
Classify structural and semantic changes
Compute a structural diff: additions, removals, renames, type changes, nullability, defaults, key, constraints, enum symbols, nested fields, and field identifiers. Apply the exact format and registry rules. Avro reader-writer resolution, Protocol Buffers field-number rules, warehouse DDL, and JSON readers have different safety properties. Never reduce compatibility to a generic widening-versus-narrowing list detached from the delivery format.
Require the producer to classify semantic impact for every changed field and dataset. Ask whether entity, grain, population, calculation, unit, timezone, precision, ordering, delete behavior, lateness, or confidentiality changes. Compare semantic test metadata and quality objectives too. A filter that excludes refunded orders can be more disruptive than adding ten columns. CI cannot infer intent reliably, but it can require a structured declaration and route high-impact classes.
| Change class | Default disposition | Additional evidence | Typical rollout |
|---|---|---|---|
| Compatible additive | Automated checks | Old-reader fixture | Deploy producer then adopt field |
| Semantic correction | Owner review | Before/after reconciliation | Canary and communicated effective date |
| Deprecation | Time-bounded approval | Consumer usage and notice | Dual support then removal |
| Breaking structure | Reject ordinary merge | Migration and rollback | Versioned interface or expand-contract |
| Emergency exception | Accountable override | Impact, controls, expiry | Repair and retrospective proof |
Run format-correct schema tests
For serialized events, execute the same registry compatibility check used in production and test actual bytes with old and new readers and writers as required. Preserve field numbers and retired identifiers in Protocol Buffers; apply Avro defaults and aliases according to the specification rather than database assumptions. For tables, compile or materialize the proposed model in an isolated schema and inspect the resulting platform types and constraints. Generated YAML alone may not reflect warehouse behavior.
Include keys and deletion. A key change can alter partition routing, ordering, compaction, joins, and upserts even when value schema is compatible. Test old delete and tombstone records against new consumers. Exercise null, missing, boundary numeric, timezone, precision, nested, enum, and unknown-field cases. Produce an actionable diff with field path, rule, old value, new value, and suggested migration rather than a single red status.
Test semantics with data and invariants
Run the proposed producer on curated fixtures that represent ordinary, boundary, invalid, and historical cases. Assert grain, key stability, unit conversion, accepted values, null meaning, referential rules, and balances. For transformations, compare outputs from current and proposed revisions over a representative immutable sample. Require expected-difference declarations so reviewers can separate intentional corrections from accidental drift.
Sample data must be safe, versioned, and representative. Use synthetic cases for sensitive domains and approved de-identified snapshots when distributions matter. Do not embed production personal data in CI artifacts or failure logs. Samples cannot prove population-wide quality, so pair them with static rules and a pre-publication canary or reconciliation for high-risk changes. Keep a regression case for every escaped contract incident.
Collect evidence from real consumers
Maintain a registry of supported consumers, contract versions, selected fields, criticality, owner, and deployment cadence. Consumer-driven tests can submit fixtures or assertions that the producer runs in CI, but the producer must not execute arbitrary untrusted code with secrets. Use isolated runners and versioned test artifacts. For low-risk additions, a representative old-reader suite may suffice; removals, key changes, and semantic shifts need explicit affected-owner evidence.
Observed field usage and lineage can help identify consumers, but absence is not proof when coverage is incomplete. Require periodic consumer attestation and expiry for dormant registrations. Define support windows so abandoned consumers cannot freeze the contract indefinitely. When a consumer cannot migrate, make the cost visible and decide whether to extend support, provide an adapter, version the interface, or retire the consumer through accountable product governance.
Build a progressive CI gate
On every contract-affecting merge request, lint identity and required metadata, resolve the production baseline, generate artifacts, compute structural and semantic diffs, run compatibility, execute fixtures, identify consumers, and assign a risk class. Publish one signed result linked to the code revision. A compatible additive change can pass automatically. A breaking or uncertain change should fail with a migration route, not a vague instruction to seek approval.
Repeat critical checks during deployment because the baseline may have advanced after CI. Use optimistic version checks to prevent two individually compatible branches from composing into an incompatible release. Register schemas or publish contracts through the deployment identity, not a developer workstation. After rollout, verify runtime schema, quality metrics, consumer errors, lag, and reconciliation. Close the change only when temporary versions and dual writes have an owner and retirement date.
Govern exceptions and coordinated breaks
An override should identify the failed rule, business reason, affected interfaces and consumers, accountable approver, compensating controls, deployment window, rollback, and expiry. Scope it to one contract revision. Do not disable compatibility globally or lower registry policy for convenience. Alert when exceptions approach expiry and verify that promised migration work completed. Track exception frequency by rule; repeated overrides may reveal a bad policy or a missing versioning mechanism.
For an intentional breaking change, create a new compatible expansion or versioned interface, deploy producer support, allow consumers to migrate, observe adoption, stop new consumers on the old version, and remove only after support and retention windows clear. Rehearse rollback before cutover. Preserve old schemas as long as retained data requires them. Breaking change governance is a lifecycle, not a one-time approval in a pull request.
Introduce contract testing on one shared product
Choose a data product with clear ownership and several known consumers. Write the existing contract from observed behavior, resolve disagreements, and establish a production baseline. Add structural checks and a small semantic fixture suite. Register consumers and rehearse an additive field, a rename attempt, a unit correction, and a key change. Measure check duration, false blocks, diagnostic quality, and migration lead time.
Keep the fast path under minutes and move expensive representative runs to a required asynchronous check. Provide local tooling that uses the same rule engine. Document migration patterns and supply generated reader fixtures. Review compatibility rules after incidents and format upgrades. The purpose is to move discovery earlier while preserving a clear route for legitimate evolution, not to make shared data immutable forever.
Key takeaways
- Compare with the immutable contract versions actually supported in production.
- Test using the exact serialization or table engine semantics, including keys and deletes.
- Require semantic declarations and invariant tests because unchanged types can carry breaking meaning.
- Use consumer evidence proportional to reach, criticality, and graph confidence.
- Scope exceptions tightly and migrate breaking changes through versions or expand-and-contract.
Frequently asked questions
Is a schema registry the same as a data contract registry?
Not necessarily. A schema registry manages serialized schemas and configured compatibility. A broader data contract also includes semantics, keys, quality, service levels, ownership, lifecycle, and consumer commitments.
Can sample-data tests prove compatibility?
No. They expose concrete semantic and reader defects but cover only selected cases. Combine them with formal schema rules, invariants, consumer tests, and runtime validation.
Must every breaking change be forbidden?
No. It must be coordinated. Version the interface or expand support, migrate consumers with evidence, preserve replay requirements, define rollback, and remove the old path only after its obligations end.
Conclusion
Data contract compatibility testing makes shared interfaces evolvable because breakage is discovered while a producer change is still reviewable. A trusted baseline, format-correct schema rules, semantic invariants, real consumer evidence, progressive CI, and governed exceptions cover the failure modes a schema diff alone cannot. The result is faster producer delivery with a credible promise to downstream teams.