Application and schema changes are not atomic when old and new processes overlap. A rolling deployment can keep old instances serving while new instances start. Workers, canaries, consumers, replicas, and self-managed upgrade paths can extend that overlap. An expand and contract database migration handles it as a compatibility protocol: first add what the new code needs, migrate behavior and data while both representations work, verify the cutover, and only then remove the old contract.
This pattern reduces deployment coupling but does not make every database operation safe or reversible. Locking, table rewrites, replication, transaction volume, storage growth, and engine behavior still matter. A data transformation may be lossy. A column drop cannot be undone by redeploying old code. Design each phase around the exact database engine and version, production data shape, fleet rollout, and recovery objective.
Define the compatibility contract
List every reader and writer of the affected schema: API instances, background jobs, analytics, change-data capture, reports, exports, administrative scripts, and external integrations. Record the oldest and newest versions that can run concurrently and the maximum overlap window. Define valid reads and writes in each phase. A migration is backward compatible only if all active consumers continue to operate correctly, not merely if the new application starts.
Separate schema compatibility, data compatibility, and behavioral compatibility. Adding a nullable column may be schema-compatible, but new code that assumes it is populated is not data-compatible. Dual writing can keep columns synchronized while differing validation creates behavioral conflict. Write invariants such as 'every accepted write produces equivalent old and new values' and specify which representation is authoritative during each transition.
Plan the full expand-to-contract sequence
The expand phase adds compatible structures: nullable columns, new tables, indexes built through an engine-appropriate online method, or permissive constraints. The application migration phase introduces read and write compatibility. The data phase backfills historical rows in bounded batches. The switch phase makes the new representation authoritative after evidence. The contract phase removes old code, synchronization, columns, constraints, or tables only after no supported consumer relies on them.
Put every phase in version control with entry criteria, exit evidence, owner, pause condition, and recovery action. Do not merge expand and contract into one release because tests are green. GitLab's public database guidance deliberately spreads destructive changes such as dropping a column across releases so running processes and self-managed upgrade sequences do not encounter a removed field. Your release cadence may differ, but the compatibility reason remains.
| Phase | Database action | Application behavior | Exit evidence |
|---|---|---|---|
| Expand | Add compatible schema and supporting index | Old code remains valid | DDL complete; lock and replication healthy |
| Dual compatibility | Keep old and new representations | Write both or translate; tolerate old data | Mixed-version tests and production canary pass |
| Migrate | Backfill historical data in batches | Reads remain backward compatible | Coverage, parity, and error thresholds met |
| Switch | Optionally enforce new invariant | Read new representation with monitored fallback | No fallback use and stable outcomes |
| Contract | Remove old schema and synchronization | Old code no longer deployable | Fleet, jobs, integrations, and rollback window cleared |
| Close | Update baseline and archive evidence | Only new contract supported | Migration history and runbook complete |
Make expansion operationally safe
Inspect generated SQL and the engine's execution plan. EF Core documentation recommends reviewing generated migrations and testing before production because a generated change can drop a column when a rename was intended. Determine lock level and duration, whether an operation rewrites the table, transaction-log or WAL volume, replica lag, disk headroom, timeout, and cancellation behavior. Test against a production-shaped copy with concurrent traffic.
Use a single migration coordinator. Liquibase records applied changesets in DATABASECHANGELOG and uses DATABASECHANGELOGLOCK to prevent concurrent updates; other tools have equivalent history and locking models. Understand failure semantics if the process loses connectivity or the database restarts. Prefer forward-only immutable changes once deployed. Idempotent deployment scripts can help across databases at different versions, but idempotency does not make an unsafe operation safe.
Choose dual-write and read behavior deliberately
When moving a value, decide whether the application, a database trigger, or another mechanism synchronizes representations. Application dual writes are visible in code and can share a transaction when both fields are in one database, but every writer must adopt them. Triggers cover more writers but add hidden behavior and operational complexity. Cross-database dual writes introduce distributed consistency concerns and need a different architecture, such as durable events and reconciliation.
Define authority. During early rollout, old data may remain authoritative and new fields are derived. After backfill and validation, switch authority explicitly. Avoid indefinite read fallback because it hides incomplete migration and makes contract removal unknowable. Instrument fallback use, mismatches, nulls, parse errors, and write failures. If new and old values disagree, preserve evidence and route to reconciliation rather than silently selecting whichever looks convenient.
| Strategy | Strength | Risk | Suitable check |
|---|---|---|---|
| Application dual write | Explicit and testable in service code | Missed writers and partial application errors | Audit all writer versions and transactional outcome |
| Database trigger | Covers direct database writers | Hidden logic, recursion, load, deployment coupling | Trigger latency, errors, and disablement test |
| Backfill only | Simple for immutable historical fields | Concurrent writes create a moving gap | High-water mark plus final catch-up |
| Read fallback | Supports partially populated data | Can persist and conceal defects | Count fallback to zero before removal |
| Shadow comparison | Validates new representation without serving it | Adds read cost and sampling bias | Mismatch rate with representative sampling |
| Change stream | Decouples large asynchronous migration | Lag, ordering, replay, duplicate handling | Offset, age, idempotency, reconciliation |
Operate backfills as production workloads
Backfill in deterministic, restartable batches using a stable key or range. Record progress and make the update idempotent. Limit rows, transaction duration, concurrency, and rate. Pause automatically on database latency, lock wait, replica lag, error rate, storage pressure, or application SLO breach. Leave headroom for customer traffic. A fast backfill that destabilizes the primary is not zero downtime.
Measure eligible rows, attempted rows, updated rows, already-correct rows, failed rows, retry count, throughput, estimated completion, high-water mark, mismatch, and resource impact. Validate semantic parity with aggregate checks and sampled row comparison. For transformations, quarantine invalid data with identifiers and reasons. Do not mark completion because a job queue is empty; prove coverage against the source population and catch writes that occurred during the scan.
Switch reads with evidence and a kill switch
Canary the new read path by instance, tenant, request class, or percentage where consistency allows. Compare correctness, latency, error rate, and fallback. Use a feature control that can restore the old read behavior without reversing schema. Ensure both paths apply the same authorization and data interpretation. Run long enough to cover batch jobs, reporting windows, and rare records, not only daytime API traffic.
After the switch, stop creating new dependence on the old representation. Remove fallback only after its observed use reaches zero across the full compatibility window. Then remove old writes or synchronization and observe again. Add the new constraint, such as NOT NULL, only when data and all writers satisfy it and the database can validate it safely. Each step should be independently deployable and pausable.
Treat contract as a destructive release
Before dropping anything, prove that no running process, queued job, view, procedure, report, CDC connector, backup workflow, rollback artifact, or supported customer release references it. Search code and query evidence, verify fleet versions, and check long-running workers. Remove application references first. GitLab notes that framework schema caches can make a dropped column fail even when code no longer obviously reads it, illustrating why deployment overlap and framework behavior must be tested.
Take a recoverable backup or snapshot appropriate to the data and test restoration. Schedule destructive DDL with lock timeouts, observation, and stop authority. Dropping a field is usually not reversed by an automated down migration once new writes omit it. Recovery may mean restoring data into a side table, deploying compatibility code, and rolling forward. State that runbook before approval and retain evidence after the schema is cleaned.
Distinguish application rollback from data recovery
During expand and dual compatibility, rolling application code back is often safe because old schema still exists. After new-format writes begin, old code may misread data even if the column remains. After contract, it may not start. Define a rollback frontier for every phase and test the exact old artifact against the current schema. Never advertise rollback based only on a migration tool's down function.
Prefer roll-forward for additive corrections once production data has changed, while keeping a separate disaster-recovery path for corruption or destructive mistakes. GitLab's migration style guide describes production roll-forward and backup restoration considerations even while requiring reversible development migrations. The correct policy depends on system risk, but application rollback, schema reversal, and data restoration must be three explicit decisions.
Use a migration release checklist
- Inventory all schema readers, writers, caches, jobs, views, and integrations.
- Review exact DDL, locks, rewrite behavior, replication, storage, and cancellation.
- Test mixed application versions against every intermediate schema state.
- Make backfills restartable, throttled, observable, and provably complete.
- Canary read authority and drive fallback and mismatch to zero.
- Approve contract only after the rollback frontier, recovery runbook, and consumer evidence are clear.
Key takeaways
- Treat schema change as a multi-release compatibility protocol.
- Test exact engine behavior and production-shaped load; additive does not always mean harmless.
- Assign authority during dual representations and instrument mismatch and fallback.
- Operate backfills with rate limits, health pauses, progress, and parity evidence.
- Separate code rollback, schema reversal, roll-forward correction, and data restoration.
Frequently asked questions
Does expand and contract guarantee zero downtime?
No. It addresses application-schema compatibility across overlapping versions. DDL locks, table rewrites, backfill load, replication, storage, and engine behavior can still cause impact. Production-shaped tests, throttles, timeouts, and pause criteria are required.
Should migrations run when the application starts?
Production systems usually benefit from one controlled migration step with review, credentials, locking, observability, and failure handling rather than every application instance competing. The appropriate mechanism depends on topology, but concurrent uncoordinated schema updates are unsafe.
When can the old column be dropped?
Only after old reads and writes are removed, backfill and parity are complete, fallback use is zero for the required window, every deployed and queued consumer is compatible, and recovery is approved. Calendar time alone is not evidence.
Conclusion
Expand, migrate, verify, switch, and contract turns a risky database release into a sequence of observable compatibility decisions. It gives old and new code room to coexist while data moves under controlled load. The discipline is completed only when destructive cleanup follows evidence, and when teams know exactly where application rollback ends and data recovery begins.