A StatefulSet persistent volume migration is not an ordinary manifest rollout. Pods have stable ordinals, PersistentVolumeClaims created from volumeClaimTemplates have durable identity, and storage backends impose access, topology, attachment, snapshot, and expansion rules. A successful Kubernetes object change does not prove filesystem growth, application consistency, complete data copy, or reversible cutover. Begin with the data contract and recovery objective, then choose expansion, replication, snapshot restore, clone, or copy.
Separate three operations that are often conflated. Expansion increases capacity of an existing claim when the StorageClass and CSI path support it. Migration creates new storage with different class, performance, topology, encryption, or provider characteristics and moves data. Rollback restores service from a known-good old volume, snapshot, backup, or replica after cutover fails. Each has a different point of no return, and shrinking a successfully expanded filesystem is generally not the inverse of expansion.
Baseline the storage change through six stages
Record StatefulSet generation, update strategy, replicas, ordinals, claim names, PVs, StorageClasses, CSI drivers, volume handles, access modes, reclaim policies, zones, capacity, filesystem, mount options, encryption keys, snapshot classes, backup status, and application replication state. Map each ordinal to its claim and application identity. Measure write rate, used bytes, inode use, latency, throughput, attach time, recovery time, and data verification duration. This inventory defines feasible maintenance windows and rollback retention.
| Goal | Preferred mechanism | Downtime profile | Rollback basis |
|---|---|---|---|
| Add capacity to supported volume | PVC expansion | Often online; filesystem dependent | Correct failed request before completion or restore backup |
| Change StorageClass or backend | New PVC plus replication, restore, clone or copy | Online replication or controlled stop | Retained old volume and cutback |
| Change zone | Application replication or snapshot restore to target topology | Cutover required | Old-zone member or volume |
| Change encryption or performance class | Backend feature if supported, otherwise migration | Provider and application dependent | Old volume plus validated backup |
| Reduce provisioned capacity | New smaller volume after data-fit proof | Copy and cutover | Retained larger source |
Expand an existing persistent volume carefully
Confirm allowVolumeExpansion: true on the StorageClass and support in the CSI driver and backend. Patch the PVC request, not the PV capacity field; manually editing PV capacity can prevent the control plane from recognizing that resize work is required. The Persistent Volumes documentation describes controller and filesystem expansion behavior. Watch PVC conditions, events, CSI controller logs, node expansion, filesystem size inside the container, and application free space.
Do not request the final enormous size as the first production experiment. Test backend quotas, maximum size, billing tier, and expansion duration, then use a reviewed target. Current Persistent Volume recovery guidance permits a user to amend a failed expansion request to a smaller value that remains above current capacity, subject to platform support and provided the earlier expansion has not completed. That protects failed expansion requests; it does not shrink storage that already expanded successfully.
Prepare a migration with application-consistent recovery
Create the target StorageClass and validate binding mode, topology, reclaim policy, expansion, mount options, and allowed topologies. The StorageClass reference explains that WaitForFirstConsumer delays binding so pod scheduling constraints can influence provisioning. Use this where zonal placement must follow the workload. Confirm target throughput and latency under a representative dataset; a successful copy to a slower class is not a successful migration.
Choose data movement from application semantics. Database-native replication and backup often preserve transactional consistency better than a crash-consistent block snapshot. Filesystems may need quiescing or an application hook. Kubernetes VolumeSnapshot objects standardize CSI snapshot requests but depend on installed CRDs, snapshot controller, sidecar, class, and driver support. Prove restore, not merely snapshot creation, and validate keys and credentials in the recovery environment.
Execute an ordinal-aware cutover
For replicated systems, add a target-backed member, let it catch up, verify lag and integrity, transfer leadership if needed, and remove the source member only after stability. Repeat one ordinal or shard at a time. For offline copy, stop writers through an application-aware mechanism, confirm no active sessions, take the final snapshot or synchronization pass, validate counts and checksums, then mount the new claim. Preserve stable network identity while ensuring the application does not confuse an empty target with an initialized member.
StatefulSet volumeClaimTemplates do not provide a universal in-place way to replace every existing claim's StorageClass. Common approaches create a parallel StatefulSet, pre-provision claims with expected names, restore data, or carefully orphan and recreate controller objects while retaining pods and PVCs. Every approach has controller-specific hazards. The StatefulSet documentation explains stable identity, ordered operations, update behavior, and PVC retention. Rehearse the exact object sequence in a disposable namespace with the same admission and GitOps controls.
| Control point | Promotion evidence | Rollback action | Data risk |
|---|---|---|---|
| Target provisioned | Correct class, zone, size and encryption | Delete target only if retention permits | Low before copy |
| Initial copy complete | Counts, checksums and restore test pass | Repeat or discard copy | Source remains authoritative |
| Writes quiesced or replication caught up | Zero lag or final sync complete | Resume source writes | Split-brain if both accept writes |
| Target serving | Health, latency and data checks pass | Return traffic and leadership to source | New writes need reconciliation |
| Rollback window closed | Backup plus target stability accepted | Restore from retained recovery point | Old source becomes stale |
Design rollback around data direction
Before cutover, rollback is usually simple: abandon the target and continue using the source. After the target accepts writes, rollback becomes data reconciliation. Decide whether writes will be blocked during the observation period, replicated back, captured in a log, or accepted as a restore-point loss under an approved recovery point objective. Name the authority for declaring the source stale. Never remount two independent copies as if they were one history.
Set old PV reclaim policy to Retain where the rollback plan requires preservation, and protect PVCs from automated cleanup. Snapshot deletion policy, retention, and backend copy behavior must match the plan. Test the command path to reattach or restore under failure conditions. Rollback also includes DNS, Services, secrets, certificates, leader election, and client connection behavior. Storage recovery without application identity recovery can still produce outage or corruption.
Validate data, performance, and lifecycle after change
Build validation from application invariants. Compare record or object counts, logical checksums, indexes, transaction-log positions, replica membership, and sampled business queries. For eventually consistent systems, define when convergence is expected and which differences are acceptable. Run read and write tests through the normal service endpoint, not only inside the migrated pod. Confirm monitoring, backup, compaction, retention, and repair jobs recognize the new member and volume.
Performance tests should include steady load, burst, recovery, checkpoint, compaction, and backup. Measure latency percentiles, throughput, IOPS, queue depth, throttling, filesystem and inode use, network path, and CPU cost. A new class can meet nominal IOPS yet behave poorly under snapshots or rebuild. Watch long enough to cover background maintenance and billing granularity. Compare against an acceptance envelope established before migration rather than a vague expectation that newer storage is faster.
Reconcile lifecycle settings after acceptance. Verify reclaim policy, snapshot class and deletion policy, backup retention, encryption-key ownership, access controls, expansion capability, and cost tags. Remove obsolete volume attachments and targets only after the rollback window closes and a fresh independent recovery point exists. Preserve the change record linking source and target volume handles, ordinals, copy method, checksums, approval, and destruction date.
Automate repeatable checks, but keep destructive cleanup separately approved. A controller can wait for snapshot readiness, create target claims, record status, and run verification jobs; it should not infer that old data may be deleted merely because the new pod is Ready. Stateful automation earns trust by making phase, authority, and irreversibility visible, especially when a partial failure leaves some ordinals on source storage and others on target storage.
Key takeaways
- Separate expansion, migration, and rollback because their reversibility differs.
- Map StatefulSet ordinals to claims, volumes, zones, and application identities before change.
- Use application-consistent replication or backup where block-level crash consistency is insufficient.
- Validate actual filesystem size, data integrity, performance, and restore behavior.
- After target writes begin, rollback requires a data-direction decision, not just remounting the old volume.
Frequently asked questions
Can a bound PVC's StorageClass simply be edited?
Generally no. Moving to a different class usually requires a new volume and a controlled data migration or restore path.
Can an expanded PVC be shrunk back?
Do not assume so. Recovery can correct some failed expansion requests, but successful volume and filesystem expansion is not generally reversible by reducing the claim.
Is a CSI snapshot equivalent to an application backup?
Not necessarily. A snapshot may be crash-consistent while an application backup coordinates transactions and logs. Test restore and application integrity for the required recovery objective.
Conclusion
Stateful storage changes succeed when Kubernetes identity, storage mechanics, and application consistency are planned together. Expand only through supported control paths, migrate into proven target volumes, cut over by ordinal or shard, and retain a recovery source with explicit write semantics. The decisive test is not whether the PVC is Bound; it is whether the application can prove complete data, acceptable performance, and a credible recovery path.