A Kubernetes upgrade PodDisruptionBudget conflict is rarely solved by adding --force. The budget represents an application availability claim, while node drain is asking the Eviction API for permission to remove pods. If the claim is impossible, selectors are wrong, replicas are already unhealthy, or no replacement capacity exists, the correct result is a blocked drain. The operator's job is to resolve the system condition without bypassing the safety claim blindly.
PodDisruptionBudgets limit voluntary disruptions, not every outage. A node crash, resource exhaustion, or network partition can reduce availability outside their protection. During planned upgrades, they work only when controllers can create healthy replacements on other nodes. Therefore upgrade readiness includes PDB arithmetic, workload health, scheduling capacity, topology, storage, termination, and provider surge. Inspect the application as a recovery system, not the PDB as an isolated manifest.
Prove upgrade readiness through six disruption gates
Inventory every PDB, selected pod, controller, desired replicas, ready replicas, disruptions allowed, node placement, volume attachment, priority, and termination grace. Flag overlapping PDB selectors, empty selectors, unmanaged pods, single replicas, and budgets that allow zero disruptions. Simulate removal of one node and one zone. Then calculate whether replacement pods can schedule while the old pod is terminating. Repeat for the managed service's actual node-surge and batch settings.
| Workload state | Budget example | Drain implication | Required action |
|---|---|---|---|
| 3 desired, 3 healthy | maxUnavailable: 1 | One voluntary eviction allowed | Proceed with one-at-a-time proof |
| 3 desired, 2 healthy | minAvailable: 2 | No healthy pod can be evicted | Restore health before drain |
| 1 desired, 1 healthy | minAvailable: 1 | Drain blocked indefinitely | Add replica or approve controlled downtime |
| 5 desired with percentage | minAvailable: 80% | Rounding affects allowed count | Inspect status.disruptionsAllowed |
| Pod selected by two PDBs | Different allowances | All applicable budgets must permit eviction | Remove overlap or satisfy both |
Write budgets that are strict and satisfiable
Use minAvailable when the application has a known minimum serving or quorum count; use maxUnavailable when the controller's desired scale should determine tolerated loss. Only one may be set. The PDB task explains selector and controller considerations. Validate selectors against live pods and revisions. In policy/v1, an empty selector matches every pod in the namespace, while a null selector matches none, so a seemingly small formatting error can have wide consequences.
Budget from normal involuntary loss plus planned maintenance. If the objective requires surviving one failed replica while another is voluntarily evicted, provision enough replicas and domains for both. A budget of zero may be correct for a singleton that cannot tolerate disruption, but it also means routine drain cannot complete without an explicit downtime or failover procedure. Encode the honest constraint and give operators the approved path; do not publish an impossible availability promise and later override it ad hoc.
Decide how unhealthy pods affect drain
The PDB API offers unhealthyPodEvictionPolicy. The API reference defines IfHealthyBudget as the default and AlwaysAllow as permitting eviction of running but unready pods regardless of budget health, while healthy pods remain protected. Kubernetes disruption guidance recommends AlwaysAllow to support eviction of misbehaving applications during drain. Choose deliberately from workload recovery behavior.
AlwaysAllow helps remove a broken pod pinned to a draining node, but it does not guarantee the replacement becomes healthy. Before adopting it, test startup, data recovery, leader election, dependency access, and volume movement. Alert when a workload enters upgrade already below desired health. A maintenance process should never normalize unready replicas as harmless; the policy prevents deadlock while root-cause and service protection remain operator responsibilities.
Sequence node drains with replacement capacity
Cordon first, create or verify surge capacity, and confirm it has joined with expected labels, taints, networking, storage drivers, and DaemonSets. Drain one low-risk node with a timeout and capture every rejected eviction. The safe drain guide explains that kubectl drain marks a node unschedulable and evicts pods, while DaemonSets and unmanaged pods need explicit handling. Do not use --disable-eviction in a routine runbook because it bypasses PDB checks.
Wait for replacements to become Ready and for service-level signals to stabilize before advancing. Ordered StatefulSets, zonal volumes, anti-affinity, topology spread, scarce GPUs, and host ports can serialize progress beyond the nominal PDB. Match upgrade batch size to the smallest application disruption allowance, not the node pool's maximum parallelism. Pause when drains exceed their expected duration; repeated retries are evidence to inspect, not noise to hide.
| Symptom | Likely cause | Safe response | Unsafe shortcut |
|---|---|---|---|
| disruptionsAllowed is zero | Budget or current health at minimum | Restore health or adjust planned capacity | Delete pod directly |
| Replacement remains Pending | No capacity or conflicting constraints | Add compatible capacity and inspect events | Increase drain parallelism |
| Unready pod cannot evict | Default unhealthy policy plus unhealthy app | Fix app or review AlwaysAllow | Disable eviction checks |
| Volume remains attached | Storage detach or zone constraint | Wait and inspect CSI operations | Force node deletion |
| Drain waits on termination | Long grace or stuck finalizer | Use workload-specific escalation | Apply universal zero grace |
Verify availability and preserve rollback
Track PDB status generation, expected pods, current healthy, desired healthy, and disruptions allowed alongside node and application telemetry. Verify request success, latency, queue age, leader stability, replication health, error budgets, and data integrity. Kubernetes-level readiness can pass while a quorum is rebuilding or a cache is cold. Define wave acceptance from service behavior and keep old node images or provider rollback mechanisms available within their supported window.
After each pool or zone, reconcile versions, nodes, pods, PDBs, failed evictions, and provider events. Update budgets when a workload's replica model changes; stale percentages and selectors are operational debt. Run a pre-upgrade disruption rehearsal regularly, not only when a release deadline arrives. Frequent small tests expose impossible budgets while there is time to repair architecture rather than negotiate an emergency bypass.
Run the upgrade with explicit command roles
Assign an upgrade commander, platform executor, application contacts, storage or database contact, and communications owner. The commander controls wave start and stop; application owners attest service health and decide on workload-specific risk; the executor does not unilaterally weaken a PDB to meet a schedule. Record current versions, target versions, provider rollback boundary, node-pool order, surge, maximum unavailable nodes, maintenance window, and stop thresholds in one change record.
Use a preflight query that fails the change when PDB observed generation is stale, disruptions allowed are unexpectedly zero, selected pods lack a controller, nodes already carry pressure, replacement capacity is unavailable, or critical workloads are below health. Inspect topology by node and zone. Confirm backup and restore evidence for stateful systems, but do not treat backup as permission to ignore live availability. Notify owners of workloads whose termination grace could consume most of the window.
During a wave, timestamp cordon, first eviction, each rejected eviction, replacement scheduling, readiness, service acceptance, and node deletion. These intervals distinguish a PDB block from image pull, scheduler constraint, volume detach, startup, or application warmup. Stop on service-objective breach, unexplained data inconsistency, replacement failure, repeated unhealthy pods, or drain duration beyond the plan. A timeout should return control to humans; it should not automatically escalate to force.
Close the change only after new nodes are Ready, old nodes are gone or explicitly retained, workloads are balanced, budgets have recovered, telemetry is complete, and provider status is healthy. Review exceptions and every bypass request even when it was denied. Convert recurring blocks into architectural work such as additional replicas, faster startup, corrected selectors, storage mobility, or a realistic downtime procedure. This is how upgrade evidence improves the next maintenance cycle rather than resetting to institutional memory.
Account for control-plane and node-version skew explicitly. Upgrade API clients, admission webhooks, CSI and CNI components, metrics adapters, and operators in a tested order, because a workload can be disruption-ready while an add-on is incompatible with the target release. Run API deprecation and manifest checks before the window. PDB success proves bounded eviction; it does not prove that replacement pods or controllers work correctly against the new Kubernetes version.
Key takeaways
- Treat a blocked eviction as a safety signal and diagnose its arithmetic and dependencies.
- Validate selectors, replicas, health, topology, storage, and spare capacity before maintenance.
- Choose unhealthyPodEvictionPolicy from tested recovery behavior.
- Drain one bounded wave at a time and wait for service recovery, not only pod readiness.
- Avoid direct deletion and eviction-bypass flags in ordinary upgrade procedures.
Frequently asked questions
Does a PDB protect against node failure?
No. It limits API-initiated voluntary eviction. Replicas, failure-domain placement, recovery, and capacity protect against involuntary outages.
Can a PDB be changed temporarily for an upgrade?
Yes, through an approved risk decision with an expiry and rollback, but first determine whether adding capacity or restoring health can satisfy the existing availability objective.
What should a single-replica workload do?
Add a tested replica when the application supports it, or define an explicit downtime and recovery procedure. A strict budget cannot manufacture redundancy.
Conclusion
Upgrades and disruption budgets are compatible when availability has been made schedulable. Write honest budgets, restore health before maintenance, provide compatible surge capacity, and let the Eviction API enforce each bounded step. The goal is not a drain command that always finishes; it is a cluster upgrade whose workload evidence shows that planned disruption stayed within the service's declared tolerance.