Kubernetes Deployments in Production: Health Signals and Rollback
A Kubernetes Deployment can report successful Pod creation while the service remains unsafe for users. Production readiness means a compatible revision receives traffic, capacity is intentional, unhealthy replicas are withheld, routine disruption stays within tolerance, and an operator can pause or restore the release from evidence. The controller manages ReplicaSets and rollout state, but it cannot decide whether a degraded application is acceptable. Before promotion, define the user-facing success signal, startup behavior, resource budget, dependency assumptions, rollback trigger, and accountable owner.
Define the release promise beyond the manifest
Describe what the Deployment must prove in normal and difficult conditions. A public API may need two ready replicas across failure domains, a maximum error rate during rollout, and a rollback if the new revision cannot serve representative requests. A batch worker may need one active consumer, enough queue capacity, and a drain policy rather than a simple availability target. Record image digest, configuration version, service account, resource requests and limits, probe behavior, disruption policy, observability, and ownership. Kubernetes documentation explains that a Deployment rollout is driven by changes to the Pod template; treat every template change as a revision with an impact review. This helps a team distinguish an intended release from an accidental restart caused by an unrelated configuration edit.
| Production question | Decision | Evidence |
|---|---|---|
| When can traffic arrive? | Readiness is true only when the app can serve its required path. | Probe test, dependency policy, and endpoint state. |
| How much capacity is required? | Set requests from measured demand and limits from safe containment. | Load test, utilization history, and scheduling result. |
| When is a rollout unsafe? | Define error, latency, saturation, or business failure thresholds. | Release dashboard and pause/rollback rule. |
| What may maintenance remove? | Set disruption tolerance based on real replicas and quorum. | PDB, topology policy, and drain rehearsal. |
Ask each probe a different health question
A startup probe answers whether a slow-starting process has completed initialization; a readiness probe answers whether a Pod should receive traffic; a liveness probe answers whether the process is stuck and should be restarted. They are not interchangeable. Kubernetes warns that an over-aggressive liveness probe can create cascading restarts, while a readiness probe that checks no meaningful dependency can send traffic to an application that will fail every request. Design endpoints around work the service can safely perform. A readiness check may verify local configuration and a critical dependency, but it should not trigger an expensive query on every interval. A liveness check should usually avoid declaring a temporary downstream outage as a process failure. Test probe timing during cold start, dependency timeout, overload, and graceful shutdown. The Kubernetes probe documentation provides the semantic baseline.
Probe success is only one part of release safety. Pair it with request-level metrics and a representative synthetic transaction. A Pod may be ready while a schema mismatch causes all checkout requests to fail; the readiness endpoint cannot know every business invariant. Conversely, a dependency outage may make readiness false for every replica and remove the last capacity, even though a degraded read-only response would be preferable. Decide whether the service fails closed, serves a bounded fallback, or remains available with a visible warning. If your deployment also changes credentials, use the secrets management guide to test refresh and rollback compatibility.
Reserve capacity for startup, traffic, and disruption
Resource requests influence scheduling and the workload's placement; limits constrain how much a container can consume. Kubernetes explains that memory limits can result in an out-of-memory kill when a container exceeds its cgroup limit, while CPU limits can throttle execution. A request copied from a developer laptop may understate cold-start or peak needs; a limit chosen without evidence may hide a leak until production. Measure steady state, startup, burst, and dependency wait. Set requests that let the scheduler make a credible placement decision and limits that prevent one workload from exhausting a node. Then test the behavior under contention, not just in an empty cluster. For a Java service, include heap sizing and native memory; for a Go worker, include queue burst and garbage-collection behavior.
| Signal | What it can reveal | Action |
|---|---|---|
| Pending Pods | Requests cannot be placed on available nodes. | Revisit capacity, requests, topology, or rollout size. |
| CPU throttling | The limit constrains work during burst or sustained load. | Compare latency and limit policy before increasing capacity. |
| Memory working set and OOM kills | The process exceeds safe memory or has a leak. | Inspect heap/native use, then adjust or fix the cause. |
| Readiness flaps | Dependencies, probes, or startup timing are unstable. | Pause rollout and analyze transitions rather than restarting blindly. |
Stage mixed versions against real user signals
A rolling update is a sequence of capacity and compatibility decisions. Set max unavailable and max surge so the cluster can run both old and new replicas, then verify that the service can tolerate the temporary mix. If the new version changes a database contract, application compatibility may matter more than Pod health: release an additive schema change first, then deploy readers, then writers, and remove the old path later. Pause a rollout when error rate, tail latency, saturation, queue age, or business success moves beyond the agreed threshold. Kubernetes retains revision history so a rollback can restore a previous Pod template, but rollback does not automatically undo a database migration, an external side effect, or a secret rotation. Document those boundaries and test a rollback with realistic traffic. Teams also need the blue-green deployment guide when simultaneous environments provide a safer compatibility boundary.
PodDisruptionBudgets protect against voluntary disruptions such as node drains; they do not guarantee availability during every failure and do not replace replicas spread across failure domains. The PDB documentation makes an important operational distinction: a budget limits how many Pods may be disrupted, while the workload and scheduler still need enough capacity to honor it. Set the budget from quorum and user impact, not a copied percentage. A three-replica stateless API may tolerate one unavailable replica; a two-replica service with one expensive startup may need a different maintenance plan. Exercise node drain, cluster upgrade, autoscaling, and a rollout at the same time. A policy that looks correct on paper can still block maintenance or fail to protect a service if labels do not match the intended Pods.
Use rollout evidence at the request boundary
Create a release view that joins Deployment revision, image digest, configuration version, replica availability, probe state, resource behavior, request outcomes, and dependency health. Review the normal path and one uncomfortable case after every material release. Signals worth tracking include time to readiness, rollout duration, rollback frequency, unavailable replicas, probe failures by type, throttling, OOM kills, eviction, queue age, and user-visible error rate. Avoid treating a successful controller update as success. The service owner should answer whether users completed the intended action, whether any manual compensation was needed, and whether the new revision changed cost or capacity behavior. Keep enough evidence to compare the old and new release without forcing an incident investigator to reconstruct it from scattered logs.
Exercise the release that should make you pause
Before a major release, run a timed drill with the people who will actually operate it. Start with a new image and configuration revision, then introduce a slow startup, a dependency timeout, a node drain, and a controlled rollback. Ask the operator to identify the revision, determine whether traffic should continue, and show which signal crosses the rollback threshold. Record time to readiness, time to diagnosis, capacity during the mixed-version period, and any manual action needed to restore service. The exercise is especially valuable for teams that inherited a chart or manifest because it exposes assumptions that are otherwise hidden in defaults.
Use the drill to improve the release contract. If readiness removed every replica during a dependency outage, decide whether the endpoint is too strict or the service needs a fallback. If a PodDisruptionBudget blocked a maintenance window, revisit the replica and topology design rather than deleting the budget. If rollback restored Pods but not data compatibility, add a migration phase to the release. Keep the final evidence with the Deployment revision so the next change starts from a known operating point.
Related Edilec reading: serverless production changes, backup and restore readiness, and Docker architecture decisions add adjacent production boundaries.
Source context: the Kubernetes Deployment documentation defines revision and rollout behavior; probe guidance separates startup, readiness, and liveness; resource management explains requests and limits; and the PodDisruptionBudget guide explains voluntary disruption protection.
Key takeaways
- Production readiness is a service contract covering health, capacity, compatibility, disruption, and ownership.
- Use startup, readiness, and liveness probes for distinct questions and test them under failure and overload.
- Requests and limits are reliability inputs; measure startup, steady state, burst, and contention behavior.
- Rollouts need pause and rollback criteria tied to user outcomes, not only Pod state.
- A PodDisruptionBudget complements replicas and topology; it does not replace them.
Frequently asked questions
Should readiness check every dependency?
Check the dependencies whose failure should remove the Pod from traffic, but keep the check cheap and bounded. If a degraded or cached response is valid, do not make readiness depend on an optional system. Pair the probe with request-level monitoring because no single endpoint proves every business path works.
How should we choose CPU and memory limits?
Use measured startup, steady, and burst behavior, then test under node contention. Memory limits should account for runtime overhead, not just heap; CPU limits should be evaluated with latency and throttling. Revisit values after code, traffic, or dependency changes.
Does rollback make a release safe?
It can restore a prior Pod template, but it cannot automatically reverse data migrations, external messages, or every configuration change. Define compatibility and compensation steps before release, and rehearse rollback with the same observability used during promotion.
For one checkout API, put the image digest, schema compatibility rule, startup budget, readiness dependency, requests and limits, replica topology, disruption budget, and rollback threshold in the release record. Rehearse a mixed-version rollout with a deliberately slow startup and a dependency timeout. The operator should identify which Pods may receive traffic, explain why the rollout is paused, and show whether the previous revision can serve data written by the new one. Compare controller state with checkout success, tail latency, queue age, and error class before declaring the release safe.
Kubernetes Deployments in Production: a decision you can operate
A production Deployment is trustworthy when controller state, user-facing health, capacity assumptions, compatibility, and recovery evidence agree. Treat each Pod-template change as a named revision, separate startup, readiness, and liveness questions, and test rollback against data and external effects as well as ReplicaSet state.
The next operator should be able to pause, restore, or continue with evidence. That is the standard that turns a green controller status into a release a production team can responsibly own.