A Kubernetes Deployment in production is more than a desired replica count and a container image. It is a contract among the workload, scheduler, service, dependencies, operators, and users. The Kubernetes Deployment documentation describes declarative updates and rollout history; the probe guidance distinguishes startup, liveness, and readiness signals. Production safety depends on making those signals truthful and connecting them to capacity, disruption, and business behaviour.
Begin with the service journey: what does a ready pod need to do, how much traffic can it accept, and what should happen when a dependency is slow? Edilec’s deployment rollback guide, blue-green deployment guide, and Kubernetes deployment checklist provide adjacent context. The workload definition should remain explicit about identity, configuration, telemetry, data compatibility, and recovery.
Define the workload contract
Record the image, ports, dependencies, startup time, steady-state capacity, graceful termination period, resource requests and limits, disruption tolerance, security context, and observable success signal. Decide whether the workload is stateless, owns a partition, consumes a queue, or participates in a transaction. A Deployment is a good fit for many stateless services, but its controller cannot make a stateful application’s migration or idempotency safe by itself. The contract should tell operators which symptoms mean not ready, broken, degraded, or overloaded.
| Contract area | Question | Production evidence |
|---|---|---|
| Readiness | When can this pod safely receive traffic? | Probe, dependency rule, and endpoint observation. |
| Liveness | What failure can a restart actually repair? | Failure test and restart outcome. |
| Capacity | What load fits the requested resources? | Load test, saturation signal, and scaling rule. |
| Termination | How does the pod finish work before exit? | Drain, grace-period, and in-flight request test. |
| Disruption | How many replicas can be unavailable safely? | Budget, maintenance simulation, and user impact. |
Make health probes truthful
Startup probes protect slow initialization from premature liveness decisions. Readiness should answer whether the pod can serve the particular traffic it receives, not merely whether a process is listening. Liveness should be reserved for a failure that a restart can repair; a probe that fails because a database is temporarily slow can create a restart storm. Test probe timeouts, dependency outage, overload, cold start, and graceful shutdown. A green process check is not evidence that the user journey is healthy.
Size requests, limits, and scaling together
Resource requests influence scheduling and capacity planning; limits influence containment and failure behaviour. Use representative load to estimate CPU, memory, network, ephemeral storage, and dependency consumption. The Kubernetes resource-management documentation explains the mechanics, but the appropriate values come from workload evidence. Watch for memory pressure, throttling, queue growth, and noisy-neighbour effects. Autoscaling without a stable signal can amplify a dependency failure, so define scale-up, scale-down, and saturation behaviour together.
| Mistake | Symptom | Fix |
|---|---|---|
| No requests | Pods are scheduled into an unsafe capacity estimate. | Set evidence-based requests and review node headroom. |
| Liveness checks dependency | A dependency outage causes restart storms. | Keep liveness local and handle dependency failure explicitly. |
| Readiness is too shallow | Traffic reaches pods that cannot complete work. | Include the relevant service contract and observe endpoint state. |
| Disruption budget is assumed | Maintenance removes too many replicas at once. | Test budget, eviction, and actual user tolerance. |
| Termination is abrupt | Requests or queue items are duplicated or lost. | Drain, stop intake, and reconcile in-flight work. |
Plan the rollout and rollback
Use a staged rollout with a known revision, compatible configuration and data, health gates, and an observation window. Confirm that old and new pods can coexist when the controller updates replicas. Define maximum unavailable and surge behaviour in relation to traffic and capacity. Keep rollback authority and data reconciliation explicit. A Deployment can return to an earlier image, but it cannot automatically undo a schema change, external side effect, or already-published event.
Design for voluntary and involuntary disruption
The Kubernetes Pod disruption guidance explains how voluntary disruptions and disruption budgets interact; it does not make a service resilient to every failure. Spread replicas across appropriate failure domains, keep capacity for maintenance, and verify that a budget reflects user tolerance rather than a desired number. Exercise node drain, zone loss, deployment surge, preemption, and dependency degradation. Observe whether queues, retries, and client timeouts multiply the original disruption.
Make the release observable
Carry version, pod, namespace, workload, and request context into metrics, logs, and traces without exposing sensitive data. Monitor readiness transitions, restart count, rollout progress, saturation, error rate, latency, queue age, dependency health, and user-facing completion. A deployment dashboard should answer which version served an affected request and what changed before the symptom. Keep a saved query and a runbook for partial rollout, stuck termination, failed image pull, and a service that is healthy at pod level but broken at the edge.
Rehearse one release with a slow-starting container, a failing dependency, and a drain during active traffic. The operator should be able to distinguish not-ready from dead, identify whether the controller is making progress, pause the rollout, preserve evidence, and restore service. If the team relies on manually editing a live object to recover, capture that as design debt and move the correction into declared configuration or the runbook.
Review the deployment as a workload contract
A production review should compare the declared Deployment with the workload’s real behaviour. Inspect container startup, readiness transitions, request and limit values, termination sequence, replica placement, disruption budget, service selectors, autoscaling signal, dependency timeouts, and user-facing completion. Review both normal operation and the conditions that make the pod look healthy while the service is failing. This catches a common gap: infrastructure evidence is present, but no one has stated what makes the business operation complete.
For a queue consumer, readiness may mean that the process can accept work, its lease or partition is valid, and its downstream dependency is within a safe operating envelope. Liveness should not restart the consumer simply because the queue is empty or a dependency is temporarily slow. Termination should stop new intake, finish or return the current item, and preserve an idempotency record. These decisions are more specific than copying a probe template and should be tested with the service owner.
Use a recurring review after image, runtime, dependency, traffic, or capacity changes. Compare restart and eviction patterns with latency, errors, queue age, and customer outcomes. Revisit resource requests when workload shape changes rather than waiting for an outage. Keep one rehearsal for node drain or dependency failure in the operational calendar. When a responder has to change a live probe or scale setting manually, capture the evidence and convert the lesson into declared configuration, a safer default, or a documented exception.
Test the states between healthy and failed
Production workloads often fail in the middle: a pod is alive but cannot reach its dependency, a process has started but has not loaded configuration, a queue consumer has a lease but cannot commit, or a replica is ready while the service has no spare capacity. Design probes and rollout gates for these states. Use readiness to protect traffic, application logic to protect work, and liveness only when restart is a sensible repair. Observe transitions rather than only current values so operators can see whether the system is recovering or oscillating.
Capacity reviews should include the rollout itself. A Deployment may temporarily require surge capacity while old pods drain, and a disruption budget may preserve replica count without preserving dependency capacity. Test the worst overlap of rollout, autoscaling, node maintenance, and a slow downstream service. If the cluster cannot sustain that combination, state the limit and schedule releases accordingly. Honest constraints are safer than a manifest that appears highly available but fails under the exact change it is meant to make safe.
Kubernetes readiness is an operational agreement, so the service owner should approve what it means to admit or remove traffic. Platform owners can provide controllers, scheduling, and observability, while application owners define dependency and completion behaviour. Security owners review service accounts, images, and policy. This division helps prevent the platform from silently becoming the owner of business availability and gives responders a clear route when a pod is technically healthy but the service journey is not.
The workload contract should be understandable to someone who did not write the manifest. Explain why a probe has its threshold, what resource request represents, how a pod drains, which replicas may be disrupted, and which user signal proves success. Keep examples for slow startup, dependency degradation, and termination. This makes reviews faster and reduces the temptation to copy a generic deployment template. It also helps incident responders decide whether to change the application, controller, service, node capacity, or dependency rather than repeatedly restarting pods that are behaving according to a flawed contract.
Key takeaways
- Define a workload contract that includes readiness, capacity, termination, disruption, and user outcome.
- Keep startup, liveness, and readiness probes distinct and test their failure modes.
- Size resource requests and scaling from representative load and dependency behaviour.
- Treat data compatibility and rollback as part of the release, not controller details.
- Observe versions, transitions, saturation, and business completion throughout rollout.
Frequently asked questions
Question: Are readiness and liveness probes interchangeable? Answer: No; readiness controls traffic eligibility, while liveness indicates whether a container should be restarted. Question: What makes a Kubernetes rollout production-ready? Answer: Honest probes, adequate capacity, disruption planning, observable progress, and a tested rollback decision.
What makes a Kubernetes Deployment ready?
Answer: The workload has truthful probes, resource expectations, scaling behavior, disruption policy, observable rollout signals, and an owner who can recover it.
Why can healthy probes still hide a bad release?
Answer: A process can be alive while serving errors, stale data, slow responses, or an incompatible dependency. Pair probes with user-facing and dependency checks.
When should a rollout stop?
Answer: Stop when availability, latency, error rate, capacity, or business outcomes cross the predeclared threshold, or when the controller cannot establish the intended state.
Should liveness check every dependency? Usually no; liveness should identify a local condition a restart can repair. What should readiness check? The conditions required for the pod to serve its assigned traffic safely. Does a disruption budget guarantee availability? No; it limits some voluntary disruptions and must be paired with capacity, topology, and application behaviour. Can a Deployment roll back every release? It can revert a workload revision, but teams must handle data, messages, external effects, and configuration compatibility.
Conclusion
Kubernetes deployments become production-ready when the platform signals reflect the real workload contract. Define truthful health, evidence-based capacity, compatible change, disruption tolerance, and an observable rollback path. The controller then becomes part of a dependable operating system rather than a substitute for one.