Production Secrets Management: Access, Rotation, and Recovery
A development secret may be inconvenient; a production secret can become an outage, privilege escalation, or breach. Moving secrets management into production raises the proof required at every boundary: the value must be scoped to a real workload, protected in storage and transit, absent from source and logs, replaceable without unsafe improvisation, and covered by an incident path. Treat the credential as an operational dependency with an owner, blast radius, change window, and recovery objective.
Name the production secret's consequence
Write down which environments are production, which identities may cross the boundary, and which operators can approve emergency access. Separate build, deploy, runtime, support, and security roles. A CI runner that builds an image should not automatically read a production payment credential; a support engineer may need to trigger a rotation without seeing the plaintext value. For each secret, record owner, purpose, consumer, data classification, expected lifetime, rotation method, dependencies, and audit source. Kubernetes recommends restricting Secret access to specific containers, configuring encryption at rest, using least-privilege RBAC, and considering external stores. Those are not checkboxes until a reviewer can trace one secret from policy to API call to process memory and back to audit evidence.
| Production control | Decision to make | Proof |
|---|---|---|
| Identity binding | Which workload or human identity can request the value? | Role policy, authentication event, and denied-request test. |
| Storage protection | What protects the value in the manager, backup, and cluster control plane? | Encryption configuration, key owner, and restore test. |
| Delivery boundary | Does the process receive only the fields and duration it needs? | Runtime path, redaction test, and process-level access review. |
| Recovery readiness | How quickly can the value be revoked and replaced? | Runbook, owner, measured exercise, and service health result. |
Deliver access through a narrow runtime path
Production delivery should begin with identity, not a long-lived shared password. Use a workload identity tied to namespace, service account, role, environment, or cloud principal, then request one named secret. Limit permissions to read or rotate as appropriate; do not give every application the ability to administer the store. Keep retrieval and consumption visible through audit events without recording plaintext. A mounted volume, sidecar, CSI provider, or SDK call may be appropriate depending on refresh behavior and framework support. The design must answer what happens during provider unavailability, process restart, and partial rollout. If a service caches a secret, set a bounded lifetime and make stale state visible. The AWS Secrets Manager best-practices guidance is useful for separating secret access, application permissions, and service-side controls.
Production also raises the question of copies. A secret can appear in an image layer, Helm value, rendered manifest, backup, crash dump, shell history, tracing attribute, or support ticket. Scan repositories and build artifacts, but also inspect the paths that move configuration between systems. Kubernetes documents that base64 encoding is not confidentiality; if a Secret is stored in etcd or a backup, protect that control-plane path and test who can restore it. Use Docker image architecture as a companion when reviewing build context and image provenance. The objective is not to eliminate every transient in-memory copy; it is to minimize copies, restrict readers, prevent accidental persistence, and know the response when a copy is discovered.
Coordinate rotation across every consumer
A production rotation is a distributed change. The secret manager, dependency, application instances, connection pools, caches, and deployment controller may all observe the change at different times. Define the transition explicitly: create a candidate, update the dependency, verify the candidate, make it current, refresh consumers, monitor, and retire the old value. For a database, dual-user or overlapping credentials can reduce downtime if the database supports them, but the overlap needs an end date and an audit trail. For TLS, publish the replacement certificate chain before it becomes required. For signing keys, retain verification history for artifacts that are still valid. OWASP recommends automated rotation where possible and emphasizes that exposed values should be revoked quickly, then replaced through a repeatable least-privilege process.
| Change case | Hidden dependency | Safer release test |
|---|---|---|
| Database password | Connection pools hold old sessions. | Refresh, reconnect, drain, and verify read/write paths. |
| API token | Rate limits or IP allowlists are tied to the old identity. | Test authorization, quotas, and consumer inventory. |
| TLS certificate | Clients trust a chain or hostname not covered by the replacement. | Exercise handshake, renewal, and rollback before expiry. |
| Signing key | Verifiers need old public material for existing artifacts. | Verify old and new signatures across the retention window. |
Measure exposure, stale access, and recovery
A production program should detect both value exposure and suspicious use. Scan code, pull requests, artifacts, logs, tickets, and configuration exports; alert on access from an unexpected identity, environment, geography, or time; and retain enough metadata to investigate without exposing the secret again. Measure time to detect, revoke, replace, validate, and close the incident. Review false positives because an alert that pages constantly will teach operators to ignore a real exposure. OWASP's incident-response guidance calls for revocation, rotation, deletion from exposed systems, and lifecycle evidence such as who had access and when the value was previously rotated. Build that evidence into the provider, deployment pipeline, and service runbook rather than asking an incident team to collect it manually under pressure.
Run two exercises: a discovered secret and a failed manager. In the first, place a test credential in a controlled repository, verify the detector, revoke it, rotate the dependency, remove the copy, and document what cannot be erased from history. In the second, simulate provider unavailability and confirm whether the service fails safely, serves a bounded cached state, or needs an emergency credential path. Include a rollback: the prior application revision may request a previous secret shape or endpoint. Teams responsible for deployment rollback readiness should include this compatibility test, because a production rollback that restores code but not credential behavior is incomplete.
Approve one service boundary before scaling
Review a small sample each month: a high-impact credential, a recently rotated value, a denied access attempt, an emergency access event, and a retired service. Confirm owner, purpose, consumer list, expiry or rotation state, policy, audit coverage, and recovery evidence. Compare intended access with observed access. Remove permissions that are no longer needed, close temporary overlaps, and retire secrets whose service has been decommissioned. The review should produce decisions, not a larger spreadsheet: keep, rotate, restrict, replace, or delete. When the platform changes, revisit the trust boundary. A new cluster, runner, backup system, or observability integration may create a new copy or identity even when application code is unchanged.
Keep the access contract current after change
When a service enters production, ask for an access contract that can be reviewed without revealing the secret. It should name the workload identity, allowed secret paths, environments, operations, cache or refresh behavior, audit destination, and emergency route. Have the service owner sign off on purpose and impact, and have the platform or security owner sign off on enforcement and evidence. Treat an exception as a temporary change with an expiry date. For example, a migration job may need write access for one release window; the review should record when that role is removed and how removal will be confirmed.
The contract should also cover decommissioning. When a service or environment disappears, revoke its identities, remove secret versions that no longer have a consumer, update backups or exports where policy permits, and close monitoring rules. Retained material may still be needed for audit or recovery, but it should have a custodian and an access boundary. This is where production discipline differs from a one-time vault rollout: the system must make both creation and retirement explainable.
Related Edilec reading: blue-green deployment in production, log aggregation in production, and Docker image architecture provide useful neighboring control patterns.
Source context: Kubernetes Secret good practices covers production access and external stores; OWASP covers detection, rotation, and containment; NIST SP 800-57 describes key states and recovery; and AWS best practices provides managed-service controls.
Key takeaways
- Production secrets are dependencies with owners, blast radius, expiry, and recovery objectives.
- Bind access to workload identity and separate read, deploy, rotate, audit, and emergency roles.
- Protect manager, cluster, backup, build, log, and runtime paths; base64 is not encryption.
- Design rotation as a compatibility release with dependency tests, overlap limits, and retirement.
- Measure exposure response and run exercises for both leaked values and manager unavailability.
Frequently asked questions
Is a vault integration enough for production readiness?
No. Production readiness also requires scoped identity, protected storage and backups, controlled delivery, audit evidence, rotation compatibility, detection, and a tested response. A vault can enforce some of these controls, but the application and platform still determine how secrets are copied and refreshed.
Should services cache secrets?
Caching can reduce dependency pressure and support bounded continuity, but it extends lifetime and may keep a revoked value in memory. Set a maximum cache age, define behavior when refresh fails, and make the state observable. High-risk actions may need to fail closed rather than continue with stale authorization.
Who owns a production secret?
The service owner accepts purpose, consumer behavior, and business impact; the platform or security owner maintains controls and evidence. A provider team may operate the manager, but ownership cannot be delegated so far that nobody can approve rotation or response.
Use one payment worker as the approval specimen. Name its cloud or Kubernetes identity, exact secret paths, read versus rotate permissions, refresh method, cache lifetime, audit destination, emergency approver, and decommissioning action. Make an allowed read, a denied cross-environment read, a refresh after rotation, and a request after the service is disabled. Then replace a database credential while old connections remain open and follow a test token through detection, revocation, log containment, replacement, and evidence preservation.
Production Secrets Management: a decision you can operate
Production secrets readiness lives at the boundary between a credential and the service that depends on it. Keep purpose, identity, permission, delivery method, cache behavior, rotation compatibility, audit trail, and retirement action in one reviewable contract. Exercise ordinary change and abnormal exposure, including the choice to fail closed or use bounded cached state.
A vault can support those decisions but cannot make them. The durable outcome is a service that can explain why it has access today, change that access safely tomorrow, and recover when the value or control plane is compromised.