Every Kubernetes practitioner learns early that a Secret resource stores data as base64-encoded strings, not encrypted blobs. The API server stores Secrets in etcd, and while encryption at rest for etcd can be configured and is strongly recommended, the default Kubernetes distribution ships with Secrets stored as plaintext in the etcd database. More importantly, any subject with read access to a Secret in a namespace — via the Kubernetes API, via kubectl, or via a compromised pod's service account — receives the decoded value. For platform teams managing production clusters where API credentials, database connection strings, TLS private keys, and cloud provider tokens flow through Secrets daily, understanding the gap between Kubernetes' default Secret handling and what constitutes production-grade secrets management is essential infrastructure hygiene.
The attack surface created by default Secret handling is broader than many teams realize. When a Secret is mounted as a volume in a pod, the kubelet writes the decoded value to a tmpfs on the node. The data is not persisted to disk under normal operation, but it exists in node memory and is accessible to any process with sufficient node-level privileges. If an attacker compromises a node — through a container escape, a kernel exploit, or a misconfigured privileged pod — every Secret mounted on that node is potentially exposed. More subtly, Secrets injected as environment variables are visible in /proc on the node, in crash dumps, and in any logging or monitoring system that captures process environments. The container runtime itself has access to these values. This is not a Kubernetes design flaw per se; it reflects the reality that Secret delivery to a workload necessarily involves making the value available to the container, and any component in that delivery chain with sufficient privilege becomes a potential exfiltration point.
Encryption at rest is the first and most fundamental hardening step that every production cluster should implement. Kubernetes supports encrypting Secret data in etcd using a configured encryption provider — AES-CBC, AES-GCM with a locally managed key, or integration with an external Key Management Service such as AWS KMS, Azure Key Vault, or Google Cloud KMS through the KMS plugin. With envelope encryption, the KMS plugin encrypts a data encryption key that in turn encrypts each Secret, and the KMS never sees the plaintext Secret data. This approach also enables key rotation: a new DEK can be generated and re-encrypted against the same KMS key without re-encrypting all existing Secrets. The operational note that matters in practice is that enabling encryption at rest only protects new and updated Secrets; existing Secrets remain in their original encoding until rewritten. A post-enablement script that annotates and re-applies every Secret in the cluster closes this gap and should be part of any encryption rollout.
Encryption at rest is necessary but not sufficient. The more fundamental architectural shift is moving secrets out of Kubernetes entirely and into a dedicated secrets management system, then synchronizing only what individual workloads need into the cluster. External Secrets Operator has become the de facto standard for this pattern. ESO watches ExternalSecret custom resources, fetches the referenced secrets from an external provider — AWS Secrets Manager, Google Secret Manager, Azure Key Vault, HashiCorp Vault, or dozens of other backends — and creates a corresponding Kubernetes Secret in the target namespace. The Kubernetes Secret remains the delivery vehicle to the pod, so application code does not need to change, but the source of truth lives outside the cluster with full audit logging, rotation automation, and access controls that are independent of Kubernetes RBAC.
The External Secrets Operator model provides several architectural benefits that are difficult to achieve with native Secrets alone. First, the same secret value can be synchronized to multiple namespaces or multiple clusters from a single source of truth, eliminating the drift and inconsistency that occur when Secrets are manually copied across environments. Second, rotation can be handled at the provider level — AWS Secrets Manager supports automatic rotation through Lambda functions, Vault supports dynamic database credentials with configurable TTLs — and ESO will refresh the Kubernetes Secret on its configured refresh interval. This means application pods eventually receive rotated credentials without a redeployment, though careful handling of connection pooling and credential caching in the application layer is still required. Third, access to secret values can be enforced through cloud IAM policies at the provider level, adding a second, independent authorization layer that Kubernetes RBAC alone cannot provide.
HashiCorp Vault integration represents the most capable pattern for organizations with complex secrets management requirements. Vault's Kubernetes auth method allows pods to authenticate to Vault using their Kubernetes service account token, and Vault maps that identity to Vault policies that determine which secrets the pod can access. Combined with the Vault CSI provider, secrets can be mounted directly into pods as files without ever creating a Kubernetes Secret resource. The secret value flows from Vault to the CSI driver on the node to the pod's tmpfs volume, completely bypassing the Kubernetes API and etcd. For highly sensitive credentials — root database passwords, signing keys, payment processing tokens — this architecture eliminates the Kubernetes Secret as an attack surface entirely. The trade-off is operational complexity: Vault requires its own highly available deployment, unsealing procedures, and backup strategy, and application pods need the Vault agent sidecar or CSI driver injection configured.
For platform teams that want to improve secrets management without deploying Vault, the Sealed Secrets controller from Bitnami provides a pragmatic intermediate step. A developer encrypts a Secret locally using the controller's public key, producing a SealedSecret custom resource that can be safely committed to Git. The controller, running in the cluster with access to the corresponding private key, decrypts the SealedSecret and creates the standard Kubernetes Secret. The encrypted SealedSecret is safe for GitOps workflows because the private key never leaves the cluster. The limitation is that Sealed Secrets does not provide rotation, dynamic secret generation, or multi-cluster synchronization — it is a Git-safe transport mechanism rather than a full secrets management solution. Combined with External Secrets Operator for credentials that support provider-side rotation, Sealed Secrets fills the gap for static, cluster-specific secrets that do not benefit from an external provider.
Multi-tenancy adds another dimension to secrets management that platform teams must address. In a cluster shared by multiple teams or business units, namespace-level Secret isolation is provided by Kubernetes RBAC, but that isolation depends on every team correctly configuring their RBAC policies and every administrator avoiding overly broad ClusterRole bindings. A single misconfigured ClusterRole that grants Secret read access across all namespaces — or a developer who accidentally commits a kubeconfig with cluster-admin privileges — can expose every Secret in the cluster to anyone capable of authenticating. Network policies, pod security standards, and admission control all contribute to defense in depth, but the most robust protection is to ensure that the most sensitive credentials never exist as Kubernetes Secrets at all, using Vault CSI or a comparable mechanism that keeps credential values outside the Kubernetes control plane.
Observability of secrets access is the final piece that separates mature secrets management from ad-hoc practices. Kubernetes audit logs capture every API request that reads a Secret, including the requesting identity, the Secret name, the namespace, and the timestamp. Platform teams should configure audit policies that log all Secret reads at the Metadata level or higher and route those logs to a centralized SIEM. The External Secrets Operator emits metrics on synchronization status, refresh failures, and last successful sync timestamps, which can alert teams to expiring or misconfigured external secret references. Vault's audit devices provide a complete record of every secret access and can be configured to send events to a syslog server or file for compliance purposes. Together, these observability signals let platform teams answer the question that every security auditor will eventually ask: who accessed which credentials, when, and from where.
The practical path forward for organizations that have not yet hardened their Kubernetes secrets posture starts with three actions. First, enable etcd encryption at rest using the KMS plugin for your cloud provider and re-write existing Secrets. Second, deploy External Secrets Operator and migrate cloud-provider-managed credentials — database passwords, API keys, TLS certificates — to the appropriate secrets manager (AWS Secrets Manager, Google Secret Manager, or Azure Key Vault), replacing static Kubernetes Secrets with ExternalSecret resources. Third, audit existing RBAC configurations for broad Secret read access and narrow permissions to the minimum required for each workload identity. For organizations handling particularly sensitive credentials — payment data, healthcare information, cryptographic keys — evaluate Vault with the CSI provider to eliminate Kubernetes Secrets from the delivery chain entirely. The journey from base64 to production-grade secrets management is not a single migration; it is a progressive hardening that aligns secrets handling with the value and sensitivity of the credentials being protected.