Every Kubernetes cluster has a single source of truth: etcd. It stores every Pod, Service, Deployment, Secret, ConfigMap, RBAC rule, and namespace definition that makes your cluster what it is. When etcd is healthy and backed up, it is a quiet workhorse. When it fails without a recoverable backup, the result is catastrophic — not a partial outage, but a total loss of cluster state. This guide explains how to protect etcd on Google Kubernetes Engine using Backup for GKE, snapshot-based approaches for self-managed clusters, and the disaster recovery procedures that turn a backup from a file on disk into a working cluster.
## Why etcd Backup Is Non-Negotiable
etcd is a distributed key-value store that holds the entire desired state and actual state of a Kubernetes cluster. The kube-apiserver reads from and writes to etcd for every API operation. If etcd loses its data — through disk corruption, accidental deletion, a misconfigured snapshot, or a control plane failure that outlasts the replication window — the cluster loses its identity. Workloads may continue running on existing nodes for a short time, but the control plane cannot schedule new pods, accept configuration changes, or recover from node failures. Rebuilding a cluster from scratch without an etcd backup means redeploying every application, recreating every Secret, re-issuing every certificate, and re-establishing every network policy by hand — a process measured in days, not hours.
## Backup for GKE: Google's Managed Solution
Google Cloud provides Backup for GKE, a managed service that automates configuration and workload state backups for GKE clusters. Once enabled, Backup for GKE periodically captures Deployments, Services, ConfigMaps, Secrets, PersistentVolumeClaims, and custom resources and stores them in a Cloud Storage bucket that you define. Backups are incremental after the first full snapshot, reducing storage costs while maintaining point-in-time recovery capability across the retention window.
Enabling Backup for GKE requires creating a BackupPlan resource that defines the backup schedule, retention policy, and encryption settings. A BackupPlan can target an entire cluster or specific namespaces, letting platform teams apply different protection levels to production, staging, and development environments. For production workloads, storing backups in a Cloud Storage bucket located in a separate region from the source cluster protects against regional failures — a practice that adds minimal latency to the backup job but dramatically improves recovery options during a region-wide event.
### What Backup for GKE Does Not Capture
Backup for GKE protects Kubernetes-native configuration and workload state, but it does not capture the underlying node configuration, cluster-level network policies managed outside of Kubernetes, or data stored in external databases that your workloads depend on. If your application state lives in Cloud SQL or Firestore, those systems need their own backup strategy. Similarly, any infrastructure provisioned through Terraform — VPC networks, firewall rules, IAM bindings — must be protected through state file management and version control, not Kubernetes backups. This is why <a href="/blog/terraform-state-management-locking-drift-detection/">Terraform state management</a> deserves the same rigor as etcd backup: losing either means losing the authoritative record of your infrastructure.
## Snapshot-Based Backups for Self-Managed Clusters
For teams running self-managed Kubernetes clusters on Compute Engine or on-premises, etcd provides a built-in snapshot mechanism through etcdctl. The etcdctl snapshot save command creates a point-in-time copy of the etcd data directory that can be used for both backup and disaster recovery. The Kubernetes documentation on operating etcd clusters recommends taking regular snapshots and storing them in a location that is physically separate from the etcd nodes themselves.
The following commands capture a snapshot from a healthy etcd member and verify its integrity:
```sh # Take a snapshot from a healthy etcd member ETCDCTL_API=3 etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db # Verify the snapshot integrity before uploading to remote storage ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20260726-0900.db --write-out=table ```
The snapshot file is a complete, consistent copy of the etcd database at the moment the command runs. After verification, copy it to an off-node location — Cloud Storage, an NFS share, or a separate disk — so that a node failure does not take the backup with it.
## Automating Backup Scheduling
Manual backups are a stopgap, not a strategy. Automating the backup schedule ensures consistency and removes the human error factor. For GKE, the BackupPlan resource defines the cron schedule, retention window, and storage location in a single declarative resource:
```yaml apiVersion: gkebackup.gke.io/v1 kind: BackupPlan metadata: name: production-backup-plan namespace: gke-backup spec: cluster: projects/my-project/locations/us-central1/clusters/production-cluster backupConfig: includeVolumeData: true includeSecrets: true backupSchedule: cron: "0 3 * * *" paused: false retentionPolicy: backupDeleteLockDays: 30 backupRetainDays: 90 ```
The backupSchedule.cron field uses standard cron syntax and supports timezone configuration for clusters that span regions. The retentionPolicy defines how long backups are kept and whether they are protected from accidental deletion through a delete lock. For self-managed clusters, the same schedule-based approach can be implemented with a Kubernetes CronJob that runs etcdctl snapshot save and uploads the result to remote storage — a pattern that is well-documented in the Kubernetes etcd operations guide.
## Disaster Recovery: Restoring from Backup
Having a backup is necessary but not sufficient — the restore procedure must be documented, tested, and executable under pressure. For Backup for GKE, restoration creates a Restore resource that applies the backed-up configuration to a target cluster. The target can be the original cluster, a different cluster in the same project, or a cluster in a separate project — a flexibility that supports both disaster recovery and environment cloning workflows.
Restoring through Backup for GKE follows a two-step process. First, create a RestorePlan that maps the backup to a target cluster and defines conflict resolution rules for existing resources — for example, whether to overwrite or skip resources that already exist in the target namespace. Second, apply the Restore resource that references both the backup and the plan. The restore operation preserves the original resource names, namespaces, and labels unless the RestorePlan specifies transformation rules. For self-managed clusters, restoring from an etcdctl snapshot involves stopping the kube-apiserver, running etcdctl snapshot restore on each etcd member, and restarting the control plane components — a procedure that the Kubernetes etcd documentation covers in detail with step-by-step instructions.
## Testing Your Recovery Plan
The only backup that matters is the one you have tested. Schedule a quarterly restore drill that validates backup integrity and measures the time from initiating a restore to having a functional control plane. Run the drill against a sandbox cluster rather than production to avoid disrupting live workloads, and treat any failure in the drill as a P1 incident — because in a real disaster, that failure is exactly what you will face under time pressure.
Document every manual step that the drill surfaces so the procedure becomes progressively more automated. Integrate the etcd restore procedure into your <a href="/blog/kubernetes-incident-response-playbook/">Kubernetes incident response playbook</a> so that the on-call engineer has step-by-step instructions rather than a search through documentation during an active incident. A restore procedure that requires tribal knowledge to execute is a restore procedure that will fail when the person with that knowledge is unavailable.
## etcd Disaster Recovery Checklist
The following checklist covers the essential steps for maintaining and testing etcd backup and disaster recovery. Each item should be verified at least once per quarter:
- Confirm that Backup for GKE is enabled for every production cluster and that the most recent backup completed without error. - Validate that backup storage is in a separate region from the source cluster to protect against regional outages. - Verify that the retention policy meets your recovery point objective; update it if business requirements have changed. - Run a restore drill on a sandbox cluster and measure the mean time to recovery from backup initiation to functional control plane. - Ensure that restore procedures are documented in a shared runbook and linked from the incident response playbook. - Audit that external state — databases, object storage, DNS records — has its own backup and restore process independent of Kubernetes. - Review the backup plan's encryption configuration and confirm that decryption keys are accessible from the recovery environment.
etcd backup is one of those investments that feels unnecessary right up until the moment it saves your team from a weekend-long recovery effort. If you want an independent review of your GKE backup configuration, disaster recovery readiness, or broader Kubernetes operations posture, Secpros can audit your clusters and return a prioritized remediation plan.
## Sources
The etcd backup and restore procedures referenced in this post are documented in the Kubernetes [Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) guide. The Backup for GKE service, including BackupPlan and RestorePlan resource definitions, is documented in the [Backup for GKE documentation](https://cloud.google.com/kubernetes-engine/docs/concepts/backups).