Few operations frighten platform teams as much as a Kubernetes cluster upgrade. The control plane changes behind your back, deprecated API versions vanish, and every node gets drained and replaced while live traffic is still flowing through it. Most upgrade outages are not caused by Kubernetes itself but by the assumptions teams bake into their workloads and node pool configuration: pods that cannot tolerate eviction, deployments pinned to removed API versions, and clusters running so far behind that a single upgrade crosses multiple minor versions at once. A zero-downtime upgrade is not luck; it is the product of version skew discipline, a deliberate node pool strategy, and a rollback plan you have actually rehearsed. This guide walks through each of those pieces.
Why upgrades fail in production
Upgrade failures tend to cluster around three root causes. The first is API deprecation: Kubernetes removes API versions on a fixed schedule, and workloads still using a removed version — such as the PodSecurityPolicy APIs removed in Kubernetes 1.25 — start failing the moment the new API server stops serving them. The second is eviction pressure: a node upgrade drains each node in turn, and if a Deployment has a single replica with no disruption budget, the drain terminates the only pod and the service goes dark for the length of the upgrade. The third is version skew, where components drift so far apart that they stop talking to each other correctly. Knowing which of these is your biggest exposure is the first step, and the official deprecation guide lists every removal with its target release so you can audit your manifests before you upgrade rather than discovering the breakage at 2 a.m.
Respect the version skew policy
The Kubernetes version skew policy defines how far apart components are allowed to be. It exists because the API server, the kubelet, and client tooling evolve at slightly different speeds, and the project only guarantees compatibility within a narrow window. In practice this means you cannot jump a cluster from 1.26 straight to 1.30 in a single hop — you must step through each minor version so no component is ever more than one version out of sync. On managed platforms such as GKE, the control plane and node version are upgraded by the platform, but the same discipline still applies to client tooling, admission webhooks, and any in-cluster controllers you operate yourself. A controller compiled against a newer client library than the API server can emit requests the server does not yet understand, producing exactly the kind of intermittent, hard-to-diagnose failure that upgrades are infamous for.
- kube-apiserver must be within one minor version of the kubelet.
- kubectl must be within one minor version of the kube-apiserver.
- Upgrade one minor version at a time — never skip releases.
Zero-downtime node pool upgrades on GKE
On GKE, the control plane and node pools upgrade separately, and the node pool upgrade is where most of the operational risk lives. GKE offers two strategies: surge upgrades, the default, which add replacement nodes to a pool before draining old ones so capacity is preserved; and blue-green upgrades, which create a brand-new pool and migrate workloads into it. Surge upgrades are simpler and keep your existing node pool configuration intact, but they temporarily require extra node capacity and are bounded by the pool's configured maximum node count. Blue-green upgrades are cleaner for stateful or latency-sensitive workloads because the new pool is validated before traffic moves, and rollback is as simple as keeping the old pool alive until you are confident.
Surge upgrades and maintenance windows
A surge upgrade respects the pool's configured maximum size, so a pool already running at its maximum cannot surge — the upgrade stalls or falls back to draining existing nodes. Set the maximum to accommodate the surge, and schedule upgrades inside a maintenance window and exclusion period so they do not collide with peak traffic. GKE release channels — Rapid, Regular, and Stable — determine when your cluster is offered new versions, and choosing a channel is a policy decision about how quickly you accept change. A production cluster on the Stable channel receives fewer, better-tested version jumps than one on Rapid, at the cost of falling further behind upstream.
Blue-green upgrades for sensitive workloads
With a blue-green upgrade, GKE creates a second node pool at the target version, and you cordon and drain the old pool only after the new pool is healthy. This is the right choice when an application has long-lived connections, relies on pod anti-affinity that a surge would violate, or simply cannot tolerate being rescheduled during business hours. The cost is an extra pool's worth of nodes for the duration of the migration — capacity that your cluster autoscaler configuration needs to account for, or the migration will compete with normal scaling for headroom.
Protect workloads from eviction
The safest upgrade still drains nodes, and drains are voluntary disruptions. PodDisruptionBudgets tell Kubernetes how many replicas of a workload must remain available during such disruptions, and kubectl drain honors them by waiting or refusing when the budget would be violated. A Deployment with three replicas and a PDB of minAvailable: 2 can lose one pod at a time, which is exactly what a rolling node upgrade produces. Without a PDB, a single-replica Deployment is the classic outage: the drain deletes the pod, there is no replacement, and the service is down until the new node is ready and the pod reschedules.
apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: minAvailable: 2 selector: matchLabels: app: web
Set minAvailable or maxUnavailable relative to your replica count and your availability target, not to an arbitrary number. Pair the budget with pod anti-affinity and topology spread constraints so that evicted replicas land on different nodes and zones, and you never drain three pods that happened to share a single rack. A budget that is too strict is also a problem: it can block a drain indefinitely and stall the whole upgrade, so tune it against how many replicas you can actually afford to run concurrently.
Pre-upgrade checklist
- Audit the deprecation guide and grep your manifests and Helm charts for API versions scheduled for removal in the target release.
- Back up etcd (or confirm Backup for GKE has a recent, restorable snapshot) before touching a live cluster.
- Confirm every Deployment either has a PodDisruptionBudget or genuinely tolerates eviction.
- Set the node pool maximum to accommodate a surge upgrade, and schedule the upgrade inside a maintenance window.
- Dry-run the upgrade on a staging cluster running the same version and release channel first.
- Prepare a rollback runbook and confirm the on-call engineer has the access to execute it.
Rollback and recovery
Even a well-planned upgrade can surface a latent bug — an admission webhook that misbehaves against the new API version, or a workload that fails health checks after rescheduling. Your rollback options depend on what changed. Control-plane changes on GKE are managed by the platform, and node-level regressions are handled by reverting the node pool to its previous version. The universal fallback is a working etcd backup: restoring cluster state is the difference between a short incident and a full rebuild.
Treat an upgrade like any other change with a blast radius: open a maintenance window, announce it, monitor error budgets during the rollout, and have your incident response playbook ready before you type the first command. If the rollout does go sideways, a rehearsed playbook — not improvised commands — is what gets you back to green.
Upgrades do not have to be quarterly all-hands events. With version skew discipline, a deliberate node pool strategy, disruption budgets on every workload, and a rehearsed rollback plan, most upgrades become routine maintenance windows that nobody notices. If your team wants a second pair of eyes before the next upgrade — including a review of your node pool configuration, PDB coverage, and rollback readiness — Secpros can audit your GKE clusters and return a short, prioritized action plan.