Kubernetes Audit Logs: Evidence for Incident Response

Every Kubernetes incident review reaches the same question: what happened to the API server between 02:10 and 02:40? Application logs show a pod restart. Metrics show a spike in 403 responses. Neither tells you that a service account token was used from an unfamiliar address to read a production Secret, or that an RBAC binding was widened four minutes before that read. Audit logging is the only record that answers those questions, and on GKE it is already running. The work is keeping it long enough, somewhere an investigator can reach, with the noise filtered out.

What Kubernetes audit logging actually records

The kube-apiserver writes an audit event for each request at one or more stages: RequestReceived, ResponseStarted, ResponseComplete, and Panic. An ordered list of rules, the audit policy, assigns each event a level, and the first rule that matches wins. None drops the event, Metadata records who called what without bodies, Request adds the request body, and RequestResponse adds the response.

That ordering is the whole design surface: a policy is a decision tree, not a filter appended at the end, so a broad Metadata catch-all above your Secret rules silently downgrades every Secret read you meant to capture.

On GKE you do not author that policy. GKE supplies its own to the API server and applies a second policy that decides what lands in Admin Activity and what lands in Data Access: create, update, and delete requests go to Admin Activity, while get, list, and updateStatus go to Data Access, and Data Access logs stay disabled until you enable them. Teams that assume the trail is complete by default find the gap mid-investigation, when the read that mattered was never written anywhere.

The questions an investigation actually asks

A useful trail is not "all events". It is the small set that answers four questions: who authenticated, from where, against which resource, and with what authority. GKE records the caller identity in the payload's authentication info and the caller address in request metadata, which is exactly what separates a compromised workload from a compromised human credential. Those two failures look identical in application logs and completely different here.

Reads are evidence too

Most teams log writes and stop there, because writes change state. The events that carry weight in a review are often reads: a Secret fetched by a workload with no reason to fetch it, or a pods/log call from a service account that normally stays inside its own namespace. GKE's default policy records Secret, ConfigMap, and token review requests at Metadata and excludes a documented set of kubelet and node requests — a sensible default, and the reason to inspect what your cluster emits before promising coverage.

Designing the policy around the events you will need

For self-managed clusters the policy is yours to write. Start by dropping known noise, keep Metadata for everything, and capture full bodies only where reconstruction matters: RBAC objects, workload controllers, authentication. Expressed as an ordered rule list, a defensible policy reads like this:

1. level: None for proxy watch traffic — users: ["system:kube-proxy"], verbs: ["watch"], resources: endpoints and services. Watch noise carries almost no investigative value.
2. level: RequestResponse for rbac.authorization.k8s.io roles, rolebindings, clusterroles and clusterrolebindings, plus apps deployments, statefulsets and daemonsets.
3. level: Metadata for secrets, configmaps and authentication.k8s.io tokenreviews, with omitStages: ["RequestReceived"] so each request is recorded once, at completion.
4. level: Metadata with no selector as the final catch-all. It has to be last, or it swallows every rule above it.

The ordering does the work, not the level values: the same rules in another sequence produce a log that looks complete and answers nothing. Take one real Secret read and confirm it appears at the level you intended.

Volume is a policy decision

RequestResponse on every resource is the fastest way to triple your logging bill and bury the signal. GKE's shipped policy is a useful reference for where Google drew the line: write requests at RequestResponse, reads at Metadata, Secret and ConfigMap access at Metadata, plus an explicit list of requests not logged at all, health checks and kubelet node status reads among them. Read that list before promising full coverage to an auditor. Knowing what your platform cannot tell you is part of the evidence.

Retention, export, and who may read the evidence

Audit logs are evidence only if they survive the investigation window, and log sinks cannot route entries retroactively: a sink created after an incident captures nothing from before it existed. That means an aggregated sink at folder or organization level, writing to Cloud Storage or BigQuery with its own retention, so a project owner cannot delete the record of their own project.

Access deserves the same care. In Cloud Logging, the Logs Viewer role reads Admin Activity, Policy Denied, and System Event logs but not Data Access logs stored in the default bucket; Private Logs Viewer is required for those. Engineers who debug day to day therefore do not automatically read every Secret access record. Keep the export in a separate project with its own IAM policy, and grant the sink's service account write-only access.

Querying the trail while it is still fresh

During an incident you have minutes, so keep the queries written down and tested against a known event. These three run first on a GKE cluster, always narrowed by cluster and time window:

1. What did this identity touch: gcloud logging read 'resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com%2Factivity" AND protoPayload.authenticationInfo.principalEmail="deploy@PROJECT_ID.iam.gserviceaccount.com"' --freshness=6h --format='table(timestamp,protoPayload.methodName,protoPayload.resourceName)'
2. What changed in the window: gcloud logging read 'resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com%2Factivity" AND protoPayload.methodName:("create" OR "update" OR "patch" OR "delete")' --freshness=2h --format=json
3. Was anyone inside a container: gcloud logging read 'resource.type="k8s_cluster" AND protoPayload.resourceName:"exec"' --freshness=24h

The first answers what an identity touched. The second produces the list you diff against your GitOps repository to find the write that did not come from a pipeline. The third is the one people forget: an interactive session leaves an audit entry but often no application trace, and it is a common path from a compromised pod to wider credentials. Those sessions belong with your <a href="/blog/securing-production-debugging-kubernetes/">production debugging practice</a>, not filed under logging.

Alerting on the events that should never happen

Retrospective queries are necessary and insufficient. A few events deserve a log-based alert, because their value is measured in minutes:

• A binding to cluster-admin or system:masters created by anything other than your platform pipeline.
• A Secret read by a service account that does not own the namespace it lives in.
• An interactive session opened into a pod in a production namespace outside a declared change window.
• A delete or deletecollection on a namespace, PersistentVolumeClaim, or custom resource definition.
• Authentication failures from a service account that has never failed before.

Keep the set narrow. A metric that fires on every Secret read is muted within a week, and a muted channel is worse than none because it looks like coverage. Who may grant that authority is a separate review, and the binding patterns that make escalation visible are covered in our <a href="/blog/kubernetes-rbac-least-privilege-guide/">Kubernetes RBAC least privilege guide</a>.

A checklist to run before the next incident

Work through this once a quarter, and before any audit or customer security review:

• Confirm Data Access audit logs are enabled for every cluster you might need to investigate, not just sensitive ones.
• Verify the export sink predates your retention window and writes to a project the cluster owners cannot administer.
• Document which roles can read Data Access logs, and confirm that list is shorter than the list of people who can deploy.
• Prove the filter syntax with one identity query and one change query against a known event, so nobody debugs query strings mid-incident.
• Record the accepted gaps in the platform's audit policy, with a reason, beside the risks you accept elsewhere.
• Alert on RBAC escalation and on interactive access to production namespaces, routed to on-call rather than a shared inbox.
• Run the query sequence alongside your restore drill, so the people who would use it under pressure have exercised it.

An audit trail nobody has queried is an assumption, not a control. If your team already runs our <a href="/blog/kubernetes-incident-response-playbook/">Kubernetes incident response playbook</a>, these queries and alerts belong in step one of it, with cluster names filled in.

If audit coverage keeps sliding behind feature work, Secpros can review your GKE audit configuration, export sinks, and alert rules, and return a prioritized checklist ranked by what each gap would cost you in a review.