Most platform teams that reach a few dozen workloads arrive at the same frustration: metrics live in one tool, logs in another, and traces in a third, each with its own agent, its own exporter, and its own idea of what a service is called. When an incident happens, the first thirty minutes are spent stitching signals across tabs instead of investigating. OpenTelemetry (OTel) exists to remove that overhead: one vendor-neutral standard for generating, collecting, and exporting metrics, logs, and traces, so your platform exports a single, well-tagged stream that any tool in your stack can consume. This guide is for engineering leaders and platform engineers who want to adopt OpenTelemetry on Kubernetes without rebuilding their whole observability estate, and it focuses on the decisions that actually move the needle.
Why a standard telemetry layer matters on Kubernetes
Kubernetes gives you an unusually good substrate for observability: every workload has labels and annotations, every log line has a namespace, and the control plane already emits structured data. The problem is not a lack of signals; it is that each signal ends up instrumented differently. One team's exporter follows a vendor's SDK, another team writes to stdout and greps it, a third ships traces through a bespoke middleware. OTel standardizes that going in: your application code emits telemetry through a single, language-neutral set of APIs and SDKs, and the OpenTelemetry Collector routes what comes out.
The collector is the piece most teams underestimate. It is a vendor-agnostic process that receives, processes, and exports telemetry, and on Kubernetes it is where you centralize routing, redaction, and sampling policy. Because it is code you own in your cluster, it is also the natural place to enforce a consistent instrumentation contract before anything reaches a backend. Treating OTel as the default on-ramp for new services means the platform never has to retrofit instrumentation later, and it keeps your tooling choices swappable.
Map the three signals to the right tooling first
Before you write any config, decide what each signal is for. A common failure is treating all three as interchangeable streams and dumping everything into one backend. They answer different questions, and Kubernetes shipping conventions make each one cheap to implement if you plan it.
- Metrics: the health of the system over time. Use Kubernetes and application metrics for HPA, capacity, and golden-signal review. Wire them through a Prometheus-compatible time series store.
- Logs: the record of what a process actually did. Route structured logs from the node or daemonset, keep them tagged by workload, and retain them for the window your compliance or debugging actually needs.
- Traces: how a single request moves across services. Sample at the edge, keep trace context propagating via W3C headers, and store only what you realistically replay for latency debugging.
- Correlation metadata: namespace, deployment, pod, and the explicit resource attributes you attach, so a log line can point at the trace that caused it and both can join to a metric.
A useful shorthand is a small mapping table your team can paste into its runbooks: for each signal you choose a collection method, a processing rule, and a destination. Keep the destinations few; every extra sink is a new debugging surface and a new operational cost.
Run the OpenTelemetry Collector in cluster
The collector ships in two common roles: the agent daemonset that runs on every node and scrapes node-level telemetry, and gateway deployments that aggregate, filter, and forward the platform-wide stream. Start with a daemonset for host metrics and a single gateway receiver for the app data, and keep the pipeline explicit in a config you can review in code.
A minimal gateway config that accepts OTLP over the cluster network and forwards to a Prometheus-compatible backend looks like this:
receivers: otlp: protocols: grpc: http: processors: batch: send_batch_size: 1024 memory_limiter: check_interval: 5s limit_percentage: 80 exporters: otlphttp/prometheus-compatible: endpoint: http://prometheus-gateway:8080 service: pipelines: metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlphttp/prometheus-compatible]
Scope the collector so it stays a thin router
The fastest way to make the collector a bottleneck is to give it too many jobs. Keep signal-specific transformation in receivers and processors that are small and reviewed, disable components you do not use, and pin resource limits so a telemetry surge cannot squeeze application pods. A good rule of thumb is that the collector should route and redact, not store: durable storage and query stay in your time series and log backends.
Correlate metrics, logs, and traces with consistent metadata
Interoperability is the entire point of OTel, and it only holds up if your attributes are consistent. The trace context needs to flow through the same namespace and workload labels that your logs carry, otherwise the three streams stay separate. Standardize a small set of resource attributes at the collector entry: cluster, namespace, deployment, and instance. Define them once, document them once, and reject telemetry that lacks them.
When a request crosses services, propagate context with the W3C Trace Context headers and make sure every service forwards them. This is a code-level change at each network boundary, but it is the step that lets an engineer open one trace and see the latency budget across the whole request path, which is exactly the outcome the platform was built to provide. Treat missing propagation as a release-blocking bug in the first quarters of adoption.
Wire telemetry into SLOs and alerting
Collecting telemetry is only half the work; the value shows up when metrics become decisions. The Google SRE approach to service level objectives gives you the language for that: define a small number of service level indicators, set an objective for each, and alert on approaching the objective rather than on every noisy threshold crossing. Pick error rate and latency for the services users actually depend on, and keep the SLO count small enough that a team can act on every one.
With a metrics-compatible exporter in front of you, SLO calculations become queryable recording rules. A 30-day availability SLO is a windowed error-rate ratio, and you can alert on the error budget burn rate so on-call gets paged only when the objective is actually at risk. This is where OpenTelemetry returns its investment: the same standardized metrics let you build the same budget alerts across every team without bespoke exporters.
Begin with a small SLO set, not a dashboard for everything
Resist the urge to put a dashboard in front of every metric on day one. Start with three to five user-facing services, define two or three SLIs each, and treat the dashboards as the byproduct of SLO questions rather than the goal. As the team gains confidence, expand the set service by service. Alerting on an SLO you can actually control produces calmer on-call and fewer false pages than a wall of gauge thresholds.
A pragmatic rollout checklist
This checklist keeps an OTel rollout honest and reversible, and it is worth revisiting at the end of every quarter.
- One standard telemetry on-ramp for new services, with the collector config reviewed in the same PR as the instrumentation.
- Resource attributes (cluster, namespace, deployment) enforced at the collector entry, with missing-attribute telemetry rejected.
- A single exporter path per signal, with no ad-hoc backends for a single team.
- Trace context propagation verified across the main request path as a release gate.
- A small SLO set for the top three services, with alerting on error-budget burn rather than raw thresholds.
- Collector resource limits and a load test that proves telemetry volume cannot starve application pods.
If you already run HPA on Kubernetes metrics, adding a standardized telemetry layer does not change how autoscaling works, it simply gives your HPA and your dashboards the same, consistently tagged source of truth rather than two divergent views of the cluster.
If your team has an observability estate that grew organically, with three tools doing overlapping jobs and no standard for how new services emit telemetry, Secpros can review your Kubernetes, GKE, or CI/CD setup and return a short prioritized audit plan: which collector topology to adopt first, which SLOs to put in front of on-call, and how to sequence the migration so no team is blocked while you standardize.