Series

Kubernetes observability

You can't fix what you can't see. This series covers the full observability stack: metrics, logs, traces, and dashboards. By the end you'll have the foundation to actually understand what your cluster is doing instead of just hoping for the best.

  1. 1

    Part 01 — Why observability is different from monitoring

    Here’s a scenario. It’s a Tuesday afternoon and your on-call phone goes off. The alert says “RabbitMQ queue depth exceeds 10,000 messages.” You look at your dashboard. Queue depth is definitely high. Memory usage…

    9 min read

  2. 2

    Part 02 — Metrics: Prometheus from scratch

    If you’ve spent any time around Kubernetes, you’ve heard of Prometheus. It shows up in helm charts, in monitoring stack recommendations, in job descriptions. It’s become so embedded in the Kubernetes ecosystem that…

    10 min read

  3. 3

    Part 03 — PromQL: querying your metrics

    Prometheus without PromQL is a database you can’t read. The metrics are there, the time series are accumulating, but until you can query them meaningfully you’re just storing numbers. PromQL is what turns those…

    9 min read

  4. 4

    Part 04 — Grafana: turning metrics into understanding

    A common failure mode in Kubernetes observability goes like this. The team installs kube-prometheus-stack. They get Grafana. They open the pre-built dashboards and see hundreds of panels showing CPU, memory, network,…

    12 min read

  5. 5

    Part 05 — Alerting: being woken up for the right reasons

    There’s a reliable way to tell if a team’s alerting is working well. Ask them how they feel about their on-call rotation. If the answer involves words like “exhausting,” “constant noise,” or “I stopped looking at…

    13 min read

  6. 6

    Part 06 — Logs: collecting, shipping, and querying

    Metrics tell you something is wrong. Alerts wake you up about it. Then you need to understand what actually happened — and for that, you almost always end up in logs. Logs are the detailed record of what your…

    11 min read

  7. 7

    Part 07 — Traces: following a request across services

    Here’s the scenario that distributed tracing exists to solve. A user reports that checkout is slow — sometimes five seconds, sometimes twelve. Your error rate is fine. Your metrics look normal. CPU is healthy. Memory…

    13 min read

  8. 8

    Part 08 — The Grafana stack: putting it all together

    The last six posts have covered the three pillars individually — Prometheus for metrics, Loki for logs, Tempo for traces. Each one is useful on its own. But the real leverage comes from using them together, connected…

    13 min read

  9. 9

    Part 09 — Observability in practice: debugging real incidents

    Everything in this series has been building to this. You have Prometheus scraping metrics, Loki collecting logs, Tempo storing traces, and Grafana connecting them. You have dashboards, alerts, and runbooks. Now let’s…

    13 min read