Observability: seeing inside a running system
A design isn't finished when it's drawn; it's finished when you can tell whether it's working. In production you can't attach a debugger to fifty services. You rely on what the system tells you: logs, metrics and traces. This module covers what each is for, how to follow one request across many services, how to avoid drowning in data, and how to alert on what users feel rather than on every twitch of a CPU graph.
1. The idea
| Signal | What it is | Answers | Cost |
|---|---|---|---|
| Metrics | Numbers aggregated over time: request rate, error rate, latency percentiles, queue depth, CPU | "Is something wrong? Since when? How bad?" Cheap to keep for months; drive dashboards and alerts | Low per data point, but grows with every label combination (cardinality) |
| Logs | Timestamped records of events, ideally structured (JSON) with fields like user_id, order_id, trace_id | "What exactly happened to this request or user?" | High: every line is stored and indexed. Sample or tier the noisy ones |
| Traces | The path of one request through every service, as a tree of timed spans sharing one trace ID | "Where did the time go? Which service failed?" | Moderate: usually sampled (keep all errors and slow ones, a fraction of the rest) |
Observability means being able to answer new questions about the system's behaviour from these signals, without shipping new code. Monitoring is the subset you watch continuously and alert on.
What to measure: RED and USE
- RED for every service (what users feel): Rate (requests per second), Errors (failed requests per second or as a share), Duration (latency percentiles: p50, p95, p99).
- USE for every resource (what machines feel): Utilization (how busy), Saturation (how much work is queued), Errors.
- Google's SRE book calls the user-facing set the four golden signals: latency, traffic, errors, saturation.
2. How it works
Following one slow request
A user says checkout was slow. Press Play to see how a distributed trace finds the cause.
How traces stay connected
The first service creates a trace ID and passes it, with its own span ID, to every service it calls, in the standard traceparent HTTP header defined by W3C Trace Context (and through message headers for queues). Each service records its spans with that trace ID and its parent. A tracing backend then assembles the tree. Put the same trace ID in every log line, and you can jump from a slow span straight to its logs.
Alerting on what matters: SLO burn rates
"CPU above 80%" alerts wake people for things users never notice, and miss things they do. Better: alert when the system is spending its error budget (Module 04) too fast. The burn rate is how many times faster than "exactly on budget" you're failing: with a 99.9% SLO, a 1% error rate is a burn rate of 10 (1% ÷ 0.1%), which would use a 30-day budget in 3 days.
Google's SRE workbook recommends multi-window, multi-burn-rate alerts, for example: page if the burn rate is above 14.4 over the last hour (2% of the monthly budget gone in an hour) or above 6 over the last 6 hours (5% gone), each confirmed by a short window so the alert also stops quickly; open a ticket for a burn rate of 1 over 3 days. Try it in section 3.
Cardinality: the hidden bill
Each unique combination of a metric's labels is a separate time series. http_requests{service, endpoint, status} with 20 services × 50 endpoints × 10 statuses is 10,000 series: fine. Add user_id with a million users and it's 10 billion: your metrics system falls over. Keep unbounded values (user IDs, order IDs, full URLs) in logs and traces, never in metric labels.
3. Try it yourself
Should this page someone?
How many time series?
Choose the labels on http_request_duration_seconds (a histogram with 12 buckets per series).
4. Trade-offs
Detail vs cost
Observability bills can rival compute bills. Keep metrics broad and cheap, sample traces (always keep errors and slow ones), and set log levels and retention by value.
Symptoms vs causes
Page on symptoms users feel (SLO burn, error rate, latency). Use cause metrics (CPU, disk, queue depth) on dashboards and for diagnosis, not for waking people.
Head vs tail sampling
Deciding to keep a trace at the start (head) is cheap but misses rare slow requests. Deciding at the end (tail, in a collector) keeps the interesting ones but needs buffering.
Build vs buy
Open-source stacks (Prometheus, Grafana, Loki, Tempo, Jaeger) give control and need operating; vendors (Datadog, New Relic, Honeycomb, Grafana Cloud, cloud-native tools) cost more and run themselves. OpenTelemetry keeps you portable.
5. In real systems
- OpenTelemetry (a CNCF project) is the standard way to instrument code: one set of APIs and SDKs for traces, metrics and logs (profiles are being added), many auto-instrumentation libraries, and the OpenTelemetry Collector to process and route data to any backend.
- Prometheus scrapes metrics from services and evaluates alert rules; Grafana builds dashboards over it and many other sources. Large setups use Thanos, Mimir or VictoriaMetrics for long-term, horizontally scaled storage (Design D18).
- Tracing backends: Jaeger, Grafana Tempo, Zipkin, or vendors. The idea goes back to Google's Dapper paper (2010).
- Logs: structured JSON, shipped by an agent to Elasticsearch/OpenSearch, Grafana Loki or a vendor; keep recent logs hot and searchable, older logs in cheap object storage.
- On-call basics: every page links to a runbook; alerts are grouped and deduplicated; and every serious incident gets a blameless post-incident review that produces fixes, not blame.
The technologies up close
RED metrics per service, traces with OpenTelemetry across the request path, structured logs with trace IDs, and paging on SLO burn rate for checkout latency and errors.Add client-side monitoring for apps. For a monitoring or logging design, cardinality, cost and the pipeline are the deep dives.
OpenTelemetry
The default way to instrumentOne vendor-neutral set of APIs, SDKs and a collector for traces, metrics and logs.
How it works
Libraries auto-instrument common frameworks (HTTP servers, database clients, queues) and propagate the trace context between services. The Collector receives telemetry, samples, enriches and batches it, and exports it to any backend, so switching vendors doesn't mean re-instrumenting.
Status today
A CNCF project supported by every major observability vendor and cloud; profiles are being added as a fourth signal.
Use it when
- Any new service; especially many services in several languages
Skip it when
- Rewriting working instrumentation just to switch
In the interview
Services are instrumented with OpenTelemetry; the collector samples traces and sends everything to our backend.
- Vendor-specific agents everywhere, locking every service to one tool.
Prometheus and Grafana (metrics)
The default metrics stackPrometheus scrapes numeric time series from services and evaluates alert rules; Grafana draws dashboards over it and other sources.
How it works
Services expose counters, gauges and histograms on an endpoint; Prometheus pulls them every 15–60 s into a local time-series database and evaluates PromQL alert rules; Alertmanager groups, deduplicates and routes alerts. Long-term, horizontally scaled storage comes from Thanos, Mimir or VictoriaMetrics (Design D18).
Status today
The de-facto open-source standard, also offered as managed services by the major clouds.
Use it when
- RED and USE metrics, SLO dashboards and burn-rate alerts
Skip it when
- High-cardinality data (per-user or per-request labels): that belongs in logs or traces
In the interview
RED metrics per service in Prometheus, SLO burn-rate alerts through Alertmanager.
- Labels with unbounded values (user ID, URL with IDs): cardinality explodes.
Distributed logging pipeline
Situational · logging-system designsGetting logs from thousands of machines into one searchable place, affordably.
How it works
- Structured logs (JSON with
trace_id,user_id, level) written to local files or stdout. - Agents on each host (Fluent Bit, Vector, the OpenTelemetry Collector) tail, parse and batch them; they buffer locally if the pipeline is down.
- A buffer (usually Kafka) absorbs spikes and lets several consumers read: the search index, the archive, security analytics.
- Hot store for recent days (Elasticsearch/OpenSearch, Loki, ClickHouse); cold store in object storage for months or years.
- Cost control: sample or drop debug logs, keep every error, set retention per log type.
Status today
Common stacks: Fluent Bit or Vector → Kafka → OpenSearch or Loki, with S3 archives; managed alternatives from every observability vendor and cloud.
Use it when
- "Design a logging system"
- Any large fleet where SSH-ing into machines doesn't scale
Skip it when
- Using logs for things metrics answer more cheaply (rates, latencies)
In the interview
Agents ship structured logs to Kafka; an indexer keeps 7 days hot in OpenSearch and archives the rest to S3; debug logs are sampled.
- Indexing every log line forever: the bill grows faster than traffic.
- Logs without trace IDs: you can't jump from a slow trace to what it logged.
Distributed tracing backends
Situational · many servicesStores and searches traces: the tree of timed spans one request made across services.
How it works
Trace context travels in the W3C traceparent header; each service sends its spans to the collector. Tail-based sampling keeps every error and slow trace and a fraction of the rest, decided after the whole trace is seen.
Status today
Jaeger, Grafana Tempo, Zipkin and vendor backends; the idea comes from Google's Dapper paper (2010).
Use it when
- Microservices, where a slow request could be anywhere
Skip it when
- A single service with one database: logs and metrics are enough
In the interview
Tail-sampled traces keep every error and slow request, so we can always find the cause of a bad p99.
- Head sampling at 1%, then wondering why the one failing request isn't there.
Client-side monitoring: RUM, crash reports, synthetic checks
Situational · what users actually seeMeasuring the product from the user's device, because server metrics can look perfect while the app is broken.
How it works
- Real-user monitoring (RUM): a small script or SDK reports page loads, Core Web Vitals (LCP, INP, CLS), API timings and JavaScript errors from real users, by device, country and release.
- Crash and error reporting: mobile and web SDKs (Sentry, Firebase Crashlytics) send stack traces with the app version and the user's last actions.
- Synthetic monitoring: scripted probes from many regions hit key flows (log in, check out) every minute, catching outages even when nobody is using that path.
Status today
Standard practice: Sentry, Crashlytics, Datadog RUM and similar vendors; browsers expose the timing data through standard performance APIs.
Use it when
- Consumer apps, mobile apps, anything where the client does real work
- Catching bad releases of the app itself
Skip it when
- Pure backend services with no user-facing client
In the interview
RUM and crash reporting by app version, plus synthetic checkout probes from five regions, alerting on errors per release.
- Only server-side metrics: a broken JavaScript bundle or a mobile crash loop never shows up as a 500.
When the page fires: incidents, runbooks and reviews
The default · how teams handle outagesAlerts (above) decide when a person is woken. What happens next decides how long users suffer. Teams that recover fast follow the same few habits: one person in charge, stop the damage before finding the cause, and learn from it afterwards without blame.
How it works
Severity sets the response. Agree the levels before you need them:
| Level | Example (a food-delivery app) | Response |
|---|---|---|
| SEV1 | No one can place orders | Page now, incident lead named, status page within 15 minutes, all hands as needed |
| SEV2 | Card payments fail for one bank; courier tracking frozen | Page the owning team; status page if customers notice |
| SEV3 | Restaurant reports a day late | A ticket for working hours |
A runbook turns an alert into actions. Each alert links to one page: what the alert means for users, how to confirm it (the dashboard and the query), safe first actions in order (roll back, turn off the flag, fail over, add capacity), what not to do, and who to call next. A runbook step that is always the same is a candidate for automation.
Design so the 3 a.m. questions have quick answers:
- What changed? Deploys, flag flips and config pushes recorded as events on the same dashboards as the metrics.
- Who is affected? Metrics broken down by region, app version and customer tier, within your cardinality budget.
- What happened to this request? A request ID in every log line and trace, returned to the client so support can quote it.
- Can we undo it? Every release and every flag can be reversed in minutes, without a new build.
Status today
Most companies run some version of this, often borrowed from emergency services' incident command and described in Google's SRE books; tools such as PagerDuty and incident.io handle paging, roles and timelines.
Use it when
- Any system with an on-call rotation
- The interview question "how would you operate this?"
Skip it when
- Formal roles for a SEV3: a ticket is enough
In the interview
Every alert links to a runbook whose first step is usually "roll back or turn off the flag". We mitigate first, find the cause after, and every SEV1 gets a blameless review with owners and dates on the actions.
- Debugging live while users are still failing when a rollback was available.
- Reviews that end with "be more careful" instead of a change to the system.
6. Remember it
How this hook works: what "dashboard, tracker, records" means
What the slogan means, phrase by phrase
- "Dashboard says something's wrong"
- Metrics (RED, USE, SLO burn) show that and when behaviour changed, cheaply, for everything.
- "tracker says where"
- A trace shows which service and which call in the request's path took the time or failed.
- "records say why"
- Structured logs, found by trace ID, give the detail: the error message, the input, the user.
Why this picture fits: a late parcel is investigated the same way, and nobody confuses the dashboard light with the repair record.
The story → the concept
| Car and parcel | Observability |
|---|---|
| The dashboard gauges | Metrics |
| The warning light | An alert on an SLO burn rate |
| The parcel's tracking history | A distributed trace |
| Each depot scan | A span |
| The tracking number | The trace ID, carried in traceparent |
| The service history | Logs |
Watch it on an example
- Dashboard: checkout p99 jumped from 400 ms to 3 s at 14:05; the burn-rate alert pages.
- Tracker: slow traces all show a 2.6 s span in the fraud-check call; other spans are normal.
- Records: fraud-check logs for those trace IDs show "connection pool exhausted" since a 14:02 config change.
- Roll back the change; p99 recovers. Time to cause: minutes, not hours.
The trap it saves you from
Grepping logs on fifty servers during an outage, or alerting on every resource metric until on-call engineers learn to ignore alerts.
It still works when…
You run AI features (Module 29): traces of model and tool calls, metrics for tokens, cost and latency, and logs (carefully redacted) of prompts and outputs.
7. Common mistakes
8. Design exercise (25 minutes)
- Instrument a design: for the URL shortener or Pastebin, list the RED metrics per service, the spans in a redirect's trace, and the fields every log line should carry.
- Write SLO alerts: redirects must succeed 99.95% of the time. Compute the burn-rate thresholds for a 1-hour and a 6-hour window, and check them in the calculator.
- Find the cardinality bomb: a team wants
orders_total{customer_id, product_id, country}. Estimate the series, and propose a better design. - Plan sampling: 20,000 requests a second, budget for 200 traces a second. What do you keep?
- Write it down: RED, USE, and the burn-rate formula.
9. Self-check quiz
Which signal shows which service a slow request spent its time in?
With a 99.9% SLO, the error rate is 1%. What's the burn rate?
Which should never be a metric label?
What does RED stand for?
Server dashboards are green, but users report the checkout button does nothing. What would have caught it?
Checkout errors spike 9 minutes after a release. The cause isn't clear yet. What's the first move?
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| Monitoring Distributed Systems (Google SRE book) | 40 min | The four golden signals, symptoms vs causes, and what makes a good alert. |
| Alerting on SLOs (Google SRE workbook) | 45 min | Burn rates and multi-window alerts, worked step by step. |
| OpenTelemetry concepts | 45 min | Signals, context propagation, the Collector and sampling. |
| Dapper (Google, 2010) | 40 min | The paper that introduced large-scale distributed tracing. |
| The USE Method (Brendan Gregg) | 15 min | A checklist for finding resource bottlenecks. |
11. Under the hood (optional)
How metrics, traces and logs are really recorded, sampled and turned into alerts. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.
Observability internalsMetric types and a percentile lab, cardinality, traceparent propagation lab, head vs tail sampling (animated), logs indexing, SLO burn-rate alerts, profiling and eBPF, 6 products comparedabout 30 min
Dashboards are only as truthful as the data types behind them. How a latency percentile is computed, what a trace ID looks like on the wire, which traces are kept, and what an alert actually measures decide whether you find the problem or a misleading average.
Metric types
| Type | What it stores | Use it for | Trap |
|---|---|---|---|
| Counter | A number that only goes up (resets on restart) | Requests, errors, bytes; graph its rate | Graphing the raw value instead of rate() |
| Gauge | A value that goes up and down | Queue length, memory, temperature | Sampling misses spikes between scrapes |
| Histogram | Counts of observations per bucket (≤ 10 ms, ≤ 25 ms, …), plus sum and count | Latency and size distributions; aggregatable across servers | Percentiles are estimated within a bucket; choose buckets around your SLO |
| Native / exponential histogram | Buckets on an exponential scale chosen automatically, stored sparsely | Accurate percentiles at low cost (Prometheus native histograms, stable since 3.8; OpenTelemetry exponential histograms) | Needs opt-in and backend support |
| Summary | Quantiles computed inside each process | A single process's latency | Quantiles can't be combined across instances |
Cardinality and cost
Every unique combination of metric name and label values is a separate series stored and indexed forever (until retention). http_requests_total{route, method, status} with 50 routes × 5 methods × 10 statuses is 2,500 series per instance, fine; adding user_id multiplies it by your user count. Put high-cardinality detail in logs and traces, where it's sampled or stored cheaply, and keep metrics for aggregates.
Traces: context propagation
A trace is a tree of spans, one per unit of work, linked by IDs passed from service to service in the W3C traceparent header:
- Head sampling decides at the first service (say, keep 1%) and passes the decision downstream in the flags, so a trace is kept or dropped as a whole. Cheap, but slow and failed requests are dropped at the same rate as boring ones.
- Tail sampling buffers all spans of a trace in a collector and decides once it's complete: keep every error and every trace over 2 s, plus 1% of the rest. Better traces, but the collector must hold whole traces in memory and route all spans of a trace to the same instance.
- Exemplars attach a trace ID to a metric sample, so a spike on a latency graph links straight to an example slow trace.
- OpenTelemetry provides the SDKs, the wire protocol (OTLP) and the Collector, which receives, processes (batches, samples, scrubs personal data) and exports to any backend.
Logs: index everything or index labels
| Index every word (Elasticsearch, OpenSearch, Splunk) | Index labels only (Grafana Loki, many cloud log stores) | |
|---|---|---|
| Storage | Inverted index on local disks, often larger than the raw logs | Compressed chunks in object storage plus a small label index |
| Queries | Fast full-text search anywhere | Pick streams by label, then scan (grep) their chunks in parallel |
| Cost | High at large volumes | Much lower; slower for needle-in-a-haystack searches |
Either way: log structured JSON with a trace ID, a consistent level and no secrets or personal data; sample or drop debug logs in production; and use a pipeline (Fluent Bit, Vector, the OpenTelemetry Collector) to buffer, enrich and route.
Alerting on SLOs: burn rates
An SLO of 99.9% success over 30 days allows 0.1% errors: the error budget. The burn rate is how fast you're spending it: a burn rate of 1 uses exactly the budget in 30 days; 14.4 uses 2% of it in one hour. The Google SRE workbook recommends paging when both a long and a short window agree, so alerts are fast for big incidents but don't fire on brief blips:
| Severity | Burn rate | Long window | Short window | Budget consumed when it fires |
|---|---|---|---|---|
| Page | 14.4 | 1 hour | 5 minutes | 2% |
| Page | 6 | 6 hours | 30 minutes | 5% |
| Ticket | 1 | 3 days | 6 hours | 10% |
Profiles, eBPF and real users
- Continuous profiling (Grafana Pyroscope, Parca, cloud profilers) samples stacks across the fleet at about 1% overhead, so you can diff CPU and memory by version; OpenTelemetry is adding profiles as a fourth signal.
- eBPF programs in the Linux kernel observe network calls, latency and CPU for every process without changing code (Cilium Hubble, Pixie, Grafana Beyla), useful for services nobody instrumented.
- Real-user monitoring measures what browsers and apps actually experienced (Core Web Vitals, crashes), which server metrics can't see.
Product by product
| Product | How it works | The detail worth knowing |
|---|---|---|
| Prometheus | Pulls metrics from targets, stores series in its TSDB, alerts with PromQL rules | Cardinality, not request volume, decides cost; Thanos and Mimir add long-term, multi-cluster storage |
| Elasticsearch / OpenSearch for logs | Inverted index over every field | Full-text search at a storage cost |
| Grafana Loki | Label index plus compressed chunks in object storage | Much cheaper; queries scan chunks |
| OpenTelemetry | SDKs, OTLP and the Collector for traces, metrics and logs | Instrument once, change backends later |
| Jaeger, Grafana Tempo | Trace storage; Tempo keeps traces in object storage, found by trace ID | Cheap retention of every sampled trace |
| Pyroscope, Parca | Continuous profiling | Flame graphs over time and versions |
Say it in the interview
- "How do you compute p99 across 50 servers?" From merged histograms (sum the bucket counts, then estimate the quantile), never by averaging each server's p99.
- "How do you keep tracing affordable?" Tail sampling in the Collector: keep all errors and slow traces, a small share of the rest; store spans in object storage.
- "When do you page someone?" On fast SLO burn (14.4× over an hour, confirmed over 5 minutes), not on CPU or single errors; slower burns open tickets.
How should you get the p99 latency of a service running on many servers?
What is the advantage of tail sampling over head sampling?
A burn rate of 14.4 sustained for one hour consumes how much of a 30-day error budget?