Design metrics monitoring and alerting

Every server, container and service emits numbers every few seconds: CPU, memory, request rates, error counts, latency percentiles. A monitoring system stores millions of these time series cheaply, draws them on dashboards in a blink, and wakes a person only when something is really wrong. The hard parts are the write volume, the storage format, cardinality and alerts that are neither silent nor noisy. It follows the interview framework phase by phase.

1. The prompt

"Design a metrics monitoring and alerting system" (think Prometheus, Datadog, Grafana Mimir, CloudWatch): collect metrics from a large fleet, let engineers query and graph them, and notify on-call engineers when rules are broken.

It is the most write-heavy design in the track: data arrives every second of every day, is almost never updated, and is mostly read for the last few hours. That shape drives a special-purpose storage engine.

2. Frame the problem

3. What it must do

1. Collect numeric metrics with labels (http_requests_total{service="checkout", status="500"}).
2. Query and graph them: rates, sums by label, percentiles over any time range.
3. Evaluate alert rules and notify the right people through pager, chat or email.

Not now:

• Logs and traces (same ideas, different storage).
• The dashboard UI itself.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Series: a metric name plus a set of labels.
• Sample: a (timestamp, value) pair.
• Block: compressed samples for a time range.
• Alert rule: a query, a condition, a duration and routing labels.

series id   hash(name + sorted labels) → {__name__="http_latency_bucket", service="checkout", le="0.5"}
samples     series id → [(t, v), (t, v), …]      in-memory chunk + WAL, then immutable 2-hour blocks
label index label=value → posting list of series ids
rule        name, expr, for, labels {severity, team}, annotations

7. The contract

GET  /metrics                         (each target exposes current values; the collector scrapes it)
POST /api/v1/write                    (or push batches of samples)
GET  /api/v1/query_range?query=histogram_quantile(0.99, sum by (le) (rate(http_latency_bucket[5m])))&start=…&end=…&step=60
rule: checkout_p99_high   expr: p99_latency_ms > 400   for: 5m   labels: severity=page, team=payments

Data flow: (1) collectors scrape or receive samples near the workloads → (2) forward batches to TSDB shards by series hash → (3) shards append to memory and a WAL, cut blocks every two hours → (4) old blocks go to object storage, compacted and downsampled → (5) the query service fans out to shards and storage → (6) a rule evaluator runs queries on a schedule and sends alerts to an alert manager.

8. First working version

Services write every measurement into one relational table (metric, labels, time, value); dashboards and a cron job query it.

8.1 Writing samples

8.2 Graphing

A dashboard runs a range query on the table with a B-tree index on (metric, time).

8.3 Alerting

A cron job runs each rule's query every minute and emails if it's true.

9. Make it hold up

9.1 Pull or push? write-heavy

9.2 The time-series database write-heavy · cheap

• Append-only, in memory first: recent samples for each series sit in an in-memory chunk, protected by a write-ahead log; every couple of hours they're cut into immutable, compressed blocks on disk (Prometheus works this way).
• Compression: timestamps arrive at regular intervals, so storing the difference of differences is usually zero; consecutive values are similar, so XOR-ing each with the previous leaves mostly zero bits. Facebook's Gorilla paper reports an average of 1.37 bytes per sample, about a 12× reduction.
• An inverted index on labels maps each label value to the series that have it, so {service="checkout"} finds the right series quickly.
• Sharding: hash the series id to pick a shard; replicate each shard (or run two collectors writing to two stores) so losing a node loses no data.

9.3 Long-term storage and downsampling cheap retention

Blocks are uploaded to object storage, compacted into larger blocks and downsampled (5-minute and 1-hour resolution) so a year-long graph reads few points. Thanos, Grafana Mimir and VictoriaMetrics add this to the Prometheus model. The query layer picks the coarsest resolution that still gives enough points for the graph's width.

9.4 Cardinality: the silent killer write-heavy · cost

Every distinct combination of label values is a separate series. Add a user_id label to a latency metric with 10M users and you create 10M series overnight, exhausting memory. Keep labels bounded (service, endpoint, status class, region), put per-user detail in logs or traces, and enforce per-tenant series limits at ingestion: reject or drop new series past the limit, and tell the team which metric grew.

9.5 Alerting that people trust reliable alerting

• Rule evaluation: every 15–60 s, evaluate each rule's query. A rule with for: 5m must be true for five minutes before it fires; until then it is pending. This filters one-off spikes. Try it in section 13.
• Alert manager: deduplicates the same alert from replicas, groups related alerts into one notification, routes by labels (team, severity), and supports silences during maintenance.
• Alert on symptoms users feel (error rate, latency, SLO burn rate; Module 25), not on every cause (CPU at 80%).

9.6 Monitoring the monitor reliable alerting

Run rule evaluation and the alert manager in pairs, close to the data, and independent of the systems they watch. A separate, simple watchdog alerts if the monitoring system itself stops working: a "dead man's switch" alert that must always be firing, so its absence pages someone through an outside service.

9.7 The whole design after the deep dives

Samples from collection to an alert. Press Play or drag the timeline.

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Resolution vs cost: 10-second samples catch short blips but cost six times more than one-minute samples; downsample old data aggressively.
• Local vs global view: one store per cluster is simple and robust; a global query layer gives one view at the cost of a bigger system to run.
• Fast vs quiet alerts: short for durations detect faster and page more often for nothing; burn-rate alerts over two windows balance both.
• Bounded labels over unlimited detail.

What we'd watch:

• Ingest rate and dropped samples.
• Active series per tenant.
• Query p99.
• Rule evaluation delay.
• Pages per on-call shift.

At 10×:

• More shards.
• Stricter cardinality limits.
• More recording rules.
• Regional stores with a global query layer.

Common mistakes:

• A general-purpose database for samples.
• Unbounded labels.
• Averaging percentiles.
• Alerting on causes, not symptoms.
• Monitoring that shares fate with what it monitors.

13. Try it, check yourself, read more

Six hours of checkout p99 latency, one point a minute. A real incident starts at 3:50 and lasts half an hour; the rest has noise and a few one-minute spikes. Tune the rule so it pages for the incident quickly but not for the spikes.