Observability stacks

"How would you know it's broken?" comes up in every senior interview, and the answer names tools. Metrics, logs and traces each need collecting, storing and querying, and the choice between running Prometheus and Grafana yourself, buying Datadog, or using your cloud's suite is mostly about cost, scale and who operates it. This guide explains how each stack works, how they differ and which to choose, with a chooser and code you can run.

1. Three signals, one pipeline

Module 25 teaches what to measure and how to alert; this guide is about the tools.

OpenTelemetry (a CNCF project) is the common way to produce all three: one set of SDKs and a collector that receives, processes and exports to whichever backend you choose. Instrument once, and you can change vendors without touching the code.

2. How telemetry flows

3. The products

4. Differences at a glance

5. When to choose which, and why

1. Instrument with OpenTelemetry whatever you choose; it keeps the backend a decision you can change.
2. One cloud, a small team? The cloud's own suite: nothing to run, and it already sees the managed services.
3. Kubernetes and a platform team? Prometheus and Grafana, with Loki and Tempo, and Mimir or Thanos for long-term metrics.
4. No time to operate it and budget to pay? A vendor such as Datadog; watch custom metrics and log volume.
5. Hard, unknown performance problems across many services? High-cardinality tracing, such as Honeycomb, alongside your metrics.

Try it: choose for your requirements

6. Head and tail sampling, in code

Keeping every trace is expensive, so systems sample. 100,000 traces, 1% with errors and 2% slow. Head sampling decides at the start, before anyone knows how the request will go: keeping 10% keeps only about 10% of the errors and slow requests, the ones you most need. Tail sampling in the collector waits until each trace is complete, keeps every error and slow trace and 1% of the rest: fewer traces stored, and none of the interesting ones lost. The program was run in all three languages; each prints the output below it.

seed = 42


def rand(n):
    """A small, repeatable random generator (Park-Miller), so all three languages draw the same numbers."""
    global seed
    seed = seed * 16807 % 2147483647
    return seed % n


# 100,000 traces: 1 in 100 has an error, 2 in 100 are slow (over 1 s), the rest are fine.
traces = []
for _ in range(100_000):
    r = rand(100)
    traces.append("error" if r == 0 else "slow" if r <= 2 else "ok")

total = {k: traces.count(k) for k in ("error", "slow", "ok")}
print(f"traces: {len(traces):,} ({total['error']:,} errors, {total['slow']:,} slow)")

# Head sampling: decide at the start, before knowing how the request will go. Keep 10%.
head = [t for t in traces if rand(10) == 0]
# Tail sampling: a collector waits for the whole trace, keeps every error and slow one, and 1% of the rest.
tail = [t for t in traces if t != "ok" or rand(100) == 0]

for name, kept in (("head, 10%", head), ("tail, errors + slow + 1%", tail)):
    e, s = kept.count("error"), kept.count("slow")
    print(f"{name:26} keeps {len(kept):6,} traces: {e:4,} of {total['error']:,} errors, {s:5,} of {total['slow']:,} slow")

let seed = 42;

// A small, repeatable random generator (Park-Miller), so all three languages draw the same numbers.
function rand(n: number): number {
  seed = (seed * 16807) % 2147483647;
  return seed % n;
}

const fmt = (n: number, width = 0) => n.toLocaleString("en-US").padStart(width);

// 100,000 traces: 1 in 100 has an error, 2 in 100 are slow (over 1 s), the rest are fine.
const traces: string[] = [];
for (let i = 0; i < 100_000; i++) {
  const r = rand(100);
  traces.push(r === 0 ? "error" : r <= 2 ? "slow" : "ok");
}
const count = (list: string[], k: string) => list.filter((t) => t === k).length;
const errors = count(traces, "error"), slow = count(traces, "slow");
console.log(`traces: ${fmt(traces.length)} (${fmt(errors)} errors, ${fmt(slow)} slow)`);

// Head sampling: decide at the start, before knowing how the request will go. Keep 10%.
const head = traces.filter(() => rand(10) === 0);
// Tail sampling: a collector waits for the whole trace, keeps every error and slow one, and 1% of the rest.
const tail = traces.filter((t) => t !== "ok" || rand(100) === 0);

for (const [name, kept] of [["head, 10%", head], ["tail, errors + slow + 1%", tail]] as [string, string[]][]) {
  console.log(`${name.padEnd(26)} keeps ${fmt(kept.length, 6)} traces: ${fmt(count(kept, "error"), 4)} of ${fmt(errors)} errors, ${fmt(count(kept, "slow"), 5)} of ${fmt(slow)} slow`);
}

import java.util.*;
import java.util.stream.*;

public class Main {
    static long seed = 42;

    // A small, repeatable random generator (Park-Miller), so all three languages draw the same numbers.
    static int rand(int n) {
        seed = seed * 16807 % 2147483647;
        return (int) (seed % n);
    }

    static long count(List<String> list, String k) { return list.stream().filter(k::equals).count(); }

    public static void main(String[] args) {
        // 100,000 traces: 1 in 100 has an error, 2 in 100 are slow (over 1 s), the rest are fine.
        List<String> traces = new ArrayList<>();
        for (int i = 0; i < 100_000; i++) {
            int r = rand(100);
            traces.add(r == 0 ? "error" : r <= 2 ? "slow" : "ok");
        }
        long errors = count(traces, "error"), slow = count(traces, "slow");
        System.out.println(String.format(Locale.US, "traces: %,d (%,d errors, %,d slow)", traces.size(), errors, slow));

        // Head sampling: decide at the start, before knowing how the request will go. Keep 10%.
        List<String> head = traces.stream().filter(t -> rand(10) == 0).collect(Collectors.toList());
        // Tail sampling: a collector waits for the whole trace, keeps every error and slow one, and 1% of the rest.
        List<String> tail = traces.stream().filter(t -> !t.equals("ok") || rand(100) == 0).collect(Collectors.toList());

        Object[][] rows = {{"head, 10%", head}, {"tail, errors + slow + 1%", tail}};
        for (Object[] row : rows) {
            @SuppressWarnings("unchecked") List<String> kept = (List<String>) row[1];
            System.out.println(String.format(Locale.US, "%-26s keeps %,6d traces: %,4d of %,d errors, %,5d of %,d slow",
                    row[0], kept.size(), count(kept, "error"), errors, count(kept, "slow"), slow));
        }
    }
}

traces: 100,000 (1,020 errors, 2,067 slow)
head, 10%                  keeps  9,966 traces:  117 of 1,020 errors,   180 of 2,067 slow
tail, errors + slow + 1%   keeps  4,059 traces: 1,020 of 1,020 errors, 2,067 of 2,067 slow

7. Say it in the interview

8. Common mistakes

9. Check yourself

10. Sources and further reading

Reviewed on 4 November 2026. Pricing and features change; check before relying on them.