Coordination services and distributed locks

"Only one server may run the nightly billing job." "If the leader dies, another must take over within seconds, and never two at once." These sound like one-line requirements, and they're among the hardest to get right, because servers pause, clocks drift and networks split. Coordination services exist so you don't build consensus yourself. This guide explains how they work, the products, and when a simpler lock is enough, with a chooser and code you can run.

1. What coordination services do

A small cluster (three or five nodes) agrees on every change with a consensus protocol (Module 21), so it stays correct when a minority fails. On top of that agreement it offers a few building blocks:

2. How leader election works

3. The products

4. Differences at a glance

5. When to choose which, and why

1. Do you need a lock at all? An idempotent operation, a unique constraint, or a single queue consumer often removes the need.
2. Only efficiency at stake (two servers doing the same work is wasteful, not harmful)? A Redis lock with an expiry is fine.
3. Correctness at stake (two holders would corrupt data)? Use a consensus-backed store or a strongly consistent database, and fencing tokens checked by the resource you protect.
4. On Kubernetes? Use Lease-based leader election from the client libraries; it's already there.
5. Building infrastructure that many services coordinate through (membership, config, discovery)? etcd, ZooKeeper or Consul.

Try it: choose for your requirements

6. Why locks need fencing tokens, in code

Client A takes a lock with a 10-second lease, then pauses for 15 seconds (a garbage-collection pause, a stalled VM). Its lease expires and B takes the lock. When A wakes, it still believes it holds the lock and writes. Without fencing, A's stale write overwrites B's. With fencing, each grant carries a higher token, and the storage rejects any write with a token lower than one it has already seen. The program was run in all three languages; each prints the output below it.

class LockService:
    """A lease-based lock, like one built on etcd, ZooKeeper or Consul. Each grant gets a higher token."""

    def __init__(self, lease):
        self.lease, self.holder, self.expires, self.token = lease, None, 0, 32

    def acquire(self, client, now):
        if self.holder and now < self.expires:
            return None                          # someone else holds a live lease
        self.token += 1
        self.holder, self.expires = client, now + self.lease
        return self.token


class AccountStore:
    def __init__(self, fenced):
        self.fenced, self.value, self.highest = fenced, None, 0

    def write(self, client, token, value):
        if self.fenced and token < self.highest:
            return f"{client} write rejected (token {token} < {self.highest})"
        self.highest = max(self.highest, token)
        self.value = value
        return f"{client} wrote {value!r} with token {token}"


for fenced in (False, True):
    lock, store = LockService(lease=10), AccountStore(fenced)
    print("with fencing tokens:" if fenced else "without fencing:")
    a = lock.acquire("A", now=0)
    print(f"  t=0   A gets the lock, token {a}; then pauses for 15 s (garbage collection)")
    b = lock.acquire("B", now=12)
    print(f"  t=12  A's lease has expired; B gets the lock, token {b}")
    print("  t=13 ", store.write("B", b, "balance=90"))
    print("  t=15 ", store.write("A", a, "balance=100"), "<- A wakes, still thinks it holds the lock")
    print(f"  final value: {store.value!r}")

// A lease-based lock, like one built on etcd, ZooKeeper or Consul. Each grant gets a higher token.
class LockService {
  holder: string | null = null;
  expires = 0;
  token = 32;
  constructor(readonly lease: number) {}

  acquire(client: string, now: number): number | null {
    if (this.holder && now < this.expires) return null; // someone else holds a live lease
    this.token++;
    this.holder = client;
    this.expires = now + this.lease;
    return this.token;
  }
}

class AccountStore {
  value: string | null = null;
  highest = 0;
  constructor(readonly fenced: boolean) {}

  write(client: string, token: number, value: string): string {
    if (this.fenced && token < this.highest) return `${client} write rejected (token ${token} < ${this.highest})`;
    this.highest = Math.max(this.highest, token);
    this.value = value;
    return `${client} wrote '${value}' with token ${token}`;
  }
}

for (const fenced of [false, true]) {
  const lock = new LockService(10), store = new AccountStore(fenced);
  console.log(fenced ? "with fencing tokens:" : "without fencing:");
  const a = lock.acquire("A", 0)!;
  console.log(`  t=0   A gets the lock, token ${a}; then pauses for 15 s (garbage collection)`);
  const b = lock.acquire("B", 12)!;
  console.log(`  t=12  A's lease has expired; B gets the lock, token ${b}`);
  console.log("  t=13 ", store.write("B", b, "balance=90"));
  console.log("  t=15 ", store.write("A", a, "balance=100"), "<- A wakes, still thinks it holds the lock");
  console.log(`  final value: '${store.value}'`);
}

public class Main {
    // A lease-based lock, like one built on etcd, ZooKeeper or Consul. Each grant gets a higher token.
    static class LockService {
        final int lease;
        String holder = null;
        int expires = 0, token = 32;
        LockService(int lease) { this.lease = lease; }

        Integer acquire(String client, int now) {
            if (holder != null && now < expires) return null; // someone else holds a live lease
            token++;
            holder = client;
            expires = now + lease;
            return token;
        }
    }

    static class AccountStore {
        final boolean fenced;
        String value = null;
        int highest = 0;
        AccountStore(boolean fenced) { this.fenced = fenced; }

        String write(String client, int token, String value) {
            if (fenced && token < highest) return client + " write rejected (token " + token + " < " + highest + ")";
            highest = Math.max(highest, token);
            this.value = value;
            return client + " wrote '" + value + "' with token " + token;
        }
    }

    public static void main(String[] args) {
        for (boolean fenced : new boolean[] {false, true}) {
            LockService lock = new LockService(10);
            AccountStore store = new AccountStore(fenced);
            System.out.println(fenced ? "with fencing tokens:" : "without fencing:");
            int a = lock.acquire("A", 0);
            System.out.println("  t=0   A gets the lock, token " + a + "; then pauses for 15 s (garbage collection)");
            int b = lock.acquire("B", 12);
            System.out.println("  t=12  A's lease has expired; B gets the lock, token " + b);
            System.out.println("  t=13  " + store.write("B", b, "balance=90"));
            System.out.println("  t=15  " + store.write("A", a, "balance=100") + " <- A wakes, still thinks it holds the lock");
            System.out.println("  final value: '" + store.value + "'");
        }
    }
}

without fencing:
  t=0   A gets the lock, token 33; then pauses for 15 s (garbage collection)
  t=12  A's lease has expired; B gets the lock, token 34
  t=13  B wrote 'balance=90' with token 34
  t=15  A wrote 'balance=100' with token 33 <- A wakes, still thinks it holds the lock
  final value: 'balance=100'
with fencing tokens:
  t=0   A gets the lock, token 33; then pauses for 15 s (garbage collection)
  t=12  A's lease has expired; B gets the lock, token 34
  t=13  B wrote 'balance=90' with token 34
  t=15  A write rejected (token 33 < 34) <- A wakes, still thinks it holds the lock
  final value: 'balance=90'

7. Say it in the interview

8. Common mistakes

9. Check yourself

10. Sources and further reading

Reviewed on 4 November 2026.