Segment: back from microservices

Segment, a customer data platform, forwards every event its customers send to the tools they use: analytics, advertising, email, more than 140 "destinations". To stop one failing destination delaying all the others, the team gave each destination its own queue and service. It worked, then it became the problem: 140 services, shared libraries in a dozen versions, autoscaling tuned by hand, and engineers paged at night for load spikes. In 2018 they merged the destinations back into one service, and wrote one of the most candid posts about what that cost them.

1. The situation

Segment's API receives events (a page view, a purchase) and sends each one to every destination the customer has switched on, translating it into that destination's format. Some destinations handle a handful of events a day; others thousands a second. Failures are routine: partners time out, rate-limit or go down. Retry-able errors (timeouts, 500s, rate limits) are retried later.

2. What mattered

• Fault isolation: one destination's outage must not delay delivery to the others. Customers rely on timely delivery.
• Developer productivity: the team added about three destinations a month and maintained over a hundred.
• Operability: a small team on call for the whole pipeline.

3. The path, in three architectures

4. The decision

Decision: Merge all server-side destinations into one service and one repo

Context: 140+ destination services, each with its own queue, repo and
  autoscaling; shared libraries diverged across them; on-call paged for load
  spikes on low-traffic destinations; a shared-library change meant testing
  and deploying dozens of services.
Decision: Replace per-destination queues with Centrifuge, which sends events
  to one destination service. Move all destination code into one repository
  with one version of each of the 120 dependencies. Record and replay partner
  HTTP traffic in tests (Traffic Recorder) so the suite is fast and reliable.
Consequences:
  + One deploy for a shared-library change; a large worker pool absorbs spikes
  + Tests for all destinations run in milliseconds instead of up to an hour
  - A crash bug in one destination can take down delivery for all
  - In-memory caches are spread over 3,000+ processes, so hit rates fall
  - Upgrading a dependency may break several destinations at once
Kept: isolation of failing endpoints, now inside Centrifuge (queues per
  customer and destination, retries and archiving)

5. How it works

One destination, X, starts timing out. Watch what each architecture does with its events.

6. The numbers

• Over 140 destination services before the merge, adding about three a month.
• 120 unique dependencies, unified to one version each.
• Some destinations took up to 5 minutes to test; the full suite up to an hour. With recorded traffic, all 140+ destinations' tests ran in milliseconds.
• Shared-library improvements: 32 in the last year of microservices, 46 in the first year of the monolith.

7. What happened

The post reports that developer productivity "substantially improved": one engineer can deploy the service in minutes, and the team no longer gets paged for small destinations' load spikes. It is equally clear about the costs, listed as trade-offs in the decision above, and that the team was working on a way to stop one destination crashing the whole service while keeping the monolith. Centrifuge, the system built to make the merge possible, later became the infrastructure behind Segment's Connections product.

The authors' own conclusion is precise: microservices "work well" in some parts of their infrastructure; the server-side destinations, where every service did nearly the same job with the same libraries, were the wrong place for them.

8. Lessons

• Solve the real problem at the right level. The problem was head-of-line blocking in a queue. A queue per destination solved it; a service and a repository per destination added costs the problem didn't need. Isolation of the backlog moved into Centrifuge, where it belonged.
• Shared libraries across services are a hidden monolith. 140 services that must all take the same library change are one deployment unit split badly: high coupling, low independence (Module 07, 31).
• Granularity has a cost per service. Deploys, scaling configs, dashboards and pages each grow with the count (Module 16). Ask what each split buys.
• The enabler was a test suite, not an architecture. Recorded partner traffic made one repository safe. Tooling decides which structures you can afford (technical debt in the form of flaky tests drove the original split).
• Write down what you give up. The post's trade-offs section is a model consequences section for an ADR (Module 27).

9. Your turn

1. Count the cost per service (10 min). For a system you know: how many deploys, dashboards, alerts and scaling configs does each service add?
2. Find the shared library (10 min). Which library do most of your services depend on? How many versions of it are running today?
3. Redesign the isolation (10 min). Where else could Segment have isolated a failing destination without a service per destination?

10. Check yourself

11. Sources and further reading