Spotify: Backstage and golden paths

Spotify organises engineering around autonomous teams. Autonomy brought speed, and also a fragmented ecosystem of tools where, as Spotify put it, "the only way to find out how to do something was to ask your colleague". They called it rumour-driven development. Their answer was not a rule book or an approval board but a platform: a recommended, supported way to build each kind of software (the Golden Path), and one portal where every service, its owner, its documentation and its tools can be found (Backstage). It is governance by making the right way the easy way.

1. The situation

Spotify's teams choose much of how they work. As the company grew, each engineer needed more tools and more specialist knowledge, across backend, data, mobile and machine learning, and the tooling fragmented. Finding the owner of a service, the recommended way to create a data pipeline, or the documentation for an internal library meant asking around. Spotify's post says this "might work in a startup, even a large startup", but as it grew, rumour-driven development "simply wasn't scalable".

2. What mattered

• Keep team autonomy. No central approval of tool choices.
• Discoverability: the recommended tools must be easy to find, the path through them clear, with good instructions and obvious support.
• Lower maintenance cost: fewer variants of the same thing, so upgrades can be automated.
• Onboarding speed: new engineers productive quickly.

3. The options

4. The decision

Decision: Golden Paths, delivered through a developer portal (Backstage)

Golden Path: "the opinionated and supported path to build something" (a backend
  service, a website, a data pipeline), with a step-by-step tutorial per
  discipline: backend, client, data engineering, data science, ML, web, audio.
Golden State: a list of checks that shows whether a system is on the path.
Backstage: one portal over all infrastructure tooling, services and docs:
  - software catalogue: every component and its owning team
  - templates: create a new service on the Golden Path in a few clicks
  - TechDocs: documentation written like code, next to it
  - plugins: each infrastructure tool appears in the same interface
Consequences:
  + New engineers find tools, owners and docs without asking around
  + Fewer variants, so upgrades can be automated
  - The platform must earn adoption; a neglected path gets abandoned
  - Leaving the path is allowed, so some fragmentation remains by design

5. How it works

A new engineer needs to build a backend service that reads from another team's service. Before Backstage, and with it.

6. The numbers

• When Backstage was open-sourced in March 2020, over 280 engineering teams inside Spotify used it to manage more than 2,000 backend services, 300 websites, 4,000 data pipelines and 200 mobile features.
• Spotify reported a 55% decrease in onboarding time, measured as time until an engineer's 10th pull request.
• Backstage had been built internally about four years earlier; the first Golden Path tutorial, for backend engineering, about six years before the 2020 post, as a Hack Week project by eight engineers.

7. What happened

Golden Paths spread beyond backend engineering to client development, data engineering, data science, machine learning, web and audio processing. The idea was popular enough to spawn spin-off terms inside Spotify (a Paved Road, a Silver Path). Spotify open-sourced Backstage in March 2020 with a roadmap of three phases: a consistent UI for tools, a central software inventory with clear ownership, and an ecosystem of plugins. It has since grown into an open-source framework with its own conference, BackstageCon, held alongside KubeCon.

Spotify's post is clear about intent: Golden Paths are "not to limit or stifle engineers, or set standards for the sake of it", but so that teams "don't have to reinvent the wheel, have fewer decisions to make" and can spend their creativity on product problems.

8. Lessons

• Platforms reduce cognitive load. A platform team offering a self-service path is Team Topologies' platform team; its product is the developer experience (Module 09a).
• Make the right thing the easy thing. A supported default that is easier than the alternatives governs better than a mandate, and keeps autonomy (Module 33's strategies for standardising).
• A catalogue with owners is governance infrastructure. Knowing who owns and who depends on each component is what makes the advice process possible (C02).
• Golden State is a fitness function for drift. Checks that show whether a system is still on the path turn a recommendation into something measurable (Module 26).
• Documentation next to code stays true. TechDocs is docs-as-code (Module 32).

9. Your turn

1. Count the rumours (10 min). List five questions new engineers ask in chat in their first month. Which could a catalogue, a template or docs answer?
2. Write one Golden Path (15 min). The steps to create, deploy and monitor a new backend service in your organisation, as a tutorial. Where does it break?
3. Define a Golden State (5 min). Five checks that show whether a service is on the path.

10. Check yourself

11. Sources and further reading