Software architecture · Level 5: Data and transactions · Module 23a of 46

Analytical data architecture

Module 23 kept each service's data private and fed reporting with events. This module is about where that data goes next. Analysts, finance and machine-learning teams need years of history from every part of the business, joined together and queried in ways no service was designed for. That is analytical data, and it needs its own architecture: where it is stored (warehouse, lake or lakehouse), and who owns it on the way (a central data team, or the domains themselves in a data mesh).

1. The idea

A hotel with one central kitchen versus a food hall. In the hotel, every dish comes from one kitchen whose chefs must learn every cuisine, and every new dish waits in their queue. In a food hall, each stall cooks its own cuisine and serves it to anyone. The hall doesn't cook: it provides power, water and seating, and its inspectors make every stall follow the same hygiene rules. The food hall scales to fifty cuisines; it only works if every stall is willing to run a kitchen.

Operational and analytical data

Operational data runs the business now: the current order, today's stock count, a customer's address. It is written by one service, read by key, and must be correct this second. Analytical data explains the business over time: revenue by region for three years, which campaigns brought customers who stayed. It is read in huge scans, joined across domains, and almost never updated. The two need different storage, different shapes and, often, different owners. Analytical queries running on operational databases slow the business down; operational schemas exposed to analysts freeze every service (Module 23).

Warehouse, lake and lakehouse

Store	What it is	Good at	Weak at
Data warehouse	A database for analysis: cleaned, modelled tables, often a star schema of facts (orders) and dimensions (customer, product, date). Schema on write	Fast, trusted SQL for reports and dashboards	Raw, semi-structured data; machine learning; cost at very large volume
Data lake	Raw files (JSON, CSV, Parquet, images) in cheap object storage. Schema on read	Keeping everything cheaply; data science on raw data	No transactions, no updates, no guarantees: without discipline it becomes a "data swamp"
Lakehouse	Lake files plus an open table format (Apache Iceberg, Delta Lake, Apache Hudi) that adds a transaction log	Warehouse-like tables (atomic writes, schemas, deletes, time travel) on lake storage, readable by many engines	More moving parts to run; performance depends on file layout and maintenance

Data mesh: four principles

Zhamak Dehghani described data mesh in 2019 as the answer to a central data team that can't keep up. It is an organisational architecture more than a technology, and it is Conway's law (Module 09a) applied to analytical data:

Principle	Meaning
Domain ownership	The team that owns a domain's operational data also owns its analytical data, end to end
Data as a product	Each domain publishes data products for others: discoverable, addressable, trustworthy, self-describing, interoperable and secure, with an owner and a contract
Self-serve data platform	A platform team provides storage, pipelines, a catalogue and access control as a service, so domain teams don't each build their own (a platform team in Team Topologies terms)
Federated computational governance	Global rules (identifiers, privacy tags, versioning, quality checks) agreed by domain and platform representatives, and enforced automatically by the platform: fitness functions for data (Module 26)

Software Architecture: The Hard Parts calls a data product beside its service a data product quantum: deployed separately from the service, coupled to it only through the service's events or change feed.

2. How it works

One column rename, two organisations

A shop's revenue dashboard is fed from the Orders service. On Sunday evening the Orders team renames a column in its own database. Watch what happens with a central pipeline, then with a data product.

How a lakehouse table makes files behave like a table

An orders table stored as Parquet files in object storage, with a transaction log. A nightly job appends a day of orders while an analyst is querying.

3. Try it yourself

Central data team or data mesh?

Choose how many domains produce analytical data and how big the central data team is. See how long a change request waits, and how many engineers each approach needs.

Assumed figures, to show the shape: each domain raises 4 change requests a month (new fields, new sources, fixes); a central engineer completes 6 a month, slower than a domain team because they must first learn the domain. Central lead time is modelled as a queue: 0.6 weeks ÷ (1 − utilisation). The mesh needs a 4-person platform team plus half an engineer per domain, and a domain changes its own product in about a week.

4. Trade-offs

Central team vs data mesh

A central team is cheaper and more consistent with few domains, and becomes a queue as the domains grow. A mesh removes the queue and puts the knowledge with the data, at the cost of every domain team running data products well.

Schema on write vs schema on read

Modelling data on the way in (warehouse) gives trusted tables and slows down every new source. Storing raw and modelling on read (lake) is fast to land and slow to trust. Most platforms do both, in layers.

Fresh vs cheap

Nightly batches are simple and cheap; streaming changes in minutes needs always-on pipelines and careful handling of late and duplicate events. Ask what decision the freshness changes before paying for it.

Open formats vs one managed warehouse

Open table formats let several engines read the same tables and make switching easier. A single managed warehouse is simpler to run and ties data and compute to one vendor.

5. In real projects

Land, clean, serve. Many platforms organise tables in three layers, often called bronze (raw, as received), silver (cleaned, deduplicated, joined to reference data) and gold (modelled for a use: a revenue fact table, a churn feature set). Each layer is rebuilt from the one before, so a bug in cleaning is fixed by re-running, not by repairing data by hand.

Feed it from events, not from the back door. Change data capture or domain events (Module 23) move operational data in without the platform reading services' private tables. The contract rules of Module 25 apply: version the schema, deprecate old versions with a date, and check compatibility in the producer's build.

Don't start with a mesh. With a handful of domains and one data team, a central platform is right. Move towards a mesh when the central team's queue, not its skill, limits the business, and build the self-serve platform and the governance rules before handing ownership to domains. A mesh announced as a reorganisation, without the platform, gives every team the job and none of the tools.

For the infrastructure underneath (ingestion, batch and stream processing, the catalogue), see the System design company data platform design and Module 23: batch and stream processing.

Say it in the interview

Two-sentence answers

"Warehouse, lake or lakehouse?" A warehouse for modelled, trusted SQL; a lake for cheap raw storage; a lakehouse when you want both on one copy of the data, using an open table format that adds transactions to the lake's files.
"What is data mesh?" Domain teams own and publish their analytical data as products, on a self-serve platform, under global rules the platform enforces. It fixes a central data team that has become the bottleneck, and needs mature domain teams.
"How do you stop a schema change breaking dashboards?" Treat analytical data as a published contract: the producer's build checks compatibility, breaking changes get a new version, and the old one is kept for an agreed period.

Red flags interviewers notice

Analytical queries running against the production database.
A lake with no catalogue, owners or schemas.
"We'll adopt data mesh" with three domains and no platform team.

6. Remember it

Memory hook: "Food hall, not central kitchen"Each domain cooks and serves its own data as a product; the hall provides the kitchen equipment and enforces the hygiene rules.

How this hook works: what "food hall, not central kitchen" means

What the slogan means, phrase by phrase

"Food hall": Many independent stalls (domains), each serving its own dishes (data products) to any customer.
"not central kitchen": No single team that must learn every cuisine and becomes the queue for every new dish.

Why this picture fits: the hall's own job, equipment and inspections, is exactly the self-serve platform and federated governance; without them, a food hall is just a car park full of camping stoves.

The story → the concept

Food hall	Analytical data
A stall and its cuisine	A domain and its data
A dish on the menu	A data product with a contract
Power, water, seating	The self-serve data platform
Hygiene inspectors	Federated, automated governance
The central kitchen	A central data team and its pipelines

The trap it saves you from

A central data team that owns every pipeline, understands no domain, and sits between every question and its answer.

7. Common mistakes

The data swampDumping everything into a lake with no owners, schemas or catalogue. Nobody can find or trust anything, so nobody uses it.

Pipelines that read private tablesExtracting straight from services' databases couples the platform to every schema, and breaks it on every rename.

Mesh as a reorganisationGiving domains ownership without a platform or governance rules. Every team builds its own pipelines, differently.

Buying a meshData mesh is about who owns what. A product labelled "data mesh" doesn't change ownership.

8. Design exercise (30 minutes)

Find the consumers (10 min). For a system you know, list three analytical questions the business asks, and which domains' data each one needs.
Design one data product (10 min). Pick one domain. Write its data product's contract: tables or events, meaning of each field, freshness, owner, and how breaking changes are versioned.
Choose the model (10 min). Count the domains and the change requests a month. Central team, mesh, or central now with a plan to move? Write down what would make you switch.

9. Self-check quiz

What does an open table format such as Apache Iceberg or Delta Lake add to a data lake?

Readers see only committed versions; writes are atomic; old versions can be queried until cleaned up.

In a data mesh, who owns the orders data product?

Domain ownership: the people who understand the data publish it.

Which data mesh principle stops every domain team building its own pipeline tooling?

A platform team provides storage, pipelines, catalogue and access control as a service.

How does a data mesh stop a column rename breaking a dashboard?

Breaking changes are caught where they're made, and published as a new version.

Where is a data mesh least likely to pay off?

With few domains, a central team is cheaper and the mesh's platform overhead isn't repaid.

10. Go deeper

Resource	Time	Why
Data Mesh Principles and Logical Architecture (Zhamak Dehghani)	45 min	The four principles, from their author.
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (Zhamak Dehghani)	30 min	The original 2019 article: why central platforms stall, and data products' qualities.
Lakehouse: A New Generation of Open Platforms (CIDR 2021)	40 min	The paper that defined the lakehouse and its transaction log.
Dimensional modeling techniques (Kimball Group)	reference	Facts, dimensions and star schemas.
Software Architecture: The Hard Parts	book	The chapter on managing analytical data and the data product quantum.