Design a payment system

A payment system handles only a few requests a second next to a feed or a chat app, yet it is one of the hardest interview designs, because every failure costs money. A response lost on the network can charge a customer twice; a crash at the wrong moment can take money with no record of why. The whole design is about making every step safe to retry and every cent accounted for. It follows the interview framework phase by phase.

1. The prompt

"Design the payment system for an online shop" (or Stripe's side of it, or a marketplace's payouts): customers pay for orders by card; sellers are paid out later; finance needs records that always add up.

The numbers are small. The design effort goes into correctness across systems you don't control: the customer's flaky network, your own crashes, and a payment provider that sometimes times out after doing the work.

2. Frame the problem

3. What it must do

1. Charge a customer for an order through a PSP.
2. Record every movement of money and show payment status to the customer and the shop.
3. Refund payments and pay sellers out on a schedule.

Not now:

• Building card networks ourselves.
• Fraud models (mention them).
• Currency exchange.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Payment: one attempt to collect money for an order.
• Idempotency record: a key and the saved result of the first request.
• Ledger entry: one side of a money movement; never updated.
• Outbox event: a message saved with the data, published later.

payments      payment_id, order_id, amount (minor units), currency, status, psp_ref, idempotency_key
idempotency   key → request_hash, status: in_progress | done, saved_response, created_at
ledger        entry_id, txn_id, account, direction: debit | credit, amount, currency, created_at  (append-only)
outbox        event_id, type, payload, created_at, published_at
psp_events    psp_event_id (unique), received_at                   (webhook dedup)

7. The contract

POST /v1/payments   Idempotency-Key: 7c9e…   {order_id, amount: 4000, currency: "USD", payment_token: "tok_…"}
     → 201 {payment_id, status: "succeeded"}        (a retry with the same key returns this same response)
GET  /v1/payments/{id}                              → status
POST /v1/payments/{id}/refunds   Idempotency-Key: …
POST /psp/webhooks   (signed by the PSP)            → 200, processed asynchronously

Amounts are integers in the currency's smallest unit (4000 = $40.00), never floating point.

Data flow: (1) the card goes straight to the PSP, which returns a token → (2) the app calls our API with the token and a key → (3) we save the key, then charge the PSP with our own key → (4) one transaction saves the status, ledger entries and an outbox event → (5) the event tells the order service → (6) reconciliation checks it all the next day.

8. First working version

A payment API, a PSP, and a payments table with a status column.

8.1 Paying

Version one charges the PSP and then saves the result. The happy path works; the unhappy one charges twice.

8.2 Showing status

Read the payment row by id.

8.3 Refunds and payouts

Call the PSP's refund API and set the status to refunded; pay sellers by summing their payments.

9. Make it hold up

9.1 Idempotency, end to end exactly-once effect

Two details interviewers probe: save the key as in progress before calling the PSP, so a concurrent repeat waits or gets "still processing" instead of charging again; and store a hash of the request with the key, so the same key with a different amount is rejected rather than replayed.

9.2 A double-entry ledger auditable

Never store "balance = 120" and update it. Record every movement as a transaction of entries that sum to zero: a $40 payment debits what the PSP owes us and credits what we owe the shop; the PSP's $1.20 fee is a further pair. Balances are sums of entries. Entries are append-only: a mistake is fixed with a new reversing transaction, so history explains every number.

9.3 A state machine and webhooks exactly-once effect

• States: created → processing → succeeded / failed, then refunded. Transitions are conditional updates (WHERE status = 'processing'), so a late or duplicate message can't move a payment backwards.
• Webhooks: some outcomes arrive later (bank confirmations, disputes). PSPs retry webhook delivery and may send an event more than once or out of order, so verify the signature, record the event id, and ignore ones already processed.

9.4 The outbox: saved and announced together never lose a payment

The payment status, ledger entries and a "PaymentSucceeded" event are written in one database transaction; a relay publishes the event to the order service (Module 22). There's no "saved but never announced" gap and no "announced but never saved" one. Delivery is at-least-once, so consumers ignore duplicates by event id.

9.5 Reconciliation: trust, but verify auditable

Every day the PSP sends a settlement report of what it actually charged and paid out. A reconciliation job matches it line by line against the ledger. Mismatches (a charge we never recorded because we crashed, a refund the PSP never got) go to an automated fix or a person. This is the last line of defence, and real payment teams rely on it. Payments stuck in processing are also checked during the day by asking the PSP for their status, using our key.

9.6 Security secure

Use the PSP's hosted payment page or embedded fields so card numbers go straight from the browser to the PSP; we only see a token. That keeps most of the PCI DSS requirements with the PSP. Encrypt data at rest, restrict who can issue refunds, require two people for large manual adjustments, and log every action for audit.

9.7 The whole design after the deep dives

One payment from tap to books, including a retry that doesn't charge twice. Press Play or drag the timeline.

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Synchronous vs asynchronous confirmation: waiting for the PSP gives the customer an instant answer; some methods finish later and need webhooks and a "processing" state in the UI.
• One PSP vs several: routing between PSPs improves availability and cost, but each adds integration, reconciliation and idempotency handling.
• A ledger over a status column: more rows, but every number can be explained.
• Correctness over latency everywhere.

What we'd watch:

• Reconciliation mismatches (should be ~0).
• Payments stuck in processing.
• PSP error and timeout rates.
• Outbox age.
• Refund rate.

At 10×:

• Partition by merchant.
• A ledger service.
• Multiple PSPs.
• Streaming reconciliation.

Common mistakes:

• Idempotency only at the API, not the PSP.
• Floating-point amounts.
• Updating balances in place.
• Trusting webhook order.
• No reconciliation.

13. Try it, check yourself, read more

Choose what goes wrong and how well the system is protected, then watch the messages play out. Count how many times the customer is charged.