TSB: when a big-bang migration fails

Over one weekend in April 2018, the UK bank TSB moved most of its operations and customer data from Lloyds Banking Group's systems to a newly built platform. When it went live, internet, mobile, telephone and branch banking all ran into serious problems. There was no way back to the old platform. The bank didn't return to business as usual until December. The regulators' findings make it one of the best-documented migration failures, and almost every lesson in this set shows up in it.

1. The situation

TSB was created in June 2014 when Lloyds Banking Group (LBG) divested it. It kept running on the LBG IT platform under an outsourcing agreement that allowed it to stay until July 2024. In July 2015, the Spanish bank Sabadell bought TSB; the rationale included the returns from moving TSB onto Sabadell's own platform, Proteo. The aim was to migrate by the end of 2017.

The FCA calls the scale "unprecedented". The target was a newly built UK version, Proteo4UK, "not yet proven", needing "significant customisation and a large number of external suppliers". Sabadell's IT subsidiary, SABIS, would design, build, test and then run it, relying on 85 third parties of its own. TSB had about 5.2 million customers.

2. What mattered

• The business case: the returns from a full migration were part of the reason for the takeover.
• A single view for customers: TSB judged that migrating by product or sort code would stop customers seeing all their products together.
• Cost and difficulty of a staged move between two providers' platforms, and TSB's belief that LBG was resistant to one.
• Resilience: a bank's customers need to reach their money every day, through every channel.

3. The options

Some functions did move earlier, in "Governed Transition Events" from 2017: the Faster Payments system, the mobile app and ATMs. Sabadell's usual method was a "big bang" migration in a single weekend, and TSB "opted for a predominantly single event data migration".

4. The decision, and what it left out

Decision: Migrate most of TSB's operations and data to Proteo4UK
  in one Main Migration Event, 20-22 April 2018

Context: TSB on LBG's platform; a newly built Proteo4UK, delivered and
  run by SABIS with 85 third parties; some functions already moved.
Consequences recorded by the regulators:
  - No roll-back: after the event it was "essentially impossible" to
    revert to the LBG platform
  - Non-functional testing reduced: around the end of February 2018, a
    decision "outside of the appropriate governance forum" cut testing in
    Active-Active configuration across TSB's data centres
  - Supplier readiness assumed: TSB did not sufficiently understand
    SABIS's capability to operate the platform
  - Incident plans sized for days: playbooks covered the first 48 hours

The risk was known. Board minutes from December 2015 record that the board "could not be put in a position where they gave up the carve-out option, migration 'stalled' or was found to be unworkable leaving the bank in a position without an IT infrastructure."

5. How it works

The programme from 2015 to the weekend of the migration, and what followed.

6. The technique in code

What TSB couldn't do: move customers in waves, stop at a readiness gate, and move a wave back. The program compares four plans when a platform defect appears early. The waves and the staff pilot are invented; the 5.2 million is TSB's customer count. It was run in all three languages; each prints the output below it.

CUSTOMERS = 5_200_000


def migrate(name, waves, pilot=0, gate=True, roll_back=True, defect_in_wave=1):
    """Move customers in waves (wave 0 = an optional pilot). A platform defect shows up in the given wave.
    With a readiness gate, migration stops there; everyone moved so far is hit by the defect."""
    sizes = ([pilot] if pilot else []) + [CUSTOMERS // waves + (1 if i < CUSTOMERS % waves else 0) for i in range(waves)]
    moved = 0
    for wave, size in enumerate(sizes, start=0 if pilot else 1):
        moved += size
        if gate and wave >= defect_in_wave:
            break                         # stop and fix before the next wave
    after = "can be moved back while it's fixed" if roll_back else "stuck on the new platform until it's fixed"
    print(f"{name:<30} {moved:>9,} customers hit; {after}")


migrate("single event, no way back", waves=1, gate=False, roll_back=False)
migrate("10 waves, no readiness gate", waves=10, gate=False)
migrate("10 waves, gate, roll-back", waves=10)
migrate("staff pilot, then 10 waves", waves=10, pilot=5_000, defect_in_wave=0)

const CUSTOMERS = 5_200_000;

function migrate(name: string, waves: number, { pilot = 0, gate = true, rollBack = true, defectInWave = 1 } = {}) {
  // Move customers in waves (wave 0 = an optional pilot). A platform defect shows up in the given wave.
  // With a readiness gate, migration stops there; everyone moved so far is hit by the defect.
  const sizes = [...(pilot ? [pilot] : []), ...Array.from({ length: waves }, (_, i) => Math.floor(CUSTOMERS / waves) + (i < CUSTOMERS % waves ? 1 : 0))];
  let moved = 0;
  for (let i = 0; i < sizes.length; i++) {
    const wave = pilot ? i : i + 1;
    moved += sizes[i];
    if (gate && wave >= defectInWave) break;   // stop and fix before the next wave
  }
  const after = rollBack ? "can be moved back while it's fixed" : "stuck on the new platform until it's fixed";
  console.log(`${name.padEnd(30)} ${moved.toLocaleString("en-US").padStart(9)} customers hit; ${after}`);
}

migrate("single event, no way back", 1, { gate: false, rollBack: false });
migrate("10 waves, no readiness gate", 10, { gate: false });
migrate("10 waves, gate, roll-back", 10);
migrate("staff pilot, then 10 waves", 10, { pilot: 5_000, defectInWave: 0 });

import java.util.*;

public class Main {
    static final int CUSTOMERS = 5_200_000;

    static void migrate(String name, int waves, int pilot, boolean gate, boolean rollBack, int defectInWave) {
        // Move customers in waves (wave 0 = an optional pilot). A platform defect shows up in the given wave.
        // With a readiness gate, migration stops there; everyone moved so far is hit by the defect.
        List<Integer> sizes = new ArrayList<>();
        if (pilot > 0) sizes.add(pilot);
        for (int i = 0; i < waves; i++) sizes.add(CUSTOMERS / waves + (i < CUSTOMERS % waves ? 1 : 0));
        int moved = 0;
        for (int i = 0; i < sizes.size(); i++) {
            int wave = pilot > 0 ? i : i + 1;
            moved += sizes.get(i);
            if (gate && wave >= defectInWave) break;   // stop and fix before the next wave
        }
        String after = rollBack ? "can be moved back while it's fixed" : "stuck on the new platform until it's fixed";
        System.out.println(String.format(Locale.US, "%-30s %,9d customers hit; %s", name, moved, after));
    }

    public static void main(String[] args) {
        migrate("single event, no way back", 1, 0, false, false, 1);
        migrate("10 waves, no readiness gate", 10, 0, false, true, 1);
        migrate("10 waves, gate, roll-back", 10, 0, true, true, 1);
        migrate("staff pilot, then 10 waves", 10, 5_000, true, true, 0);
    }
}

single event, no way back      5,200,000 customers hit; stuck on the new platform until it's fixed
10 waves, no readiness gate    5,200,000 customers hit; can be moved back while it's fixed
10 waves, gate, roll-back        520,000 customers hit; can be moved back while it's fixed
staff pilot, then 10 waves         5,000 customers hit; can be moved back while it's fixed

Waves have real costs, which the FCA records as TSB's reasons: running two platforms, a staged move between two providers, and customers seeing their products split across systems. The point isn't that waves are free. It's that a plan with no gate and no way back puts every customer behind one decision.

7. The numbers

• About 5.2 million customers.
• 20–22 April 2018: the Main Migration Event; live on 22 April.
• 10 December 2018: TSB back to business as usual.
• 225,492 complaints (about 4.3% of customers) and £32.7 million in redress, from 22 April 2018 to 7 April 2019.
• £48.65 million in fines in December 2022: FCA £29.75 million, PRA £18.9 million, after a 30% settlement discount.
• £81,620: the PRA's fine on TSB's former Chief Information Officer, April 2023.

Try it: plan the cutover

A platform defect shows up early. How many customers does it reach under your plan? Waves and pilot sizes are illustrative.

8. What happened

From early after go-live, customers met data breaches, failures in internet and mobile banking, failures in telephone banking, branch technology failures and problems with card payments. The FCA describes a cascade: digital failures pushed customers to the phones, which were overloaded, which pushed them into branches, where systems were running slowly. The direct causes "substantially related to issues with IT configuration, capacity and coding". The FCA notes that had the reduced Active-Active testing been done, a configuration problem in the data centres that took internet and mobile banking down "would have been found".

The FCA found TSB in breach of Principle 3: it failed to organise and control the programme with adequate risk management, including its outsourcing to SABIS. TSB itself told the regulators it would not have gone ahead had it thought an incident of that scale might happen.

9. Lessons

• A cutover with no way back needs the strongest evidence. TSB's own board named the risk in 2015. When roll-back is impossible, readiness has to be proven, not assumed (Module 28).
• Non-functional testing is where big migrations fail. Load and data-centre configuration, not features, caused the outage. Cutting that testing was a risk decision, taken as "a purely technical decision".
• An outsourced platform is still your platform. The regulators' central finding was about TSB's oversight of SABIS and its suppliers.
• Plan for weeks, not 48 hours. Incident playbooks and call-centre capacity sized for a short disruption were overwhelmed by one that lasted months.
• Big bang is a choice with reasons. TSB's reasons were real (cost, a single customer view). Weigh them against waves, a pilot and a way back, in writing (Module 30a, Module 27).

10. Your turn

1. Find your roll-back (10 min). For the riskiest change on your roadmap: after go-live, how would you go back, and until when can you?
2. List the non-functional tests (10 min). Load, failover, data-centre configuration: which ones are planned, and who could cut them?
3. Design a wave plan (10 min). Who would go first (staff, a region, a product), and what would have to be true before the next wave?

11. Check yourself

12. Sources and further reading