Klarna: when the launch metric isn't the outcome

In February 2024, Klarna announced that its AI assistant had handled 2.3 million conversations in its first month: two-thirds of its customer service chats, "the equivalent work of 700 full-time agents". Fifteen months later its CEO told Bloomberg the AI focus had led to "lower quality", and that "investing in the quality of human support is the way of the future for us." Both statements came from Klarna. The gap between them is the lesson.

1. The situation

Klarna, a payments company, launched an assistant built with OpenAI to handle customer service chats in 23 markets, around the clock, in more than 35 languages. A customer service operation is a natural first target for AI: high volume, repeated questions and a clear cost per contact.

2. What Klarna said, and when

The launch announcement did report quality measures: satisfaction on par with people and fewer repeat questions, after one month. The later remarks don't say which measures changed, or by how much. What the public record shows is that headline numbers at launch didn't settle the question, and that the company itself changed course.

3. Launch metrics and outcome metrics

Two kinds of number tell different stories about the same assistant.

4. The control in code

A metrics report that puts launch metrics (volume, speed, cost) next to outcome metrics (first-contact resolution, repeat contacts, requests for a person) and warns when the first improve while the second get worse. The company and its numbers are invented, not Klarna's. The program was run in all three languages; each prints the output below it.

# An invented company's monthly support metrics before and after an AI assistant launch (not Klarna's data).
# "better": which direction is an improvement for each metric.
METRICS = [
    # name,                         before, after, better,  kind
    ("chats handled by AI (%)",          0,    60, "higher", "launch"),
    ("average time to resolve (min)",   12,     3, "lower",  "launch"),
    ("cost per contact ($)",          5.00,  1.50, "lower",  "launch"),
    ("resolved on first contact (%)",   78,    64, "higher", "outcome"),
    ("contacts back within 7 days (%)", 12,    19, "lower",  "outcome"),
    ("asked for a human (%)",            4,    14, "lower",  "outcome"),
]


def change(before, after, better):
    improved = after > before if better == "higher" else after < before
    return improved, after - before


print(f"{'metric':<33}{'before':>8}{'after':>8}  verdict")
worse_outcomes = []
for name, before, after, better, kind in METRICS:
    improved, delta = change(before, after, better)
    verdict = "better" if improved else "WORSE"
    if kind == "outcome" and not improved:
        worse_outcomes.append(name)
    print(f"{name:<33}{before:>8g}{after:>8g}  {verdict} ({kind})")

launch_ok = all(change(b, a, d)[0] for _, b, a, d, k in METRICS if k == "launch")
print(f"launch metrics all better: {launch_ok}; outcome metrics worse: {len(worse_outcomes)}")
if launch_ok and worse_outcomes:
    print("warning: speed and cost improved while customers' problems got solved less often")

// An invented company's monthly support metrics before and after an AI assistant launch (not Klarna's data).
// "better": which direction is an improvement for each metric.
type Metric = [name: string, before: number, after: number, better: "higher" | "lower", kind: "launch" | "outcome"];
const METRICS: Metric[] = [
  ["chats handled by AI (%)", 0, 60, "higher", "launch"],
  ["average time to resolve (min)", 12, 3, "lower", "launch"],
  ["cost per contact ($)", 5.0, 1.5, "lower", "launch"],
  ["resolved on first contact (%)", 78, 64, "higher", "outcome"],
  ["contacts back within 7 days (%)", 12, 19, "lower", "outcome"],
  ["asked for a human (%)", 4, 14, "lower", "outcome"],
];

const improved = (before: number, after: number, better: string) => (better === "higher" ? after > before : after < before);

console.log(`${"metric".padEnd(33)}${"before".padStart(8)}${"after".padStart(8)}  verdict`);
const worse: string[] = [];
for (const [name, before, after, better, kind] of METRICS) {
  const ok = improved(before, after, better);
  if (kind === "outcome" && !ok) worse.push(name);
  console.log(`${name.padEnd(33)}${String(before).padStart(8)}${String(after).padStart(8)}  ${ok ? "better" : "WORSE"} (${kind})`);
}
const launchOk = METRICS.filter((m) => m[4] === "launch").every(([, b, a, d]) => improved(b, a, d));
console.log(`launch metrics all better: ${launchOk ? "True" : "False"}; outcome metrics worse: ${worse.length}`);
if (launchOk && worse.length) console.log("warning: speed and cost improved while customers' problems got solved less often");

import java.math.BigDecimal;
import java.util.*;

public class Main {
    record Metric(String name, double before, double after, String better, String kind) {}

    // An invented company's monthly support metrics before and after an AI assistant launch (not Klarna's data).
    static final List<Metric> METRICS = List.of(
        new Metric("chats handled by AI (%)", 0, 60, "higher", "launch"),
        new Metric("average time to resolve (min)", 12, 3, "lower", "launch"),
        new Metric("cost per contact ($)", 5.00, 1.50, "lower", "launch"),
        new Metric("resolved on first contact (%)", 78, 64, "higher", "outcome"),
        new Metric("contacts back within 7 days (%)", 12, 19, "lower", "outcome"),
        new Metric("asked for a human (%)", 4, 14, "lower", "outcome"));

    static boolean improved(Metric m) { return m.better().equals("higher") ? m.after() > m.before() : m.after() < m.before(); }
    static String num(double x) { return new BigDecimal(Double.toString(x)).stripTrailingZeros().toPlainString(); }

    public static void main(String[] args) {
        System.out.println(String.format("%-33s%8s%8s  verdict", "metric", "before", "after"));
        int worse = 0;
        boolean launchOk = true;
        for (Metric m : METRICS) {
            boolean ok = improved(m);
            if (m.kind().equals("outcome") && !ok) worse++;
            if (m.kind().equals("launch") && !ok) launchOk = false;
            System.out.println(String.format("%-33s%8s%8s  %s (%s)", m.name(), num(m.before()), num(m.after()), ok ? "better" : "WORSE", m.kind()));
        }
        System.out.println("launch metrics all better: " + (launchOk ? "True" : "False") + "; outcome metrics worse: " + worse);
        if (launchOk && worse > 0) System.out.println("warning: speed and cost improved while customers' problems got solved less often");
    }
}

metric                             before   after  verdict
chats handled by AI (%)                 0      60  better (launch)
average time to resolve (min)          12       3  better (launch)
cost per contact ($)                    5     1.5  better (launch)
resolved on first contact (%)          78      64  WORSE (outcome)
contacts back within 7 days (%)        12      19  WORSE (outcome)
asked for a human (%)                   4      14  WORSE (outcome)
launch metrics all better: True; outcome metrics worse: 3
warning: speed and cost improved while customers' problems got solved less often

5. Try it: how far should the AI go?

Move more contacts to the AI. Launch metrics improve steadily. In this invented model, outcomes hold up while the AI takes simple contacts, then fall once it takes complex ones too, unless customers can still reach a person.

6. What it means

Klarna's announcements are a useful pair because they come from the same company, about the same assistant, fifteen months apart. A month of launch numbers can show the assistant is fast, cheap and busy. Whether customers' problems are solved, whether they come back, and whether they trust the service can take longer to show, and need their own measures. The company's own conclusion was to keep a person available.

7. Lessons

• Volume and speed are not the outcome. Measure whether the problem was solved: resolution, repeat contacts, escalations, complaints (Module 17).
• A month is a launch, not a result. Track the same measures for quarters, and compare against a control group if you can.
• Keep a route to a person. "There will always be a human if you want" became Klarna's own promise.
• Count the cost of failure, not just of contact. A cheap contact that brings the customer back twice isn't cheap (Module 19).
• Read launch numbers as claims. A company announcing its own AI has a reason to lead with the best ones (Module 21).

8. Your turn

1. Split your metrics (10 min). For an AI feature you know, list its launch metrics and its outcome metrics. Which are on the dashboard?
2. Find the delay (10 min). Which outcome would take longest to show a problem?
3. Design the escape hatch (10 min). How does a customer reach a person, and how fast?

9. Check yourself

10. Sources and further reading