Measuring AI productivity: 56% faster or 19% slower?

In 2023, a controlled experiment found developers finished a task 55.8% faster with GitHub Copilot. In 2025, a randomised trial found experienced open-source developers took 19% longer with AI tools, while believing they'd been 20% faster. In 2026, the same researchers reported that developers were probably faster now, and that their own data was weak evidence of by how much. None of these is wrong. Together they're the best lesson there is in how to measure AI.

1. The situation

Every team adopting AI tools asks the same question: are we faster? Vendors, surveys and studies give answers from "twice as fast" to "slower". Leaders have to decide what to buy, what to expect and how to tell if it's working (Module 21). These three studies, read together, show why the answers differ.

2. Three studies, three answers

METR is careful about its own limits: the 2025 result is "a snapshot of early-2025 AI capabilities in one relevant setting", and "we do not claim that our developers or repositories represent a majority or plurality of software development work."

3. Why they differ

Five things change the answer: the task, the people, the tools, what's measured, and who drops out.

4. Measuring it yourself, in code

A small trial analyser. Tasks are randomly assigned to AI or no AI before work starts; the effect is the difference in mean log time, with a 95% interval. On invented data, the same tools make boilerplate tasks much faster and deep work in a familiar codebase slower, and the overall number depends on the mix. With 14 tasks, the interval is wide enough to include both "much faster" and "much slower". The program was run in all three languages; each prints the output below it.

import math

# Invented tasks from a small randomised trial: (kind, condition, minutes).
# Each task was randomly assigned to "ai" or "no ai" before work started.
TASKS = [
    ("deep", "no ai", 60), ("deep", "no ai", 90), ("deep", "no ai", 45), ("deep", "no ai", 120),
    ("deep", "ai", 75), ("deep", "ai", 110), ("deep", "ai", 55), ("deep", "ai", 140),
    ("boilerplate", "no ai", 40), ("boilerplate", "no ai", 30), ("boilerplate", "no ai", 50),
    ("boilerplate", "ai", 22), ("boilerplate", "ai", 18), ("boilerplate", "ai", 28),
]
BELIEVED = -0.20   # what developers said afterwards: "about 20% faster"


def mean(xs):
    return sum(xs) / len(xs)


def var(xs):
    m = mean(xs)
    return sum((x - m) ** 2 for x in xs) / (len(xs) - 1)


def effect(tasks):
    """Change in time with AI (negative = faster), from the means of log times, with a 95% interval."""
    a = [math.log(m) for _, c, m in tasks if c == "ai"]
    b = [math.log(m) for _, c, m in tasks if c == "no ai"]
    diff = mean(a) - mean(b)
    se = math.sqrt(var(a) / len(a) + var(b) / len(b))
    return math.exp(diff) - 1, math.exp(diff - 1.96 * se) - 1, math.exp(diff + 1.96 * se) - 1


def report(label, tasks):
    e, lo, hi = effect(tasks)
    word = "longer" if e > 0 else "shorter"
    print(f"{label}: tasks took {abs(e):.0%} {word} with AI (95% interval {lo:+.0%} to {hi:+.0%}), n={len(tasks)}")


report("all tasks", TASKS)
report("deep work only", [t for t in TASKS if t[0] == "deep"])
report("boilerplate only", [t for t in TASKS if t[0] == "boilerplate"])
print(f"developers believed: {abs(BELIEVED):.0%} faster")

// Invented tasks from a small randomised trial: [kind, condition, minutes].
// Each task was randomly assigned to "ai" or "no ai" before work started.
type Task = [string, "ai" | "no ai", number];
const TASKS: Task[] = [
  ["deep", "no ai", 60], ["deep", "no ai", 90], ["deep", "no ai", 45], ["deep", "no ai", 120],
  ["deep", "ai", 75], ["deep", "ai", 110], ["deep", "ai", 55], ["deep", "ai", 140],
  ["boilerplate", "no ai", 40], ["boilerplate", "no ai", 30], ["boilerplate", "no ai", 50],
  ["boilerplate", "ai", 22], ["boilerplate", "ai", 18], ["boilerplate", "ai", 28],
];
const BELIEVED = -0.2;   // what developers said afterwards: "about 20% faster"

const mean = (xs: number[]) => xs.reduce((s, x) => s + x, 0) / xs.length;
const variance = (xs: number[]) => { const m = mean(xs); return xs.reduce((s, x) => s + (x - m) ** 2, 0) / (xs.length - 1); };
const pct = (x: number, sign = false) => `${sign && x >= 0 ? "+" : ""}${Math.round(x * 100)}%`;

function effect(tasks: Task[]): [number, number, number] {
  // Change in time with AI (negative = faster), from the means of log times, with a 95% interval.
  const a = tasks.filter((t) => t[1] === "ai").map((t) => Math.log(t[2]));
  const b = tasks.filter((t) => t[1] === "no ai").map((t) => Math.log(t[2]));
  const diff = mean(a) - mean(b);
  const se = Math.sqrt(variance(a) / a.length + variance(b) / b.length);
  return [Math.exp(diff) - 1, Math.exp(diff - 1.96 * se) - 1, Math.exp(diff + 1.96 * se) - 1];
}

function report(label: string, tasks: Task[]) {
  const [e, lo, hi] = effect(tasks);
  console.log(`${label}: tasks took ${pct(Math.abs(e))} ${e > 0 ? "longer" : "shorter"} with AI (95% interval ${pct(lo, true)} to ${pct(hi, true)}), n=${tasks.length}`);
}

report("all tasks", TASKS);
report("deep work only", TASKS.filter((t) => t[0] === "deep"));
report("boilerplate only", TASKS.filter((t) => t[0] === "boilerplate"));
console.log(`developers believed: ${pct(Math.abs(BELIEVED))} faster`);

import java.util.*;

public class Main {
    record Task(String kind, String condition, double minutes) {}

    // Invented tasks from a small randomised trial. Each task was randomly assigned to "ai" or "no ai" before work started.
    static final List<Task> TASKS = List.of(
        new Task("deep", "no ai", 60), new Task("deep", "no ai", 90), new Task("deep", "no ai", 45), new Task("deep", "no ai", 120),
        new Task("deep", "ai", 75), new Task("deep", "ai", 110), new Task("deep", "ai", 55), new Task("deep", "ai", 140),
        new Task("boilerplate", "no ai", 40), new Task("boilerplate", "no ai", 30), new Task("boilerplate", "no ai", 50),
        new Task("boilerplate", "ai", 22), new Task("boilerplate", "ai", 18), new Task("boilerplate", "ai", 28));
    static final double BELIEVED = -0.20;   // what developers said afterwards: "about 20% faster"

    static double mean(List<Double> xs) { double s = 0; for (double x : xs) s += x; return s / xs.size(); }
    static double variance(List<Double> xs) { double m = mean(xs), s = 0; for (double x : xs) s += (x - m) * (x - m); return s / (xs.size() - 1); }
    static String pct(double x, boolean sign) { return (sign && x >= 0 ? "+" : "") + Math.round(x * 100) + "%"; }

    static double[] effect(List<Task> tasks) {
        // Change in time with AI (negative = faster), from the means of log times, with a 95% interval.
        List<Double> a = new ArrayList<>(), b = new ArrayList<>();
        for (Task t : tasks) (t.condition().equals("ai") ? a : b).add(Math.log(t.minutes()));
        double diff = mean(a) - mean(b);
        double se = Math.sqrt(variance(a) / a.size() + variance(b) / b.size());
        return new double[]{Math.exp(diff) - 1, Math.exp(diff - 1.96 * se) - 1, Math.exp(diff + 1.96 * se) - 1};
    }

    static void report(String label, List<Task> tasks) {
        double[] e = effect(tasks);
        System.out.println(label + ": tasks took " + pct(Math.abs(e[0]), false) + " " + (e[0] > 0 ? "longer" : "shorter") + " with AI (95% interval "
            + pct(e[1], true) + " to " + pct(e[2], true) + "), n=" + tasks.size());
    }

    public static void main(String[] args) {
        report("all tasks", TASKS);
        report("deep work only", TASKS.stream().filter(t -> t.kind().equals("deep")).toList());
        report("boilerplate only", TASKS.stream().filter(t -> t.kind().equals("boilerplate")).toList());
        System.out.println("developers believed: " + pct(Math.abs(BELIEVED), false) + " faster");
    }
}

all tasks: tasks took 12% shorter with AI (95% interval -56% to +76%), n=14
deep work only: tasks took 21% longer with AI (95% interval -32% to +118%), n=8
boilerplate only: tasks took 43% shorter with AI (95% interval -61% to -16%), n=6
developers believed: 20% faster

5. Try it: design a trial

Choose the mix of tasks, the number of tasks and how many AI-friendly tasks people refuse to do without AI. The dashed line is the true effect for your task mix; the bar is what a trial would estimate, with its interval. The effects per task type are illustrative.

6. What the 2026 update teaches

By late 2025, METR found it harder to run the experiment at all. Developers increasingly declined to work without AI; 30% to 50% said they had skipped tasks they didn't want to do without it; pay for participants had dropped from $150 to $50 an hour; and time tracking broke down when developers ran agents while working on something else. METR concluded the central estimate is "likely a bad proxy for the real productivity impact", probably an underestimate, and is changing its design. Even careful measurement gets harder as tools change how people work.

7. Lessons

• Perception isn't data. Developers who were 19% slower believed they were 20% faster. Surveys of how productive people feel measure something else (Module 17).
• The task decides the answer. A fresh, well-defined task and real issues in a mature codebase you know well are different experiments.
• Small samples give wide intervals. Report the interval, not just the headline number.
• Watch who drops out. When people refuse the no-AI condition, the tasks left aren't representative.
• Results expire. Tools change fast; date every number and re-measure (Module 21).

8. Your turn

1. Pick the outcome (10 min). What would "more productive" mean for your team: cycle time, defects, review time, issues closed?
2. Design a fair comparison (10 min). How would you randomise which tasks use AI, and how many tasks would you need?
3. Plan for dropout (10 min). What will you do when people don't want to work without AI?

9. Check yourself

10. Sources and further reading