Amazon: AI-assisted upgrades

Every large company has thousands of applications on old language versions, because upgrading them is tedious and nobody's top priority. In 2024 Amazon reported upgrading tens of thousands of its own production applications from Java 8 or 11 to Java 17 with an AI agent, and estimated the savings in thousands of developer-years. The numbers are Amazon's own, from a company that sells the tool. The method behind them is the useful part, and it works with any coding agent.

1. The situation

Language upgrades are the most common legacy work there is. Old Java versions stop getting fixes, libraries drop support, and every year behind makes the next jump bigger (C14 GitHub shows the same with a framework). Amazon's CEO described updating foundational software as one of the most tedious but critical jobs a software team has, and said an upgrade to Java 17 typically took "50 developer-days".

2. What mattered

• Volume: tens of thousands of applications, each a small project nobody wants.
• Safety: each one is a production system; an upgrade that compiles but behaves differently is worse than none.
• Cost: newer Java versions ran with fewer servers, according to Amazon's estimate.
• People's time: engineers on upgrades aren't building anything new.

3. The options

4. The decision

Decision: Upgrade Java with recipes plus an agent, checked by builds and tests

Context: Tens of thousands of applications on Java 8 or 11; ~50 developer-
  days per manual upgrade (Amazon's estimate).
Decision (as documented for Amazon Q Developer):
  - Build the project as it is, then make a transformation plan: new
    dependency versions, major code changes, replacements for deprecated APIs.
  - Apply changes with deterministic tooling (the 2023 launch post says it
    "uses parts of OpenRewrite") and generative AI for what rules can't do.
  - Re-build and run the existing unit tests after each step, and fix
    errors iteratively; make the minimal changes the new JDK needs, with
    library upgrades as a separate transformation.
  - Show a summary and a diff; nothing changes until a person accepts it.
Consequences:
  + Upgrades in hours instead of weeks, by Amazon's estimate
  + Every change reviewed as a diff, every result checked by tests
  - Results are only as trustworthy as each application's tests
  - Some transformations are partial and need a person to finish

5. How it works

The upgrade loop as Amazon documents it, and how to build the same loop with any agent.

6. The technique in code

A tiny version of the loop. Deterministic recipes rewrite what's safe (boxing constructors deprecated since Java 9; Runtime.traceInstructions, removed in Java 17). Patterns no rule covers go to the agent, represented here by a fixed patch; whatever the agent can't fix goes to a person. In a real pipeline the build and tests then decide whether each patch stays. The program was run in all three languages; each prints the output below it.

import re

# Deterministic recipes: safe, mechanical rewrites (the kind OpenRewrite recipes make).
RECIPES = [
    (r"new (Integer|Long|Double|Boolean)\(", r"\1.valueOf("),        # boxing constructors, deprecated since Java 9
    (r"Runtime\.getRuntime\(\)\.traceInstructions\(\w+\);\s*", ""),  # removed in Java 17
]
# Patterns no safe rule covers: left for the agent, then checked by the build and tests.
FOR_AGENT = [r"sun\.misc\.BASE64Encoder", r"\.finalize\(\)"]
# What the agent proposes for each flagged pattern (a stand-in for a model call).
AGENT_FIXES = {r"sun\.misc\.BASE64Encoder": ("new sun.misc.BASE64Encoder().encode(", "java.util.Base64.getEncoder().encodeToString(")}


def upgrade(files):
    report = {"recipe": 0, "agent": 0, "human": 0}
    for name, code in files.items():
        for pattern, replacement in RECIPES:
            code, n = re.subn(pattern, replacement, code)
            report["recipe"] += n
        for pattern in FOR_AGENT:
            if re.search(pattern, code):
                fix = AGENT_FIXES.get(pattern)
                if fix and fix[0] in code:
                    code = code.replace(*fix)       # the agent's patch; the build and tests decide if it stays
                    report["agent"] += 1
                else:
                    report["human"] += 1
                    print(f"  needs a person: {name} ({pattern.replace(chr(92), '')})")
        files[name] = code
    return report


FILES = {
    "Prices.java": "Integer qty = new Integer(3); Double rate = new Double(0.2);",
    "Debug.java": "Runtime.getRuntime().traceInstructions(true); log(\"start\");",
    "Token.java": "String t = new sun.misc.BASE64Encoder().encode(bytes);",
    "Pool.java": "conn.finalize(); Long id = new Long(7);",
}
r = upgrade(FILES)
print(f"recipes: {r['recipe']} rewrites | agent: {r['agent']} fixes to verify | people: {r['human']} to review")
for name, code in FILES.items():
    print(f"  {name}: {code}")

// Deterministic recipes: safe, mechanical rewrites (the kind OpenRewrite recipes make).
const RECIPES: [RegExp, string][] = [
  [/new (Integer|Long|Double|Boolean)\(/g, "$1.valueOf("],          // boxing constructors, deprecated since Java 9
  [/Runtime\.getRuntime\(\)\.traceInstructions\(\w+\);\s*/g, ""],   // removed in Java 17
];
// Patterns no safe rule covers: left for the agent, then checked by the build and tests.
const FOR_AGENT = ["sun\\.misc\\.BASE64Encoder", "\\.finalize\\(\\)"];
// What the agent proposes for each flagged pattern (a stand-in for a model call).
const AGENT_FIXES: Record<string, [string, string]> = { "sun\\.misc\\.BASE64Encoder": ["new sun.misc.BASE64Encoder().encode(", "java.util.Base64.getEncoder().encodeToString("] };

function upgrade(files: Map<string, string>) {
  const report = { recipe: 0, agent: 0, human: 0 };
  for (const [name, original] of files) {
    let code = original;
    for (const [pattern, replacement] of RECIPES) {
      report.recipe += (code.match(pattern) ?? []).length;
      code = code.replace(pattern, replacement);
    }
    for (const pattern of FOR_AGENT) {
      if (new RegExp(pattern).test(code)) {
        const fix = AGENT_FIXES[pattern];
        if (fix && code.includes(fix[0])) {
          code = code.split(fix[0]).join(fix[1]);   // the agent's patch; the build and tests decide if it stays
          report.agent++;
        } else {
          report.human++;
          console.log(`  needs a person: ${name} (${pattern.replaceAll("\\", "")})`);
        }
      }
    }
    files.set(name, code);
  }
  return report;
}

const FILES = new Map<string, string>([
  ["Prices.java", "Integer qty = new Integer(3); Double rate = new Double(0.2);"],
  ["Debug.java", "Runtime.getRuntime().traceInstructions(true); log(\"start\");"],
  ["Token.java", "String t = new sun.misc.BASE64Encoder().encode(bytes);"],
  ["Pool.java", "conn.finalize(); Long id = new Long(7);"],
]);
const r = upgrade(FILES);
console.log(`recipes: ${r.recipe} rewrites | agent: ${r.agent} fixes to verify | people: ${r.human} to review`);
for (const [name, code] of FILES) console.log(`  ${name}: ${code}`);

import java.util.*;
import java.util.regex.*;

public class Main {
    // Deterministic recipes: safe, mechanical rewrites (the kind OpenRewrite recipes make).
    static final String[][] RECIPES = {
        {"new (Integer|Long|Double|Boolean)\\(", "$1.valueOf("},          // boxing constructors, deprecated since Java 9
        {"Runtime\\.getRuntime\\(\\)\\.traceInstructions\\(\\w+\\);\\s*", ""},   // removed in Java 17
    };
    // Patterns no safe rule covers: left for the agent, then checked by the build and tests.
    static final String[] FOR_AGENT = {"sun\\.misc\\.BASE64Encoder", "\\.finalize\\(\\)"};
    // What the agent proposes for each flagged pattern (a stand-in for a model call).
    static final Map<String, String[]> AGENT_FIXES = Map.of("sun\\.misc\\.BASE64Encoder",
        new String[]{"new sun.misc.BASE64Encoder().encode(", "java.util.Base64.getEncoder().encodeToString("});

    static int[] upgrade(Map<String, String> files) {
        int recipe = 0, agent = 0, human = 0;
        for (Map.Entry<String, String> e : files.entrySet()) {
            String code = e.getValue();
            for (String[] r : RECIPES) {
                Matcher m = Pattern.compile(r[0]).matcher(code);
                while (m.find()) recipe++;
                code = code.replaceAll(r[0], r[1]);
            }
            for (String pattern : FOR_AGENT) {
                if (Pattern.compile(pattern).matcher(code).find()) {
                    String[] fix = AGENT_FIXES.get(pattern);
                    if (fix != null && code.contains(fix[0])) {
                        code = code.replace(fix[0], fix[1]);   // the agent's patch; the build and tests decide if it stays
                        agent++;
                    } else {
                        human++;
                        System.out.println("  needs a person: " + e.getKey() + " (" + pattern.replace("\\", "") + ")");
                    }
                }
            }
            e.setValue(code);
        }
        return new int[]{recipe, agent, human};
    }

    public static void main(String[] args) {
        Map<String, String> files = new LinkedHashMap<>();
        files.put("Prices.java", "Integer qty = new Integer(3); Double rate = new Double(0.2);");
        files.put("Debug.java", "Runtime.getRuntime().traceInstructions(true); log(\"start\");");
        files.put("Token.java", "String t = new sun.misc.BASE64Encoder().encode(bytes);");
        files.put("Pool.java", "conn.finalize(); Long id = new Long(7);");
        int[] r = upgrade(files);
        System.out.println("recipes: " + r[0] + " rewrites | agent: " + r[1] + " fixes to verify | people: " + r[2] + " to review");
        for (Map.Entry<String, String> e : files.entrySet()) System.out.println("  " + e.getKey() + ": " + e.getValue());
    }
}

  needs a person: Pool.java (.finalize())
recipes: 4 rewrites | agent: 1 fixes to verify | people: 1 to review
  Prices.java: Integer qty = Integer.valueOf(3); Double rate = Double.valueOf(0.2);
  Debug.java: log("start");
  Token.java: String t = java.util.Base64.getEncoder().encodeToString(bytes);
  Pool.java: conn.finalize(); Long id = Long.valueOf(7);

The same loop with any agent

1. Recipes first. Run the deterministic migration, for example OpenRewrite's org.openrewrite.java.migrate.UpgradeToJava17 (guide). It updates the build to Java 17, adds the Java EE libraries the JDK no longer bundles, and replaces deprecated APIs where the migration is clear. Check its licence first: the rewrite-migrate-java recipes are published under the Moderne Source Available License.
2. Build and test, and keep the log. The failing compile errors and tests are the agent's task list.
3. Give the agent the log, with rules. Change only what the error needs; never edit or delete a test to make it pass; stop after a set number of attempts. Hooks and CI checks can enforce these (AI 12a).
4. Loop until green or out of attempts. Partial results go to a person with the remaining errors listed.
5. Review the diff, then roll out like any upgrade: a share of servers, compared on errors and performance (C14).

7. The numbers

All of these are Amazon's own estimates, from a company that sells the tool.

• 50 developer-days to "just a few hours": the average time to upgrade an application to Java 17, according to Andy Jassy.
• Tens of thousands of production applications migrated from Java 8 or 11 to Java 17 (AWS blog, August 2024).
• 4,500 developer-years saved, estimated from "the number of Java dependencies we migrated".
• $260 million a year in cost savings, estimated from "the number of hosts we were able to remove".
• 79% of the auto-generated code reviews were shipped "without any additional changes" (Jassy).
• 1,000 applications in 2 days by a team of five, in the 2023 launch post: about 10 minutes each on average.

Try it: where do the upgrades end up?

A portfolio of applications goes through recipes, then the agent loop. Change how much each stage handles and see how much work is left for people. The shares are illustrative, not Amazon's.

8. What happened

By Amazon's account, more than half of its production Java systems were upgraded in under six months, and the work moved from teams' backlogs to a review step. AWS describes its own estimates as "conservative". The documentation is equally clear about the limits: a transformation "might be partially successful", and then "you have to manually update the remaining code". Library upgrades are a separate step after the minimal JDK change.

9. Lessons

• Deterministic first, AI second. Rules do the mechanical part repeatably; the agent handles what no rule covers. Both produce a diff a person reviews.
• Tests are the judge. The loop is only as safe as each application's tests. Add characterisation tests to untested code before upgrading it (Module 29a).
• Constrain the agent. Minimal changes, no edits to tests, a limit on attempts; enforce them with hooks and CI, not just instructions (AI 12a).
• Read vendor numbers as claims. Ask how they were estimated: here, from dependencies migrated and hosts removed. Measure your own pilot before you plan.
• Upgrades are now cheap enough to do often. If an upgrade costs hours, falling years behind is a choice (Module 30a).

10. Your turn

1. Count the backlog (5 min). How many of your services are more than one major version behind on their language or framework?
2. Pick a pilot (10 min). One service with good tests. Run a recipe, then an agent on what's left. Record the time and the diff size.
3. Write the agent's rules (15 min). What may it change, what may it never touch, and when must it stop and ask?

11. Check yourself

12. Sources and further reading