DoorDash: guardrails and a judge for a RAG bot

DoorDash built a chatbot to answer its couriers', the Dashers', support questions from the company's help articles. Like every retrieval-based bot, it sometimes produced answers that sounded right and weren't. Instead of trusting the model, DoorDash checks every answer before it's sent, with a cheap test first and a stricter one only when needed, and scores its quality with a second model. It's the opposite of the Air Canada story: the checks Air Canada didn't have.

1. The situation

Couriers on the road need quick answers: what to do when a customer isn't there, how pay works, what to do about a damaged order. A retrieval-augmented chatbot finds the relevant help articles and has a model answer from them (Module 15). DoorDash's engineers describe the catch: their RAG chatbot "has in some cases generated responses that diverged from the intended context. Although these responses sound natural and legitimate, users may not realize that they are inaccurate."

2. What DoorDash built

Why two layers? A similarity comparison is cheap and fast, so it can run on every answer. A model-based evaluator is slower and costs tokens, so it runs as a backup. That's the design pattern this page builds; the details of DoorDash's own thresholds and models aren't in the quoted text.

3. How a two-layer guardrail works

4. The control in code

A two-layer guardrail. Layer 1 compares the answer with the retrieved article (here, the share of words in common) and decides at once when it's clearly similar or clearly not. Only answers in the uncertain middle go to layer 2, a stricter check (here, a rule standing in for an LLM evaluator: every number in the answer must appear in the article). The third draft shows why both are needed: it's similar enough to pass a single loose check, but says 15 minutes where the article says 5. The program was run in all three languages; each prints the output below it.

import re

# An invented delivery app's help article, retrieved for a courier's question.
ARTICLE = "If a customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery."
STOP = {"a", "the", "is", "at", "for", "you", "are", "then", "if", "not", "and", "to", "your", "will", "be", "it"}
LOW, HIGH = 0.25, 0.60     # below LOW: block; above HIGH: send; in between: ask the stricter check


def words(text):
    return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}


def similarity(answer, article):
    """Layer 1, cheap: share of words the answer and the article have in common (Jaccard)."""
    a, b = words(answer), words(article)
    return len(a & b) / len(a | b)


def strict_check(answer, article):
    """Layer 2, stricter and slower: a stand-in for an LLM evaluator. Every number in the answer must be in the article."""
    return set(re.findall(r"\d+", answer)) <= set(re.findall(r"\d+", article))


def guard(answer):
    s = similarity(answer, ARTICLE)
    if s >= HIGH:
        return s, "send", "layer 1"
    if s < LOW:
        return s, "hand to a person", "layer 1"
    return s, ("send" if strict_check(answer, ARTICLE) else "regenerate or hand to a person"), "layer 2"


DRAFTS = [
    "If the customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.",
    "Wait 5 minutes at the address, then contact support; you are still paid.",
    "Wait 15 minutes at the address, then contact support; you are still paid.",
    "Leave the order at the door and mark it delivered.",
]
calls = 0
for d in DRAFTS:
    s, decision, layer = guard(d)
    calls += layer == "layer 2"
    print(f"{s:.2f}  {layer}  {decision:<30} {d}")
print(f"{len(DRAFTS)} answers checked; {calls} needed the slower check")

// An invented delivery app's help article, retrieved for a courier's question.
const ARTICLE = "If a customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.";
const STOP = new Set(["a", "the", "is", "at", "for", "you", "are", "then", "if", "not", "and", "to", "your", "will", "be", "it"]);
const LOW = 0.25, HIGH = 0.6;   // below LOW: block; above HIGH: send; in between: ask the stricter check

const words = (text: string) => new Set((text.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter((w) => !STOP.has(w)));

function similarity(answer: string, article: string): number {
  // Layer 1, cheap: share of words the answer and the article have in common (Jaccard).
  const a = words(answer), b = words(article);
  const inter = [...a].filter((w) => b.has(w)).length;
  return inter / new Set([...a, ...b]).size;
}

function strictCheck(answer: string, article: string): boolean {
  // Layer 2, stricter and slower: a stand-in for an LLM evaluator. Every number in the answer must be in the article.
  const known = new Set(article.match(/\d+/g) ?? []);
  return (answer.match(/\d+/g) ?? []).every((n) => known.has(n));
}

function guard(answer: string): [number, string, string] {
  const s = similarity(answer, ARTICLE);
  if (s >= HIGH) return [s, "send", "layer 1"];
  if (s < LOW) return [s, "hand to a person", "layer 1"];
  return [s, strictCheck(answer, ARTICLE) ? "send" : "regenerate or hand to a person", "layer 2"];
}

const DRAFTS = [
  "If the customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.",
  "Wait 5 minutes at the address, then contact support; you are still paid.",
  "Wait 15 minutes at the address, then contact support; you are still paid.",
  "Leave the order at the door and mark it delivered.",
];
let calls = 0;
for (const d of DRAFTS) {
  const [s, decision, layer] = guard(d);
  if (layer === "layer 2") calls++;
  console.log(`${s.toFixed(2)}  ${layer}  ${decision.padEnd(30)} ${d}`);
}
console.log(`${DRAFTS.length} answers checked; ${calls} needed the slower check`);

import java.util.*;
import java.util.regex.*;

public class Main {
    // An invented delivery app's help article, retrieved for a courier's question.
    static final String ARTICLE = "If a customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.";
    static final Set<String> STOP = Set.of("a", "the", "is", "at", "for", "you", "are", "then", "if", "not", "and", "to", "your", "will", "be", "it");
    static final double LOW = 0.25, HIGH = 0.60;   // below LOW: block; above HIGH: send; in between: ask the stricter check

    static Set<String> matches(String text, String regex) {
        Set<String> out = new LinkedHashSet<>();
        Matcher m = Pattern.compile(regex).matcher(text);
        while (m.find()) out.add(m.group());
        return out;
    }

    static Set<String> words(String text) {
        Set<String> out = new LinkedHashSet<>();
        for (String w : matches(text.toLowerCase(), "[a-z0-9]+")) if (!STOP.contains(w)) out.add(w);
        return out;
    }

    static double similarity(String answer, String article) {
        // Layer 1, cheap: share of words the answer and the article have in common (Jaccard).
        Set<String> a = words(answer), b = words(article), inter = new HashSet<>(a), union = new HashSet<>(a);
        inter.retainAll(b); union.addAll(b);
        return (double) inter.size() / union.size();
    }

    static boolean strictCheck(String answer, String article) {
        // Layer 2, stricter and slower: a stand-in for an LLM evaluator. Every number in the answer must be in the article.
        return matches(article, "\\d+").containsAll(matches(answer, "\\d+"));
    }

    public static void main(String[] args) {
        List<String> drafts = List.of(
            "If the customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.",
            "Wait 5 minutes at the address, then contact support; you are still paid.",
            "Wait 15 minutes at the address, then contact support; you are still paid.",
            "Leave the order at the door and mark it delivered.");
        int calls = 0;
        for (String d : drafts) {
            double s = similarity(d, ARTICLE);
            String decision, layer;
            if (s >= HIGH) { decision = "send"; layer = "layer 1"; }
            else if (s < LOW) { decision = "hand to a person"; layer = "layer 1"; }
            else { decision = strictCheck(d, ARTICLE) ? "send" : "regenerate or hand to a person"; layer = "layer 2"; calls++; }
            System.out.println(String.format(Locale.US, "%.2f  %s  %-30s %s", s, layer, decision, d));
        }
        System.out.println(drafts.size() + " answers checked; " + calls + " needed the slower check");
    }
}

1.00  layer 1  send                           If the customer is not at the address, wait 5 minutes, then contact support. You are paid for the delivery.
0.70  layer 1  send                           Wait 5 minutes at the address, then contact support; you are still paid.
0.55  layer 2  regenerate or hand to a person Wait 15 minutes at the address, then contact support; you are still paid.
0.00  layer 1  hand to a person               Leave the order at the door and mark it delivered.
4 answers checked; 1 needed the slower check

5. Try it: tune the thresholds

Twenty invented answers, some correct and some not, each with a similarity score. Set where layer 1 sends and blocks; everything between goes to the slower check. Watch wrong answers caught, good answers wrongly blocked, and how many expensive checks you pay for.

6. What it shows

The figures are DoorDash's own: 90% fewer hallucinations and 99% fewer potentially severe compliance issues, by its account. Without the full post, how they were measured isn't something this page can report. What the design shows is the point: a model's answer is a draft until something independent has checked it against the source, and quality needs its own measurement, which is what the judge is for (Module 17).

7. Lessons

• Every answer is a draft. Check it against the retrieved source before it reaches a user (C03 shows what happens without that).
• Layer the checks by cost. A cheap test on everything; an expensive one only where the cheap one is unsure.
• Have a safe fallback. When a check fails, regenerate or hand over to a person, rather than sending the answer anyway.
• Measure quality continuously. A judge scoring real answers tells you what to fix: retrieval, prompts or the articles themselves (Module 17).
• Guardrails are part of security too. The same output checks catch policy and compliance problems (Module 18).

8. Your turn

1. Pick the cheap check (10 min). For an assistant you know, what fast test could flag an answer that doesn't match its source?
2. Pick the strict check (10 min). What would a slower evaluator verify: numbers, policy terms, refusal rules?
3. Choose the fallback (10 min). Regenerate, refuse, or a person? For which kinds of question?

9. Check yourself

10. Sources and further reading