Design an AI customer-support assistant

A shopper types "Where's my order, and can I still cancel it?" into the chat window. An assistant answers in seconds from the help centre and the shopper's own orders, cancels the order after they confirm, and hands the conversation to a person, with a summary, when it shouldn't decide alone. The model is the easy part; the design is about grounding, safe actions, handoffs and measuring whether it actually helps.

1. The prompt

"Design an AI assistant for customer support on a large online store. It answers questions in chat using our help centre and the customer's orders, can do simple things like cancelling an order or starting a return, and hands off to human agents when needed."

What makes it hard:

• A confident wrong answer is worse than no answer. Inventing a refund policy costs money and trust.
• Actions are real. Cancelling the wrong order, or someone else's, is a serious incident.
• Success is hard to measure. A closed chat can mean "solved" or "gave up".

2. Frame the problem

3. What it must do

1. Answer questions from the help centre, citing the articles used, and say so when it doesn't know.
2. Look up the customer's orders and take allowed actions, always after the customer confirms.
3. Hand off to a human with a summary when it can't or shouldn't help, and learn from how humans resolve those cases.

Not now: voice calls, proactive outreach ("your parcel is late"), sales recommendations.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Conversation and message; the conversation belongs to one signed-in customer.
• Article and its chunks, each with an embedding and an effective date.
• Action: a proposed tool call (cancel order 123), with a status: proposed, confirmed, done, failed.
• Handoff: a ticket for a human with the summary and transcript.
• Eval case: a question with what a correct answer must and must not say.

conversations  conversation_id, customer_id, channel, language, status (bot | waiting_human | human | closed), created_at
messages       conversation_id, seq, role (customer | assistant | agent | tool), text, citations[], created_at
chunks         chunk_id, article_id, language, heading_path, text, embedding, effective_from, version
actions        action_id, conversation_id, tool, args, status, idempotency_key, confirmed_at, result
handoffs       conversation_id, reason, summary, ticket_id, created_at

7. The contract

POST /v1/conversations                                   → {conversation_id}
POST /v1/conversations/{id}/messages {text}              → stream of events (server-sent events):
       token… · citation {article_id, title} · action_proposed {action_id, summary} · handoff {eta}
POST /v1/conversations/{id}/actions/{action_id}/confirm  → {status, result}       (the customer's button press)
POST /v1/conversations/{id}/handoff {reason}             → {ticket_id}

Tools the model may request (run by our server, never by the model):
search_help(query)            get_orders(limit)            track(order_id)
cancel_order(order_id)        start_return(order_id, items, reason)        (both need confirmation)

Note what's missing: no tool takes a customer_id. The server fills it in from the session, so no wording in a message can make the assistant read someone else's orders.

8. First working version

A conversation service streams to the chat widget. For each message an orchestrator retrieves help-centre chunks, calls the model with the instructions, the chunks, the history and the tool list, runs any tool the model requests (after checks), and streams the answer with citations.

8.1 Answering a policy question

Retrieval runs before the model: hybrid search (keywords plus embeddings) returns the best few chunks, which go into the prompt with an instruction to answer only from them and cite them. The answer streams token by token.

8.2 Taking an action

The model only proposes a cancellation. The orchestrator checks that the order belongs to this customer and is cancellable, shows a confirmation card, and calls the order service only when the customer presses it, with an idempotency key so a double tap cancels once.

9. Make it hold up

9.1 Retrieval and grounded answers grounded

Filter chunks by the customer's language and country (return windows differ by country), and by effective date, so a policy that starts next month isn't quoted today.

9.2 Actions that can't go wrong quietly safe actions

• The server owns authority. Tools are an allowlist; arguments are validated; the customer comes from the session; each tool calls the same order APIs (and rules) the website uses.
• Propose, confirm, execute. Anything that changes state shows a card in plain words ("Cancel order #4821, 2 items, $63.10?") and runs only on the customer's press.
• Idempotency keys on every write, so retries and double taps are harmless (Module 22).
• Limits by policy, in code: refunds over $100, repeated cancellations, or flagged accounts route to a human whatever the model says.
• Audit log of every proposal, confirmation and result, linked to the conversation.

9.3 Prompt injection and data leaks security

9.4 Handing off to a person never a dead end

Hand off when the customer asks, when retrieval confidence is low, after two failed attempts, when sentiment turns angry, for excluded topics (legal threats, safety, bereavement), or when a policy limit is hit. The agent receives a short model-written summary, the transcript, the articles cited and any actions taken, inside the ticketing tool, so the customer never repeats themselves. How agents resolve these cases becomes new eval cases and, where the answer was missing, new help articles.

9.5 Measuring quality helpful · grounded

9.6 Latency and cost fast

• Stream every answer; show "checking your order…" while a tool runs (Module 19).
• Route by difficulty: a small, fast model classifies intent and handles simple turns ("thanks!"); a stronger model handles policy reasoning.
• Prompt caching for the long fixed instructions and tool definitions cuts cost and time to first token.
• Summarise history beyond a few turns instead of resending everything.
• Fallbacks: a second model provider for outages, and humans if both fail (Module 24).

9.7 Keeping knowledge current grounded

When an article is published, an event re-chunks and re-embeds just that article, and the old chunks are removed in the same step, so search never returns two versions. Chunks carry an effective date for scheduled policy changes. For urgent changes (a recall, a storm delaying deliveries), a short "notice" article with a high retrieval boost goes live in minutes. Eval cases that mention a changed policy are updated in the same change.

9.8 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Answer less often, but correctly: low-confidence questions go to people.
• Confirmation steps for every write, at the cost of one extra tap.
• Retrieval instead of fine-tuning, so facts stay current and citable.
• Quality gates on every change, which slows prompt tweaks.

What we'd watch: confirmed resolution rate · satisfaction (bot vs human) · handoff reasons · eval pass rate · time to first token · cost per conversation · tool error rate.

At 10×: more of the same model calls (the API scales), a bigger human team for handoffs, per-language eval sets, and retrieval sharded by language if the knowledge base grows.

Common mistakes:

• Letting the model pass customer IDs to tools.
• Actions without confirmation or idempotency.
• Measuring deflection instead of resolution.
• No path to a human when the AI is unsure or down.
• Shipping prompt changes without an eval run.

13. Try it, check yourself, read more

Answer-or-hand-off lab. Ten real-looking questions, each with the retrieval confidence of its best source and whether that source really answers it. The assistant answers when confidence is at least the threshold, and hands off otherwise. Move the threshold and watch automation trade against wrong answers.

The scores are illustrative. In practice you set the threshold from an eval set, so that the wrong-answer rate stays under the target.