Design an AI code assistant

Two products share one name. Inline completion suggests the rest of a line as a developer types, and is useless if it arrives after they've typed it themselves. The chat and agent mode reads the codebase, edits several files and runs the tests, and is dangerous if it runs the wrong command. Both depend on picking the right few thousand tokens of context out of millions of lines of code.

1. The prompt

"Design an AI coding assistant that works inside the editor: inline completions as you type, and a chat that can answer questions about the codebase, edit files and run tests. It's a product used by many companies."

What makes it hard:

• A latency budget of a few hundred milliseconds for completions, including a network round trip and a model.
• Context selection. The model sees a few thousand tokens; the repository has millions of lines.
• Trust. Customers' code is their most sensitive asset, and an agent that runs commands can damage a machine or leak secrets.

2. Frame the problem

3. What it must do

1. Suggest completions at the cursor while typing, using the surrounding code and relevant code elsewhere in the project.
2. Answer questions and make edits through chat, grounded in the codebase, shown as diffs the developer accepts.
3. Run an agent loop: plan, read, edit, run tests, repeat, with the developer approving risky steps.

Not now: code review bots on pull requests, autonomous agents in the cloud working on tickets, training custom models per company.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Completion request: file path, language, prefix and suffix around the cursor, plus context snippets.
• Suggestion and its outcome: shown, accepted, partially accepted, or dismissed, and whether the code survived.
• Repository index: code chunks (functions, classes) with symbols and embeddings, per repository and commit.
• Agent session: a goal, a sequence of tool calls (read, search, edit, run) and their approvals.
• Policy: per company: retention, indexing, allowed models, which commands auto-run.

completion_req   req_id, user_id, org_id, lang, path, prefix (tail), suffix (head), snippets[], created_at   (not stored by default)
suggestion_event req_id, shown_ms, accepted (none | partial | full), chars_kept_after_5min                   (telemetry, no code)
index_chunk      org_id, repo_id, commit, path, symbol, kind (function | class | …), embedding, text        (opt-in, encrypted)
agent_step       session_id, n, tool (read | search | edit | run), args, approval (auto | user | denied), result_summary
org_policy       org_id, retention, indexing_enabled, allowed_models[], auto_run_commands[]

7. The contract

POST /v1/completions        {lang, path, prefix, suffix, snippets[]}         → stream of tokens; client may cancel any time
POST /v1/completions/events {req_id, outcome, kept_chars}                    → 204      (telemetry, batched)
POST /v1/chat               {messages, context: {files, selection, index_hits}} → stream: text · tool_call · diff
Tools run in the editor (client side), never on our servers:
read_file(path) · search(query) · edit(path, diff) · run(command)           each checked against the org policy

The editor extension does most of the work that touches code: it gathers context, runs tools, and shows diffs. The server sees only what each request needs.

8. First working version

An editor extension, a completion service with a small model on GPUs in a few regions, and a chat service that calls a large hosted model, with tools executed by the extension.

8.1 A completion

After a short pause in typing, the extension sends the code before and after the cursor (the model is trained for "fill in the middle") plus a few snippets from open files. The model streams a suggestion; the editor shows it as grey "ghost text". Any keystroke cancels the request.

8.2 An agent task

The chat sends the goal and context to a large model. The model asks for tools; the extension runs them locally, subject to policy and approvals, and sends results back, until the tests pass or a step budget is reached. Edits appear as diffs to accept.

9. Make it hold up

9.1 The completion latency budget completions under 200 ms at p50

Also show cached suggestions instantly: if the developer types exactly what the previous suggestion predicted, keep showing the rest of it without a new request.

9.2 Choosing the context useful

• Around the cursor: the prefix and suffix, trimmed to a token budget, nearest lines first.
• From open and recently edited files: snippets most similar to the code near the cursor (a cheap token-overlap score is enough here).
• From the language's structure: signatures of functions and types used nearby, found through the editor's language server or a parser such as tree-sitter.
• A budget with priorities: when everything doesn't fit, keep the cursor neighbourhood first, then definitions, then similar snippets.

For chat, retrieval reaches further: search the repository index (9.3) and let the model ask for more with tools.

9.3 Indexing a repository useful · private

Indexing can run locally (small repos, maximum privacy) or on our servers for large monorepos, encrypted per company, only when the company opts in.

9.4 A safe agent loop safe agent

• Tools, not a shell: read, search, edit (as a diff) and run, each checked against policy.
• Approval levels: reads run freely; edits show diffs; commands run only if on the company's allowlist (tests, linters) or after the developer approves. Destructive commands and network access need explicit approval.
• Sandbox when possible: run in a container or dev environment, not on the bare laptop.
• Untrusted input: files, READMEs, issue text and web pages can contain instructions aimed at the model (prompt injection). Treat tool results as data, keep approvals in code, and never put secrets into context.
• Budgets: a maximum number of steps, tokens and wall-clock time per task, and a clear stop when tests keep failing.

9.5 Privacy and tenants private

By default prompts and suggestions are processed in memory and not stored; telemetry records outcomes, not code. Companies can choose retention, regions and which models may see their code. Indexes and caches are keyed by company and encrypted with per-company keys; prefix caches never cross company boundaries. A filter blocks suggestions that match public code verbatim if the company requires it, and a secret scanner strips keys from prompts before they leave the machine.

9.6 Serving and cost completions at 100 K a second

9.7 Measuring quality useful

• Online: acceptance rate, how much accepted code is still there after 30 s and 5 minutes, latency percentiles, and A/B tests for every model or context change. A study of Copilot users found acceptance rate was the best predictor of how productive developers felt.
• Offline: completion benchmarks built by hiding lines in real repositories; for agents, tasks with tests that must pass (SWE-bench-style), run before every model or prompt change.
• Guardrails: rates of approval denials, reverted agent edits, and security filter hits.

9.8 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• A small model for completions: less clever, but fast enough to be used.
• Tools run in the editor, so code stays local and permissions are the user's own.
• Approvals that slow the agent down, for safety.
• Two serving paths instead of one, to protect completion latency.

What we'd watch: completion p50/p95 by region · acceptance and kept code · cancelled-request share · GPU utilisation · prefix-cache hit rate · agent task success and denied approvals · cost per active developer.

At 10×: more regions, smaller or distilled models, more aggressive caching, and per-company dedicated capacity for the largest customers.

Common mistakes:

• Treating completions and chat as one service.
• A request per keystroke with no cancellation.
• Fixed-size chunks that split functions.
• Letting the agent run arbitrary commands on a laptop.
• Storing customer code "for debugging" without a policy.

13. Try it, check yourself, read more

Completion-latency lab. A developer pauses typing. How long until the grey suggestion appears, and how many requests does it cost? Change the model, the context size, the debounce and the distance to the GPUs. The target is under 300 ms from the pause.

Illustrative model: prefill at 50 K / 15 K / 5 K tokens a second and decode at 200 / 80 / 40 tokens a second for small / medium / large; a first line is ~15 tokens; a prefix-cache hit skips 80% of prefill. Requests a minute assume steady typing with pauses. GPU baseline = small model, 2,000 tokens, cache on, 200 ms debounce.