AI and LLM components in an architecture
Large language models arrived in most products as "just another API call". Architecturally they're stranger than that: the answer to the same request can differ every time, quality can't be checked with an equality test, latency is seconds, cost scales with words, and the provider may change the model under you. This module treats an LLM feature as a set of components with characteristics, and shows where the architectural decisions are.
1. The idea
The components that keep showing up around a model call:
| Component | Its job | Architectural reason |
|---|---|---|
| Model gateway | One internal entry point for all model calls: authentication, routing, fallbacks, rate limits, caching, cost tracking, logging | Isolates the volatile provider APIs behind a stable port (Module 07a); one place for policy |
| Prompt and configuration registry | Prompts, model choices and parameters as versioned artifacts, deployed like code | Changing a prompt is a release: it needs review, rollback and an audit trail |
| Retrieval (RAG) | Find relevant documents and put them in the prompt | Grounds answers in your data; it's a data architecture: ingestion, chunking, an index, access control, freshness |
| Guardrails | Checks on input and output: personal data, prompt injection, unsafe content, schema validation | The model is not a trust boundary (Module 28a) |
| Evaluation pipeline | Score outputs against test sets and judges, on every change | Evals are the fitness functions of AI features (Module 26) |
| Tool access | The model asks to call functions; your code decides and executes them with least privilege | An agent with broad credentials is a security architecture problem |
New characteristics to name
- Non-determinism: same input, different output. Tests compare against criteria, not exact strings.
- Quality becomes a measured characteristic with a target (a pass rate on an eval set).
- Cost per request scales with tokens in and out, so prompt size is an architectural decision.
- Latency of seconds, often streamed; time to first token matters more than total time.
- Provider volatility: models are deprecated and change behaviour; portability has a price.
2. How it works
A support assistant answers one question
A customer asks about a refund. Watch the request pass through each component, including a provider outage.
3. Try it yourself
Route, cache, and keep quality
Two models behind a gateway: a small one (cheap and fast, weaker on hard requests) and a large one. Set how many requests are easy, how much traffic the router sends to the small model, and the cache hit rate.
Assumed figures to show the shape, not any vendor's prices: 1 million requests a month, 2,000 tokens each; small model $0.40 and large model $4.00 per million tokens; 0.6 s and 2.0 s typical latency; pass rates on the eval set: small 98% on easy and 60% on hard requests, large 99% and 92%. The router sends easy requests to the small model first.
4. Trade-offs
Quality vs cost and latency
Bigger models are better on hard tasks and cost several times more per token and per second. Routing, caching and shorter prompts move the curve; evals tell you how far you can go.
Portability vs features
A gateway with a common interface makes switching providers cheap, but hides provider-specific features (tool formats, caching options). Abstract the call, not every capability.
Hosted vs self-hosted models
Hosted APIs need no GPUs and improve constantly; self-hosted open-weight models give control over data, cost at high volume and stability, at the price of running GPU infrastructure.
Autonomy vs control
An agent that acts on its own does more; every tool it can call widens the blast radius. Confirmations, narrow tools and audit logs are the architectural brakes.
5. In real projects
Put a gateway in front of every model call from the first feature. It's where you add a second provider, a cache, a budget limit or a new logging rule without touching every service.
Treat prompts and model versions as deployable configuration: versioned, reviewed, rolled out gradually and rolled back, with the eval suite as the release gate.
RAG quality is mostly data quality: what's indexed, how it's chunked, whether permissions follow the documents, and how fresh the index is. Retrieval must apply the user's access rights, or the assistant leaks documents the user couldn't open.
Log the whole trace (prompt version, retrieved documents, model, tokens, latency, guardrail decisions), with personal data handled under the same rules as any other user data.
Know the risks by name: the OWASP Top 10 for LLM applications (prompt injection, sensitive information disclosure, excessive agency and others) is a useful checklist in reviews, and the NIST AI Risk Management Framework for governance.
Say it in the interview
- "How would you add an LLM to this system?" Behind an internal model gateway, with prompts versioned as configuration, retrieval for grounding, guardrails on input and output, and an eval suite as the release gate.
- "How do you control the cost?" Route easy requests to smaller models, cache repeated answers and prompt prefixes, trim context, and track cost per request as a characteristic with a budget.
- "How do you know it works?" An eval set of real cases scored on every change, plus production sampling and user feedback; quality is a measured number with a target.
- Every service calling a provider's SDK directly with its own key.
- "The model will refuse bad requests" as the security plan.
- No way to tell whether a prompt change made things better or worse.
6. Remember it
How this hook works: what "gate it, ground it, guard it, grade it" means
What the slogan means, phrase by phrase
- "Gate it"
- One model gateway for routing, fallbacks, limits, caching and cost.
- "ground it"
- Retrieval brings your data into the prompt, with the user's permissions applied.
- "guard it"
- Checks on what goes in and comes out; tools with least privilege.
- "grade it"
- Evals on every change: quality as a measured characteristic.
Why this picture fits: it's how you'd manage the brilliant, hourly-billed, sometimes-wrong contractor: a front desk, a briefing pack, a supervisor and a review of their work.
The story → the concept
| The contractor | The LLM feature |
|---|---|
| The front desk that books them | The model gateway |
| The briefing pack | Retrieved documents in the prompt |
| Only the keys they need | Least-privilege tools and guardrails |
| Checking work against known answers | The eval suite |
| A cheaper colleague for simple jobs | Routing to a smaller model |
The trap it saves you from
A demo that works on stage and becomes an unmeasured, unbudgeted, unsecured dependency in production.
7. Common mistakes
8. Design exercise (30 minutes)
- Components (10 min). Pick an AI feature in a product you know. Draw the gateway, retrieval, guardrails, evals and tools, and who owns each.
- Characteristics (10 min). Write targets for quality (eval pass rate), latency (time to first token), and cost per request.
- An ADR (10 min). Hosted API or self-hosted model? Write the decision with its drivers and the revisit trigger (Module 27).
9. Self-check quiz
Why put a model gateway in front of LLM calls?
What plays the role of fitness functions for an AI feature?
How do you stop a RAG assistant revealing documents a user can't open?
How should a prompt change be released?
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| Emerging Patterns in Building GenAI Products (martinfowler.com) | 60 min | RAG, guardrails, evals and more, as patterns. |
| OWASP Top 10 for LLM Applications | 30 min | The main risks, for architecture and security reviews. |
| NIST AI Risk Management Framework | reference | Governance language for AI risk. |
| The AI engineering track | course | Modules 14–19 cover model APIs, RAG, agents, evals, security and cost in depth. |
| D23 LLM chat platform and D31 AI support assistant | 90 min each | Worked designs that use these components. |