Cost, latency and observability

An AI feature that works in a demo can still fail in production: too slow, too expensive at scale, or quietly looping on some inputs. You can't fix what you can't see, so the first step is recording every model and tool call. Then the savings are usually easy to find.

1. The idea

Three things to get right:

1. Observe. Record a trace for every request: each model call and tool call, with the model, tokens, cost, time, errors and (where allowed) the content. OpenTelemetry has standard GenAI fields for this, so any tracing tool can read it.
2. Cut cost with the levers from Module 04, applied automatically: route easy requests to a smaller model, cache the stable start of prompts, batch anything that can wait, and ask for shorter outputs.
3. Cut latency where users feel it: stream the answer so the first words appear quickly, run independent tool calls in parallel, use a faster model or less reasoning effort for simple steps.

And set budgets (turn caps, token limits, per-user and per-feature spend limits) so one bad input can't burn a month's budget overnight.

2. How it works

One model-call span, with the standard OpenTelemetry GenAI attribute names (simplified):

{
  "name": "chat claude-sonnet-5-5",
  "duration_ms": 2410,
  "attributes": {
    "gen_ai.request.model": "claude-sonnet-5-5",
    "gen_ai.usage.input_tokens": 3120,
    "gen_ai.usage.output_tokens": 412,
    "gen_ai.response.finish_reasons": ["end_turn"],
    "app.feature": "support-answer",
    "app.cost_usd": 0.0104
  }
}

The gen_ai.* names come from the OpenTelemetry GenAI semantic conventions, which are still evolving. The app.* fields are your own: tag every span with the feature and customer so you can slice cost by them.

3. Try it yourself

Trim the bill

A support feature handles 20,000 requests a day on Claude Sonnet 5.5: 3,000 input tokens and 400 output tokens each. Switch on the levers and watch the monthly cost.

Make it feel fast

One agent request: a planning call, two tool calls, then the answer. Change how it runs and watch the total time and the time until the user sees the first words.

4. Why it matters at work

Measure what matters: cost per successful task (not per request), p95 latency and time to first token (not just averages), error and retry rates, and turns per agent run. Track cost per feature and per customer, so a price change is a decision, not a surprise.

5. In your five tools

Coding tools need the same discipline: know what a session and a month cost you. Checked against vendor documentation, October 2026.

6. Remember it

7. Common mistakes

8. Hands-on lab (roadmap week 24)

1. Trace your feature from Modules 14–16: one span per model and tool call with model, tokens, duration and a feature tag. Send it to any OpenTelemetry-compatible tool, or start with a log file.
2. Price it: compute cost per request and per successful task from the token counts and current prices.
3. Add caching: reorder the prompt so the stable part comes first, and confirm cache reads in the API's usage numbers.
4. Try routing: run your eval set on a smaller model. For which cases is it good enough? Estimate the saving.
5. Estimate cost per 1,000 users, and set a budget and alert. Turn on telemetry or check usage in your own coding tool for a week.

9. Self-check quiz

10. Go deeper