AI Latency: The Metrics That Make Products Feel Faster

Start with the user-visible journey

A model can return its first token quickly while the product still feels slow because it is waiting for retrieval, authentication, a tool, a database or a final validation step. Conversely, a longer task may feel responsive when the interface shows useful progress and allows the user to understand what is happening. Map the journey from the user's action to a usable result. Include browser or app processing, network time, request queues, policy checks, retrieval, model inference, tool execution and final rendering. Record the moments the user can observe rather than treating the model API duration as the only metric. This produces a latency budget that engineering, product and design can jointly improve.

Measure time to first token and time between tokens

For a streaming language model, time to first token (TTFT) measures how long a user waits before any generated output appears. Time between tokens measures how quickly subsequent output arrives. Total completion time records when the response is finished. These measures answer different questions. A low TTFT can make the application feel responsive, but slow generation may still delay the final answer. A fast completion time may not help if a tool call blocks before the model can start responding. Capture the definition and start/stop points used by your platform, because one vendor’s metric may not be directly comparable to another’s. Report both API-level and end-to-end measurements.

Use percentiles and segment the request mix

An average can look healthy while a smaller group of users waits far too long. Track median or p50 latency and a tail metric such as p95, and consider p99 for workflows where rare delays matter. Segment results by task type, input length, output length, region, model, retrieval path, tool use and customer plan. A single combined chart can hide a slow document-processing task inside a fast short-answer workload. Compare like with like: requests with long contexts should not be treated as directly equivalent to tiny prompts without recording that difference. Add sample size and the measurement window to dashboards so decision-makers do not overreact to a handful of requests or a brief traffic spike.

Find the bottleneck before changing models

Latency often comes from several stages rather than a single slow model. Instrument each stage with timestamps and correlation identifiers, then compare the slowest request traces. If retrieval takes most of the time, a faster model may barely change the user experience. If the model waits on a serial tool call, improve that dependency or run independent operations in parallel where doing so is safe. If a large prompt dominates preprocessing, remove irrelevant context and avoid retrieving documents that cannot affect the answer. If the UI waits to render until every optional detail is ready, progressive rendering may help. Optimise the measured constraint instead of choosing the most fashionable technique or switching providers without a clear hypothesis.

Reduce unnecessary work and long answers

The simplest improvement is sometimes not to call an LLM at all. Use ordinary application code for exact lookup, validation, routing and other deterministic tasks when it meets the requirement. In multi-step chains, remove redundant model calls and combine steps only if evaluation shows the combined prompt remains reliable. Ask for the level of detail the product needs, and set sensible output limits when long answers do not help the user. Large input contexts can add processing time and cost, so retrieve only relevant material and avoid resending unchanged content unnecessarily. OpenAI’s latency guidance also discusses reducing output tokens, parallelising independent work and showing progress. Each change should be measured against correctness and completeness, not just stopwatch time.

Use model choice, caching and parallelism carefully

A smaller or faster model may suit bounded classification, extraction or short responses, but it may perform worse on a complex reasoning task and cause retries or human correction that erase the apparent gain. Evaluate on the target request mix before routing traffic. Cache only when the input, user permissions, freshness requirements and output semantics make reuse safe. A cache that serves another user’s private result is a security failure, not a performance improvement. Parallelise independent retrieval or service calls, but avoid racing writes or actions where order matters. Batching can improve throughput while making individual requests wait longer. Match each technique to the product objective: a real-time assistant values responsiveness, while a background summarisation job may care more about throughput and cost.

Make streaming communicate progress, not certainty

Streaming can reduce the blank-screen feeling because useful output appears before the full response is complete. It also creates design choices. If the application displays an unfinished answer, users may act on a statement that changes in later tokens. For high-impact or transactional steps, wait for required validation and explicit confirmation before execution. Show clear states for waiting on tools, searching sources, generating, and verifying when those states are real. Do not invent a progress percentage that the system cannot support. Where a response must be structured or cited, consider what can be shown safely before final validation. Design cancellation and recovery too: when the user stops generation, the product should make clear whether any underlying tool action has already occurred.

Set latency targets that reflect the task

A single latency target for every AI feature is rarely useful. A short autocomplete suggestion, a research answer and a document analysis task have different expectations and output sizes. Define service-level objectives per workflow, including the relevant start and end points, target percentile, timeout and acceptable failure rate. Set a budget for each subsystem so one dependency does not consume the entire allowance. Connect the target to user research or task evidence: what wait is acceptable, what information can be shown while waiting, and when does delay cause abandonment? Use a holdout or staged rollout to test whether an optimisation really improves task completion. If users get faster but need more edits or retries, the product has moved time rather than removed it.

Monitor quality and total cost beside speed

Latency dashboards should sit beside quality metrics such as task success, factual errors, tool-call success, corrections, abandoned sessions and human review time. A shortcut that makes the model return faster but increases mistakes can be a net loss. Include cost per accepted outcome because retries, extra context, longer outputs and human rework all have a price. Alert on changes in tail latency and failure rate, not just traffic averages. Add trace sampling and retain only the data justified by the debugging purpose, especially when prompts or retrieved documents contain personal information. Use a consistent definition across staging and production, and rerun benchmark scenarios after provider or application changes. Performance is an ongoing product property, not a one-time optimisation task.

A practical measurement plan

For a representative workload, capture end-to-end duration, TTFT where applicable, token generation rate, p50, p95, time spent in retrieval and tools, timeout rate, task quality and cost. Start with the slowest high-volume or high-impact workflows. Form one hypothesis at a time, for example reducing an unnecessary tool call or trimming irrelevant context, and test it on the same evaluation set and traffic conditions. Keep the change only if it improves the user outcome without creating an unacceptable quality, privacy or reliability trade-off. Document the before-and-after results, the traffic mix and any caveats. This creates an optimisation loop that is repeatable and understandable instead of a sequence of unconnected model tweaks.