Designing an LLM latency benchmark that matches what users feel

SSpeed is a feature. A support assistant that answers in two seconds feels helpful; the same answer after twelve seconds feels broken. Yet many teams compare models on speed using a handful of requests from a laptop, or a vendor's published "tokens per second," and are surprised when production feels slower.

LLM latency has several components that matter differently depending on the product, and it changes with input length, output length, load and time of day. This guide explains the metrics that matter, how to design a benchmark that reflects real use, and the mistakes that make latency numbers misleading.

The metrics that matter

Time to first token (TTFT)

The time from sending the request until the first token of the response arrives. With streaming, this is when users see something happen. It includes network time, queueing at the provider, and processing the whole input (often called the "prefill" phase). TTFT grows with input length, so long prompts and large retrieved contexts slow it down.

Output speed: tokens per second and time per output token

Once generation starts, how fast do tokens arrive? Often reported as tokens per second, or as time per output token (also called inter-token latency). This determines how long a long answer takes to finish and whether streamed text feels smooth.

Total response time

TTFT plus generation time for the full output. For non-streaming uses, such as background jobs, structured extraction or tool calls inside an agent, this is the number that matters.

Tail latency: p95 and p99

Averages hide pain. The 95th and 99th percentiles show what your slowest requests look like, and users remember the slow ones. A model with a fast median but a long tail generates more complaints than its average suggests.

Throughput under load

How many requests per second, or tokens per second in total, the system sustains before latency degrades. For self-hosted models this is a capacity planning number; for APIs it interacts with rate limits.

Why published numbers rarely match yours

• Different prompt and output lengths. Speed depends heavily on both. A benchmark with 100-token prompts says little about a RAG system sending 8,000.
• No load. Single requests from an idle client do not experience queueing.
• Location and network. Your servers may be further from the provider's region.
• Reasoning tokens. Reasoning models generate hidden tokens before the visible answer, adding to TTFT as experienced by the user; see what reasoning tokens actually cost.
• Time of day and launch congestion. Shared infrastructure is busier at peak hours and in the weeks after a popular release.

Designing the benchmark

1. Use realistic request shapes

Sample real prompts from logs, or build synthetic ones with the same distribution of input and output lengths. Include your system prompt, retrieved context and conversation history; these usually dominate input length.

2. Measure from where your users are

Run the benchmark from the same region and network as your production servers. Measure on the client side, including network time, because that is what users experience.

3. Stream and timestamp every token

Use streaming even if production does not, so you can record TTFT and the arrival time of each token.

import time

def measure(client, prompt, **kw):
    t0 = time.perf_counter()
    first, n = None, 0
    for chunk in client.stream(prompt, **kw):
        if chunk.text:
            if first is None:
                first = time.perf_counter() - t0
            n += chunk.token_count
    total = time.perf_counter() - t0
    gen_time = total - first if first else 0
    return {"ttft": first, "total": total, "tokens": n,
            "tok_per_s": (n - 1) / gen_time if gen_time > 0 and n > 1 else None}

4. Apply realistic concurrency

Run at your expected peak concurrency, and at two or three times that level to see where latency degrades. Use an open-loop load pattern (requests arrive at a set rate regardless of responses) to mimic real users; closed-loop tests that wait for each response before sending the next one can hide queueing.

5. Run long enough and at different times

Collect hundreds or thousands of requests per configuration, at several times of day and on different days. Report p50, p95 and p99 for each metric, not only means.

6. Track errors and timeouts as results

Rate-limit errors, timeouts and retries are part of latency. A model that is fast but returns errors under load is slow in practice.

Ways to reduce latency

• Stream responses so users see progress immediately.
• Shorten inputs: trim conversation history, retrieve fewer but better chunks, compress system prompts.
• Use prompt caching where the provider supports it, keeping stable content at the start of the prompt.
• Limit output length with clear instructions and sensible maximums.
• Choose a smaller or faster model for latency-sensitive steps, and route only hard requests to slower models; see choosing a model is a budget decision.
• Parallelise independent calls in agents and pipelines.
• For self-hosted models, techniques like speculative decoding, quantization and batching tune the trade-off between speed and throughput. For local machines, see how to run an 8B LLM on a laptop.

Perceived latency: what users actually notice

Not all waiting feels the same. Classic usability guidance, popularised by Jakob Nielsen, describes rough thresholds: around a tenth of a second feels instant, around one second keeps a user's train of thought intact, and beyond roughly ten seconds attention starts to wander. LLM interfaces fit this pattern well.

• Show immediate acknowledgement. A typing indicator or a status message within a few hundred milliseconds tells users the system is working, even before the first token arrives.
• Stream meaningful content early. Ask the model to put the answer first and supporting detail after, so the first streamed words are useful.
• Show progress for long tasks. For agents and reasoning models, short status updates such as "Searching orders…" make long waits tolerable.
• Avoid layout jumps. Text that reflows or reorders while streaming feels slower and is harder to read.

These techniques do not change your benchmark numbers, but they change how the same numbers feel.

Reporting results

A useful latency report fits on one page:

• the request shape (input and output length distributions) and concurrency;
• p50, p95 and p99 for TTFT, tokens per second and total time;
• error and timeout rates;
• when and from where it was measured;
• a comparison with your latency budget for each product surface.

Re-run it after model changes, prompt changes that alter length, and on a regular schedule, since provider performance changes over time.

Frequently asked questions

What is time to first token (TTFT)?

TTFT is the time between sending a request and receiving the first token of the response. It includes network time, queueing and processing of the input, and it largely determines how responsive a streaming interface feels.

How do I measure LLM tokens per second?

Stream the response, record when the first and last tokens arrive, and divide the number of generated tokens after the first by the time between them. Measure under realistic load and input lengths.

Why does my LLM get slower with longer prompts?

The model must process the entire input before generating the first token, so longer inputs increase time to first token. Long contexts also use more memory, which can reduce throughput on the server.

What latency should I aim for?

It depends on the surface. Interactive chat usually needs the first visible text within one to two seconds; inline suggestions need complete results in well under a second; batch jobs care about throughput and cost more than per-request latency.