LLM Latency: What Makes Models Feel Fast or Slow
Two key metrics, time to first token and tokens per second, determine how responsive an LLM feels. Here's what drives each.
Key Takeaways
| Takeaway | Details |
|---|---|
| Latency Metrics | LLM latency splits into time to first token (TTFT) and tokens per second (TPS) with different drivers. |
| TTFT Impact | Time to first token determines perceived responsiveness more than generation speed for user experience. |
| Prompt Length | Longer prompts take proportionally longer to process, with 10,000-token prompts taking roughly 10x longer than 1,000-token prompts. |
| Model Size | Larger models generate fewer tokens per second than smaller models on the same hardware due to increased computation. |
| MoE Architecture | Mixture-of-Experts models like Mixtral achieve large model quality at smaller inference costs by activating only a fraction of parameters. |
| Hardware Acceleration | Groq's LPU architecture achieves 10-20x faster TPS than GPU-based inference by eliminating memory bandwidth bottlenecks. |
Two Metrics That Define LLM Latency
LLM latency is not a single number, it splits into two distinct metrics with different drivers. Time to first token (TTFT) measures how long from sending a request until the first character of output appears. Tokens per second (TPS) measures how quickly the model generates subsequent output. Both matter, but for different reasons.
TTFT determines perceived responsiveness, users feel an LLM as 'snappy' or 'laggy' based on this number more than generation speed. TPS determines how quickly long responses complete. For chat interfaces, a fast TTFT with moderate TPS feels much better than slow TTFT with fast TPS, because streaming text feels engaging while waiting for the first word feels like the application is frozen.
What Drives Time to First Token
TTFT is dominated by the prefill phase of inference: processing the input prompt through the model. Longer prompts take longer to prefill. A 10,000-token prompt takes roughly 10x longer to process than a 1,000-token prompt. This means RAG applications with long retrieved contexts, or applications that stuff system prompts with many instructions, will have higher TTFT.
Infrastructure also affects TTFT: geographic distance to the inference server adds network round-trip time, queue depth adds waiting time, and whether the provider has GPU capacity to start immediately affects it significantly. Providers with dedicated capacity (vs. shared/spot) typically offer more consistent TTFT.
Model Size and Speed
Larger models are generally slower. A 70B parameter model generates fewer tokens per second than a 7B model on the same hardware, because each forward pass involves more computation. However, model architecture matters too: Mixture-of-Experts (MoE) models like Mixtral and Llama 4 Scout activate only a fraction of parameters per token, achieving large model quality at smaller model inference costs.
Specialized hardware changes the equation significantly. Groq's LPU architecture achieves extraordinary TPS, often 10-20x faster than GPU-based inference for the same model, by eliminating memory bandwidth bottlenecks. For applications where generation speed is critical (real-time voice, live coding assistants), hardware choice can matter as much as model selection.
Read next
Time to First Token (TTFT): The Most Important Latency Metric
Why time to first token defines perceived AI responsiveness, what drives TTFT differences between models and providers, and how to optimize for low-latency applications.
Tokens Per Second: Measuring LLM Generation Speed
What tokens per second (TPS) measures, how it affects real-world AI applications, which models are fastest, and how to interpret speed vs. quality tradeoffs.
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.