Time to First Token (TTFT): The Most Important Latency Metric
Why time to first token defines perceived AI responsiveness, what drives TTFT differences between models and providers, and how to optimize for low-latency applications.
What TTFT Is and Why It Matters
Time to First Token (TTFT) is the elapsed time from sending a request to receiving the first token of the response. It represents the delay before text starts appearing in the UI. Even if a model generates at 200 TPS, a 3-second TTFT creates a frustrating experience, users see nothing for 3 seconds, assume the system is broken, and may abandon the interaction.
Research on human-computer interaction consistently finds that users tolerate around 1-2 seconds of wait time before feeling friction. TTFT above 2 seconds creates noticeably poor user experience for conversational applications. For voice AI, TTFT must be under 500ms to maintain natural conversation flow.
What Drives TTFT
TTFT is driven by: prompt processing time (all tokens in the prompt must be processed before generation starts, longer prompts = higher TTFT), Inference infrastructure queuing (shared infrastructure under load adds waiting time), model size (larger Foundation Models process prompts slower), and geographic distance (network latency to the inference cluster). KV Cache hits dramatically reduce TTFT for repeated System Prompt prefixes, a key optimisation for production deployments.
For reasoning models specifically, TTFT includes thinking time: the model generates many thinking tokens before the final answer. A model that thinks for 30 seconds before answering has extremely high effective TTFT even if actual generation speed is fast. Some implementations stream thinking tokens separately to give users feedback during this phase.
Typical TTFT Values
Fast providers: Groq with Llama achieves 50-200ms TTFT consistently. Gemini 2.5 Flash: 300-600ms. Claude Haiku 4.6: 400-700ms. GPT-4o Mini: 300-600ms. Frontier models (GPT-5, Claude Opus 4): 800ms, 2s+ depending on prompt length. Reasoning models on hard problems: 5-60 seconds for thinking phase.
TTFT varies significantly by time of day (infrastructure load), geographic region, and prompt length. Caching (prompt prefix caching available from Anthropic and OpenAI) can dramatically reduce TTFT for requests that share common prompt prefixes, saving the cost and time of reprocessing the system prompt on every request.
Read next
Tokens Per Second: Measuring LLM Generation Speed
What tokens per second (TPS) measures, how it affects real-world AI applications, which models are fastest, and how to interpret speed vs. quality tradeoffs.
Fastest LLMs: Latency Benchmarks Compared
Time to first token and tokens per second compared across major AI providers. Which models are fast enough for real-time applications?
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.