Context Windows Explained: The Working Memory of LLMs
What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.
Key Takeaways
| Takeaway | Details |
|---|---|
| Context Window | Maximum amount of text an LLM can consider at once, including prompts, history, and documents. |
| Token Growth | Context limits expanded from GPT-3's 4K tokens to Meta's Llama 4 Scout reaching 10 million tokens by 2025. |
| Lost in Middle | LLMs recall information near the start and end of context more reliably than content in the center. |
| Technical Costs | Long contexts require hundreds of gigabytes of GPU memory and quadratically more expensive attention computation. |
| Management Strategy | Place important information at start or end, summarize histories, and use RAG for selective document retrieval. |
| Architecture Trade-offs | Smaller contexts work with RAG systems while 100K+ contexts enable whole-document analysis at higher cost. |
What Is a Context Window?
The context window is the maximum amount of text an LLM can consider at once. Everything the model reads, your system prompt, conversation history, any documents you paste in, and all previous responses, must fit within this limit. Tokens beyond the limit are simply not visible to the model.
Think of it as working memory. A human reading a very long document will remember the beginning well, follow the argument, and have the ending fresh in mind. The middle may blur. LLMs have an analogous phenomenon called 'lost in the middle', they recall information near the start and end of the context more reliably than content buried in the center.
The Rapid Growth of Context Windows
Context windows have expanded dramatically. GPT-3 had a 4,096-token limit (about 3,000 words). GPT-4 expanded to 32K, then 128K. Claude 2 introduced 100K. By 2025, Claude and GPT models offer 200K tokens; Google's Gemini models offer 1 million tokens; and Meta's Llama 4 Scout reaches an unprecedented 10 million tokens.
Each leap opened new use cases: 8K enabled basic document Q&A; 128K enabled full book analysis; 1M enabled whole codebase understanding; 10M enables processing entire software repositories, years of correspondence, or complete scientific literature in a single prompt.
The Technical Cost of Long Contexts
Long context windows are expensive in memory and compute. The KV cache, which stores attention key-value pairs for all context tokens, grows linearly with context length. A 1M token context with a large model requires hundreds of gigabytes of GPU memory. This is why long-context queries cost more and why models need specialized infrastructure to serve them efficiently.
Attention computation is also O(n²) in sequence length, quadratically more expensive as context grows. FlashAttention and chunked attention implementations dramatically improve this in practice, but serving 1M-token contexts at scale remains a significant engineering challenge. This is why Gemini's 1M context pricing is higher than standard rates.
Practical Context Management
For most applications, effective context management requires being deliberate about what you include. Best practices: put the most important information at the start or end of the context (where retrieval is most reliable), summarize rather than include entire conversation histories for long interactions, use RAG to retrieve only relevant document sections rather than pasting everything, and monitor token usage to control costs.
Context window size enables different product designs. Smaller effective contexts work well with retrieval-based architectures (RAG). Very large contexts (100K+) enable whole-document analysis without chunking, which eliminates the retrieval problem at the cost of higher per-query cost. The right trade-off depends on your document sizes, query patterns, and budget.
Read next
Tokens and Tokenization: The Building Blocks of LLMs
Everything you need to know about tokens, how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.
Retrieval-Augmented Generation (RAG) Explained
How RAG systems work, why they're the standard architecture for enterprise AI, the common failure modes, and how to build a production-quality RAG pipeline.
KV Cache: How LLMs Remember Context Efficiently
The key-value cache is the mechanism that lets LLMs process long conversations without recomputing everything from scratch on every token.