Sampling Strategies: How LLMs Choose the Next Word
Every token an LLM generates is chosen via a sampling strategy. Understanding temperature, top-p, and top-k reveals how models balance quality and creativity.
Key Takeaways
| Takeaway | Details |
|---|---|
| Probability Distribution | LLMs generate probability distributions over 30,000-150,000 tokens at each step, reflecting likelihood of each token following current context. |
| Temperature Control | Temperature adjusts randomness from 0 (deterministic) to 2.0+ (highly creative), with 0.7 being a common balanced default. |
| Top-K Sampling | Restricts sampling to K highest-probability tokens, with K=50 being a typical setting that ignores the rest of vocabulary. |
| Top-P Filtering | Dynamically includes tokens until cumulative probability reaches P (0.9-0.95), creating adaptive sampling sets based on model confidence. |
| Modern Implementation | Most systems combine temperature with top-P filtering, while top-K is less commonly used in current practice. |
The Probability Distribution Over Tokens
At each generation step, an LLM produces a probability distribution over its entire vocabulary, typically 30,000-150,000 possible tokens. The distribution reflects how likely each token is to follow the current context. 'Paris' might have 0.7 probability after 'The capital of France is', while 'Lyon' might have 0.05 and 'cheese' might have 0.001.
The sampling strategy determines how to convert this distribution into a concrete token choice. Different strategies make different tradeoffs between determinism (always picking the highest-probability token) and diversity (sometimes choosing lower-probability, more surprising tokens).
Temperature: The Master Dial
Temperature is a scalar applied to the logits (raw scores) before computing probabilities. A temperature of 1.0 leaves the distribution unchanged. Lower temperatures (0.1-0.7) sharpen the distribution, the highest-probability tokens become even more likely, reducing randomness. Higher temperatures (1.2-2.0) flatten the distribution, all tokens become more equally likely, increasing creativity and diversity.
Temperature 0 (greedy decoding) always picks the single highest-probability token. This is fully deterministic and often produces high-quality but repetitive output. Temperature settings around 0.7 are a common default balancing quality and variety. Creative applications like story generation typically use 0.8-1.1; factual applications use 0.2-0.5.
Top-K and Top-P Filtering
Top-K sampling restricts the sampling pool to the K highest-probability tokens, redistributing probability mass among only those tokens before sampling. Top-K = 1 is equivalent to greedy decoding. Top-K = 50 considers the 50 most likely options at each step, ignoring the rest of the vocabulary.
Top-P (nucleus sampling) is more dynamic: it includes tokens in order of descending probability until their cumulative probability reaches P (e.g., 0.9 or 0.95), then samples from that adaptive set. When the model is confident, the nucleus is small (few tokens); when uncertain, it's large. Most modern systems use temperature + top-P together, with top-K less common in practice. These parameters work together: temperature shapes the distribution, top-P or top-K then filters it.
Read next
Temperature and Sampling: Controlling LLM Creativity
A clear explanation of temperature, top-p, top-k, and how sampling parameters control the balance between determinism and creativity in LLM outputs.
Top-P Sampling: Nucleus Sampling Explained
Top-P (nucleus) sampling dynamically selects the smallest set of tokens covering P% of the probability mass, adapting to model confidence at each step.
Top-K Sampling: Limiting Randomness in Text Generation
Top-K sampling restricts token selection to the K most probable options at each step, balancing quality and diversity in LLM outputs.