Tokens and Tokenization: The Building Blocks of LLMs

Key Takeaways

What Is a Token?

A token is the fundamental unit of text that a language model processes. Before an LLM reads your text, a tokenizer splits it into tokens, subword units that are mapped to integer IDs. The model never sees raw characters; it sees a sequence of these integer IDs, which are then converted to numerical vectors for processing.

In English, tokens roughly correspond to words or word parts. Common words like 'the', 'is', 'and' are typically single tokens. Longer or less common words get split: 'tokenization' might become ['token', 'ization']. Numbers, punctuation, and whitespace each tokenize differently. A useful rule of thumb: 1 token ≈ 4 characters ≈ 0.75 English words.

Byte Pair Encoding: How GPT Tokenizes

OpenAI's models use Byte Pair Encoding (BPE). BPE starts by splitting text into individual characters, then iteratively merges the most frequent adjacent pairs into single tokens. Trained on a large corpus, BPE produces a vocabulary of typically 50,000-100,000 tokens that efficiently represents the language's common patterns.

The GPT-4 tokenizer (cl100k_base) has 100,257 tokens. You can explore any text's tokenization using OpenAI's Tokenizer playground. Seeing how your text is tokenized is illuminating, 'ChatGPT is great!' tokenizes to ['Chat', 'G', 'PT', ' is', ' great', '!'], not word-by-word as you might expect.

Why Tokenization Matters for You

Token count directly determines API cost. LLM APIs charge per million tokens for both input (your prompt) and output (the response). A 1,000-word document is roughly 1,300 tokens. Sending 10,000 such documents to GPT-4o at $2.50/M input tokens costs about $32.50. Understanding token efficiency helps control costs at scale.

Tokenization also affects performance. Some tasks encode poorly in tokens: large numbers (1,234,567 becomes ['1', ',', '234', ',', '567'], 5 tokens instead of 1 meaning), code in uncommon languages, and non-English text. Models struggle more with arithmetic partly because numbers tokenize inefficiently, breaking the model's number sense.

How Tokenization Differs Across Languages

English is the most efficiently tokenized language in most models because their training data is English-heavy. Chinese, Japanese, and Arabic text tokenizes far less efficiently, a Chinese sentence might use 3-4× as many tokens as the equivalent English sentence. This means non-English API usage is proportionally more expensive and context windows accommodate fewer words.

Newer models and tokenizers are improving multilingual efficiency. Llama 3's tokenizer (128K vocabulary) is significantly better for non-English languages than Llama 2's. Qwen's tokenizer is specifically optimized for Chinese. Multilingual tokenization efficiency is an important consideration for global applications.

Tokens and Context Windows

A model's context window is measured in tokens, not words or characters. A 128,000-token context window holds roughly 96,000 English words, about 300 pages. This sounds generous, but context fills up quickly when you include system prompts, conversation history, retrieved documents, and tool outputs.

Effective context management is a key skill in building LLM applications. Strategies include semantic chunking (splitting documents at natural boundaries), summarization (compressing earlier conversation history), and selective retrieval (only including the most relevant document sections). Every token in context costs money and can affect model quality.