Training vs Inference: Two Phases of an LLM's Life

Key Takeaways

Two Completely Different Activities

Training and inference are fundamentally different computational activities that happen at different times, at different scales, and require different infrastructure. Training is done once (or occasionally repeated for new versions), by the model's creators, using thousands of GPUs over months. Inference is done billions of times per day, by API users, using a much smaller slice of hardware per request.

Understanding this distinction helps explain LLM economics: the enormous cost of training (GPT-4 was reported to cost over $100M) is amortized across billions of API calls. Inference costs, the marginal cost of generating your response, are what API pricing reflects. They're much smaller per call but enormous in aggregate.

Pre-Training: Learning from the World

Pre-training is where an LLM's capabilities are established. The model is initialized with random weights and trained on trillions of tokens of text using next-token prediction. For each training example, the model predicts the next token, the prediction is compared to the actual token (cross-entropy loss), and gradient descent updates the weights to improve future predictions.

Modern pre-training runs consume extraordinary resources: Llama 3 70B was trained on 15 trillion tokens using thousands of A100 GPUs for months. These runs cost tens to hundreds of millions of dollars, which is why frontier pre-training is concentrated among a handful of well-funded labs. The resulting weights encode an enormous compression of human knowledge, these weights are what makes the model valuable.

Fine-Tuning and Alignment

After pre-training, models typically undergo supervised fine-tuning (SFT) on curated instruction-response pairs, teaching the model to follow directions rather than just complete text. This is followed by reinforcement learning from human feedback (RLHF) or similar alignment techniques, which shape the model to be helpful, honest, and to avoid harmful outputs.

Fine-tuning is far cheaper than pre-training, orders of magnitude less compute. This is why organizations can fine-tune open-weight models for their specific domains without billion-dollar budgets. LoRA and other parameter-efficient methods reduce fine-tuning costs further, enabling adaptation on consumer hardware.

Inference: Running the Model

Inference is the forward pass of the model: given an input sequence, compute the next token probability distribution, sample a token, append it to the context, and repeat. The computational cost scales with model size and sequence length. A 70B parameter model requires roughly 140GB of GPU memory for the weights alone (at 16-bit precision).

Modern inference optimization includes batching (processing multiple requests together), KV caching (reusing computed attention states), quantization (reducing weight precision to 4-bit or 8-bit), and speculative decoding (using a small draft model to propose tokens that a large model verifies in parallel). These techniques together reduce inference cost by 5-20× compared to naive implementation.