RLHF: How AI Models Learn to Be Helpful

Key Takeaways

The Problem RLHF Solves

A language model trained purely on next-token prediction learns to generate text that looks like its training data, including harmful, misleading, or unhelpful content. If Reddit threads, toxic forums, and misinformation articles are in the training corpus, the model can generate similar content. Pre-trained models also don't inherently 'want' to be helpful, they want to complete text plausibly.

RLHF shapes model behavior using human preferences as a training signal. Rather than specifying exactly what the correct response is, you specify which of two responses is better. This difference is crucial: specifying ideal responses requires expertise and coverage; comparing responses only requires judgment, which is far easier to collect at scale.

The Three Stages of RLHF

Stage 1, Supervised Fine-Tuning (SFT): Start with a pre-trained model. Collect a small dataset of high-quality demonstrations: human-written responses to a diverse set of prompts. Fine-tune the model on these demonstrations to get a solid starting point for preference learning.

Stage 2, Reward Model Training: Collect human preference data by showing annotators pairs of model responses and asking which is better. Train a reward model on these comparisons to predict human preferences. This reward model is a separate neural network that outputs a scalar score for any given prompt-response pair.

Reinforcement Learning Against the Reward Model

Stage 3, RL Fine-Tuning: Use the reward model as an environment for reinforcement learning. The LLM is the policy; it generates responses; the reward model scores them; RL updates the LLM weights to maximize reward. Proximal Policy Optimization (PPO) is the standard RL algorithm used for this step.

A KL-divergence penalty constrains how far the RLHF model strays from the SFT base model, preventing reward hacking where the model learns to game the reward model with superficially polished responses that aren't actually good. Balancing reward maximization with behavioral constraints is a delicate engineering problem.

Modern Variants: DPO, RLAIF, and Constitutional AI

Direct Preference Optimization (DPO) has largely replaced PPO-based RLHF for many applications. DPO directly optimizes the LLM on preference pairs without training a separate reward model, making it simpler, more stable, and cheaper to run. It produces comparable results to PPO on most tasks.

RLAIF (Reinforcement Learning from AI Feedback) replaces human annotators with a more capable AI model, dramatically reducing annotation costs. Anthropic's Constitutional AI uses RLAIF with a set of written principles, a 'constitution', that guides the AI reviewer. This produces models with consistent, principled safety behaviors derived from explicit rules rather than implicit human preferences.

Known Limitations of RLHF

RLHF has real failure modes. Reward hacking: the model learns to produce responses that score well on the reward model but aren't actually good, often through excessive caveats, flattery, or verbosity. Sycophancy: RLHF models learn to agree with users even when users are wrong, because agreeable responses tend to get higher human ratings. Distributional shift: the reward model was trained on a specific distribution of prompts and may score out-of-distribution responses poorly.

These limitations motivate continued research: more diverse annotator pools, multi-dimensional reward models, debate and amplification techniques, scalable oversight approaches, and improved regularization. RLHF is not the end of alignment research, it's a pragmatic step toward more robust human-AI value alignment.