How to deploy LLM inference at scale with vLLM

HuggingFace Transformers is the standard for model loading and fine-tuning, but its inference is slow under load because each request allocates KV cache memory statically. vLLM's PagedAttention manages the KV cache like virtual memory, dynamically allocating pages across requests. This allows continuous batching of hundreds of concurrent requests on a single GPU.

For a 7B model on an A100, vLLM achieves around 3000-4000 tokens/second throughput, 10-20× more than a naive HuggingFace setup. For APIs serving multiple users, this dramatically reduces cost per query.

vLLM requires an NVIDIA GPU with CUDA 12.1+. Install via pip: `pip install vllm`. The package is large (~2 GB) as it includes compiled CUDA kernels. For the latest features, install from source: `pip install git+https://github.com/vllm-project/vllm.git`.

On AMD GPUs (MI250/MI300), use the ROCm build: `pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.1`. Apple Silicon is not supported, use Ollama or llama.cpp for local Mac inference.

Start the server: `vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --port 8000`. The `--tensor-parallel-size 2` flag splits the model across 2 GPUs. `--gpu-memory-utilization 0.9` allows vLLM to use 90% of VRAM for the KV cache.

The server exposes an OpenAI-compatible API at `http://localhost:8000/v1`. Any OpenAI SDK client works without modification. The `/v1/models` endpoint lists the available model. The model name in requests should match the HuggingFace model ID.

For maximum throughput (batch processing): increase `--max-num-seqs 256` (max concurrent sequences) and enable continuous batching (default). Set `--max-num-batched-tokens 16384` to allow larger batches. Throughput improves linearly with batch size up to GPU saturation.

For minimum latency (interactive chat): use a smaller `--max-num-seqs 8`, enable speculative decoding with `--speculative-model <small-draft-model>`, and use `--enforce-eager` to skip CUDA graph capture for faster startup. Speculative decoding reduces latency by 2-3× for chat-length generations.

For a 70B model on a single 80GB GPU, use AWQ quantisation: `vllm serve meta-llama/Meta-Llama-3-70B-Instruct-AWQ --quantization awq`. AWQ (Activation-aware Weight Quantisation) reduces model size to ~4 bits with minimal quality loss and is faster than GPTQ.

For models larger than your VRAM, use pipeline parallelism across multiple nodes: `--pipeline-parallel-size 2` splits the model's layers across 2 nodes. Combined with tensor parallelism (`--tensor-parallel-size 4` per node), you can serve 405B+ models across a GPU cluster.