AI Glossary: Inference
Inference is the process of using a trained AI model to make predictions on new data. It's where models transition from the lab to the real world—powering chatbots, recommendations, image generation, and every AI application you interact with.
What is Inference?
Inference (also called prediction or model serving) is the phase of machine learning where a trained model is used to make predictions on new, unseen data. During inference, the model takes input data, performs a forward pass through its neural network layers, and produces an output—all without modifying any of its learned parameters. This is the operational phase of AI: the model has been trained, its weights are frozen, and now it's delivering value by processing real-world inputs.
Inference is what makes AI useful in practice. When you type a prompt into ChatGPT, the model performs inference to generate a response. When Netflix recommends a movie, a recommendation model runs inference on your viewing history. When your phone's camera identifies a face, a vision model runs inference on the image. Every AI application you interact with is powered by inference.
Core Definition
Using a trained model to make predictions on new data. Only forward pass; no weight updates; no backpropagation.
Key Difference from Training
Training updates weights via backpropagation; inference uses fixed weights for forward computation only.
Primary Goal
Low latency (fast response) and high throughput (many predictions), with minimal computational cost.
Inference Is Where Value Is Delivered
Training is an investment; inference is the return. A model that was trained for weeks on thousands of GPUs at a cost of millions of dollars must perform inference efficiently to justify that investment. This is why inference optimization is a critical engineering discipline—the difference between a model that costs $0.01 per query and one that costs $0.0001 per query can determine whether an AI product is economically viable. Companies like OpenAI, Anthropic, and Google invest heavily in inference infrastructure because it directly affects their bottom line.
The Forward Pass
During inference, the model performs a forward pass: input data flows through the network's layers, each layer applying its learned transformation, until the final layer produces the output. There is no backward pass (backpropagation), no gradient computation, and no weight updates. The forward pass is deterministic—given the same input and the same model weights, the output is always the same (unless the model includes stochastic components like dropout, which is typically disabled during inference).
Forward Pass in Different Model Types
- Classifiers: Input → layers → logits → softmax → predicted class probabilities. A single forward pass produces the prediction.
- LLMs (autoregressive): Process input tokens → generate one output token → append to input → repeat. Each token requires a full forward pass, making LLM inference sequential and computationally intensive.
- Diffusion models: Start with random noise → iteratively denoise through multiple forward passes (typically 20-50 steps) → final image. Each denoising step is a forward pass.
- Object detectors: Input image → feature extraction → region proposals → classification + bounding box regression. Single forward pass typically produces all detections.
During inference, data flows forward through the network with fixed weights. No gradients are computed, and no weights are updated.
Inference Optimization Techniques
Making inference fast and cost-effective is a major engineering challenge. Here are the key techniques:
Model Quantization
Reducing the precision of model weights and activations from 32-bit floating point (FP32) to lower precision formats like FP16 (half precision), INT8 (8-bit integer), or even INT4. This reduces memory usage by 2-8× and speeds up computation, especially on hardware with dedicated low-precision math units. Post-training quantization (PTQ) applies quantization after training; quantization-aware training (QAT) simulates quantization during training for better accuracy. Modern LLMs are almost always served in FP16 or INT8.
Model Pruning
Removing weights or entire neurons that contribute little to the model's output. Unstructured pruning removes individual weights (creating sparse matrices), while structured pruning removes entire channels or layers. Pruning can reduce model size by 50-90% with minimal accuracy loss, but achieving actual speedup requires hardware support for sparse computation.
Knowledge Distillation
Training a smaller "student" model to mimic the outputs of a larger "teacher" model. The student learns not just the final predictions but also the teacher's internal representations and probability distributions. Distilled models can be 10-100× smaller while retaining most of the teacher's accuracy. This is how models like DistilBERT and TinyLLaMA are created.
KV-Cache for Autoregressive Models
When generating text token by token, LLMs recompute attention for all previous tokens at each step. The KV-cache stores the Key and Value matrices from previous tokens, so new tokens only need to compute attention with the new token—a massive speedup for long sequences. Without KV-cache, generating 1000 tokens would be O(n²); with KV-cache, it's O(n).
Speculative Decoding
A technique for accelerating LLM inference: a small, fast "draft model" generates several candidate tokens, and the large model verifies them in parallel. If the draft tokens are correct, the large model effectively generates multiple tokens per forward pass. This can provide 2-3× speedup without any quality degradation.
Latency vs Throughput
Inference performance is measured by two key metrics that often trade off against each other:
| Metric | Definition | Priority | Example Requirement |
|---|---|---|---|
| Latency | Time to process a single request (ms or s) | Real-time applications | Chatbot: < 200ms per token |
| Throughput | Requests processed per unit time | Batch processing | Document analysis: 1000 docs/min |
| Time to First Token (TTFT) | Latency until first output token appears (LLMs) | Interactive LLM apps | ChatGPT: < 500ms |
| Tokens Per Second (TPS) | Generation speed after first token | Streaming applications | Code completion: > 20 tokens/s |
Batching is the classic latency-throughput trade-off: processing multiple requests together increases throughput (more efficient GPU utilization) but increases latency for each individual request (they must wait for the batch to fill). Dynamic batching—where the server waits for a short time window to accumulate requests—is a common compromise.
Inference Deployment Patterns
Online Inference
Real-time serving where each request must be processed immediately with low latency. This requires always-on infrastructure (GPU servers, load balancers) and is the most expensive deployment pattern. Examples: ChatGPT, Google Search, real-time translation, voice assistants. Online inference typically uses frameworks like vLLM, TensorRT-LLM, or Triton Inference Server.
Offline (Batch) Inference
Processing large datasets without real-time constraints. The model runs on a schedule, processing accumulated data in bulk. This is much cheaper because you can batch aggressively and use spot/preemptible instances. Examples: nightly recommendation updates, embedding computation for search indexes, bulk document classification.
Edge Inference
Running inference directly on the user's device (phone, laptop, IoT sensor) rather than in the cloud. This eliminates network latency, works offline, and preserves privacy. Edge inference requires highly optimized models (quantized, pruned) that fit within device constraints. Examples: on-device speech recognition (Siri, Google Assistant), camera scene detection, keyboard autocorrect.
Hybrid Inference
Combining edge and cloud: small models run on-device for common cases, and complex cases are escalated to cloud models. This balances latency, cost, and capability. Apple's on-device + Private Cloud Compute for Apple Intelligence is a prominent example of this pattern.
Frequently Asked Questions
LLM inference is expensive because: (1) Large models require massive memory—a 70B parameter model in FP16 needs ~140GB of GPU memory just to load the weights; (2) Autoregressive generation is sequential—each token requires a full forward pass through all layers, and the KV-cache grows with sequence length; (3) High demand requires many GPUs running simultaneously; (4) The attention mechanism has quadratic complexity in sequence length. A single ChatGPT query might cost $0.001-0.01 in compute; with millions of users, this adds up to millions of dollars daily. This is why inference optimization is a multi-billion-dollar industry.
Inference uses a frozen model for prediction without modifying weights. Fine-tuning continues training on new data, updating model weights through backpropagation. Fine-tuning is a training process (albeit shorter than pre-training) that requires labeled data, backpropagation, and gradient computation. After fine-tuning, the updated model is used for inference. Fine-tuning adapts the model to specific tasks or domains; inference deploys the adapted model for actual use.
The optimal hardware depends on the model size and latency requirements: GPUs (NVIDIA A100, H100, L40S) are the standard for large model inference, offering high memory bandwidth and dedicated tensor cores; TPUs (Google) excel at high-throughput batched inference; CPUs are sufficient for small models (< 100M parameters) and can be more cost-effective; specialized inference chips (Groq LPU, Cerebras, AWS Inferentia) offer extreme throughput for specific architectures; Apple Neural Engine and Qualcomm Hexagon provide efficient on-device inference for mobile models. The trend is toward specialized hardware that trades some flexibility for massive efficiency gains.
Model warm-up refers to running a few initial inference requests to prepare the model for production serving. This includes: loading model weights into GPU memory, compiling and caching optimized CUDA kernels (JIT compilation), allocating KV-cache memory, and warming up the GPU's memory allocation. The first inference request is often 10-100× slower than subsequent requests because of this overhead. Production systems typically send "dummy" warm-up requests before exposing the model to real traffic, ensuring consistent latency from the first real request.
Large models that don't fit on a single GPU use distributed inference: Tensor parallelism splits individual layers across multiple GPUs, with each GPU computing part of the matrix multiplication and communicating results; Pipeline parallelism splits the model into sequential stages, with each GPU handling a subset of layers and passing activations to the next; Data parallelism replicates the model across GPUs and distributes requests. Modern serving frameworks like vLLM and TensorRT-LLM combine these techniques automatically. For the largest models (Llama 3 405B, GPT-4), inference requires tens of GPUs working in concert.
Continue your AI learning journey
Next, explore the critical differences between Training and Inference—two phases that are often confused but fundamentally distinct in their goals, processes, and infrastructure.
Read Next: Training vs Inference →