What are FLOPs in AI?

FLOPs (Floating Point Operations) measure the number of mathematical operations (additions, multiplications, etc.) performed on floating-point numbers during a neural network computation. One FLOP is one such operation. FLOPs quantify the computational work required to train or run a model. For example, training GPT-3 required approximately 3.14 × 10^23 FLOPs—an enormous number that helps explain why training large models costs millions of dollars. FLOPs are the currency of AI compute: they determine training time, cost, and hardware requirements.

What is the difference between FLOPs and FLOPS?

FLOPs (with lowercase 's') is a count: the total number of floating-point operations performed. FLOPS (with uppercase 'S') is a rate: Floating-point Operations Per Second, measuring computational speed. An NVIDIA H100 GPU can perform about 2,000 TFLOPS (2 × 10^15 FLOPS) in FP16. If a model forward pass requires 10^15 FLOPs, an H100 can process it in about 0.5 seconds. The distinction is like miles vs. miles per hour: FLOPs = distance traveled; FLOPS = speed. When someone says 'the training run required 10^24 FLOPs,' they're talking about total work. When they say 'this GPU does 500 TFLOPS,' they're talking about speed.

How are FLOPs calculated for a neural network?

For a dense layer with n_in inputs and n_out outputs: forward FLOPs = 2 × n_in × n_out (one multiply and one add per weight). For a transformer model, the approximate FLOPs per forward pass is: ~2 × P × N, where P is the number of parameters and N is the sequence length (in tokens). For training, the total FLOPs ≈ 6 × N_tokens × P, because training requires both forward and backward passes (~3× forward cost). For autoregressive generation, the FLOPs scale quadratically with sequence length due to attention. Estimating FLOPs is essential for planning training budgets and comparing model efficiency.

How do FLOPs relate to training cost?

Training cost is directly proportional to FLOPs. The relationship: Cost ≈ (Total FLOPs) / (Hardware FLOPS × Utilization). For example, training Llama 3 8B on 15T tokens requires approximately 15T tokens × 6 × 8B parameters = ~7.2 × 10^23 FLOPs. On H100 GPUs running at ~50% utilization (~1,000 TFLOPS effective each), this takes ~720,000 GPU-hours, or ~$1.4M at $2/GPU-hour. The Chinchilla scaling laws formalized the relationship between FLOPs (compute budget), parameters, and training tokens, providing the theoretical foundation for planning training runs. FLOPs are the universal unit for comparing model efficiency across architectures.

What is compute-efficient training?

Compute-efficient training means getting the best possible model performance for a given FLOPs budget. The Chinchilla paper showed that for compute-optimal training, models should be trained with roughly 20× more tokens than parameters (e.g., a 10B model should see ~200B tokens). However, many recent models are 'overtrained'—using more FLOPs on smaller models than Chinchilla-optimal would suggest. This is economically rational because inference is cheaper than training: it's worth spending extra FLOPs on training a smaller model that's cheaper to serve. Compute efficiency is about the total cost of ownership (training + inference), not just training FLOPs in isolation.

How many FLOPs do major AI models require?

Approximate training FLOPs: BERT (2018): ~10^20 FLOPs; GPT-3 (2020): ~3.1 × 10^23 FLOPs; Llama 2 70B (2023): ~8.4 × 10^23 FLOPs; Llama 3 8B (2024): ~7.2 × 10^23 FLOPs; Llama 3 70B: ~6.3 × 10^24 FLOPs; Llama 3 405B: ~3.8 × 10^25 FLOPs; GPT-4 (2023): estimated ~2 × 10^25 FLOPs (approximate). For inference, a single GPT-3 query requires roughly 350 × 10^9 FLOPs (350 GFLOPS per token × 1000 output tokens). These numbers have been roughly doubling every 6-12 months, following trends in compute scaling that have held for over a decade.

⚙️Compute & Infrastructure

AI Glossary: FLOPs

FLOPs (Floating Point Operations) are the universal currency of AI compute. They measure the mathematical work required to train and run models—from the 10^20 FLOPs needed for early BERT models to the 10^25+ FLOPs consumed by today's frontier systems. Understanding FLOPs is essential for estimating costs, planning infrastructure, and following AI scaling trends.

📅Published: Jan 2024

⏱️12 min read

📊Level: Intermediate

Table of Contents

What are FLOPs?
FLOPs vs FLOPS
How FLOPs are Calculated
FLOPs and Scaling Laws
From FLOPs to Cost
Frequently Asked Questions

What are FLOPs?

FLOPs (Floating Point Operations) measure the number of fundamental mathematical operations—additions, multiplications, and fused multiply-adds—performed on floating-point numbers during a neural network computation. Each individual addition or multiplication counts as one FLOP. A fused multiply-add (FMA), which performs a × b + c in a single operation, typically counts as 2 FLOPs (one multiply + one add).

FLOPs are the fundamental measure of computational work in AI. They are hardware-independent—unlike wall-clock time, which depends on the specific hardware used, FLOPs measure the abstract mathematical work required. This makes them ideal for comparing model efficiency, estimating training costs, and understanding scaling trends. The AI research community uses FLOPs as the primary metric for quantifying compute requirements.

Definition

A count of individual floating-point mathematical operations (additions, multiplications) performed during computation.

Why It Matters

FLOPs determine training cost, inference cost, hardware requirements, and model efficiency comparisons.

Scale

Modern AI training spans 10^20 to 10^25+ FLOPs. An H100 GPU performs ~2 × 10^15 FLOPS (FP16).

The Scale of AI Compute

AI compute requirements have grown by approximately 4-5× per year since 2012. GPT-3 (2020) needed ~3 × 10^23 FLOPs. GPT-4 (2023) is estimated at ~2 × 10^25 FLOPs for training—roughly 100× more. To put this in perspective: the human brain performs roughly 10^18 operations per second. Training GPT-4 required more total mathematical operations than a human brain performs in a year. This exponential growth in compute is the primary driver of AI progress, enabled by Moore's Law, specialized AI hardware, and massive investments in compute infrastructure.

FLOPs vs FLOPS: The Critical Distinction

This is one of the most common confusions in AI infrastructure:

FLOPs (lowercase 's'): A count of operations. "Training this model required 10^23 FLOPs." This is like saying "the trip was 500 miles."
FLOPS (uppercase 'S'): Floating-point Operations Per Second. A rate of computation. "This GPU delivers 500 TFLOPS." This is like saying "the car goes 60 miles per hour."

Notation	Meaning	Unit of	Example
FLOP	1 operation	Single operation	One multiply: a × b
FLOPs	Total operations	Work (like miles)	Model needs 10^23 FLOPs
FLOPS	Operations per second	Speed (like mph)	GPU does 500 TFLOPS
KFLOPs / MFLOPS / GFLOPS / TFLOPS / PFLOPS	Kilo (10³), Mega (10⁶), Giga (10⁹), Tera (10¹²), Peta (10¹⁵) operations	Scale prefixes	H100: 2000 TFLOPS (FP8)

How FLOPs are Calculated

Per-Layer Calculation

For a dense (fully-connected) layer with n_in inputs and n_out outputs: forward FLOPs ≈ 2 × n_in × n_out (one multiply and one add for each weight, or one FMA counting as 2 FLOPs). For a convolutional layer with kernel size k, c_in input channels, c_out output channels, and output spatial dimensions h_out × w_out: forward FLOPs ≈ 2 × k² × c_in × c_out × h_out × w_out.

Transformer FLOPs Approximation

For a transformer model, the dominant operations are matrix multiplications in attention and FFN layers. A useful approximation for forward pass FLOPs per token is: FLOPs_per_token ≈ 2 × P + O(N² × d), where P is parameter count, N is sequence length, and d is hidden dimension. The first term covers linear layers; the second covers attention's quadratic computation. For training, total FLOPs ≈ 6 × P × D, where D is the total number of training tokens. The factor of 6 accounts for forward pass (2×) and backward pass (4×, including gradient computation and weight update).

Autoregressive Generation

Generating text token-by-token with an LLM: the first token requires processing the entire input sequence. Each subsequent token requires processing all previous tokens plus the new one, with KV-cache optimization reducing the per-token cost. Total inference FLOPs for generating T output tokens from an input of length N is approximately: FLOPs ≈ 2 × P × (N + T) + O((N+T)² × d). This is why long generation is expensive—FLOPs scale with the square of total sequence length.

FLOPs and Scaling Laws

The relationship between FLOPs, parameters, and performance is governed by scaling laws—empirically discovered mathematical relationships that predict how model performance improves with more compute:

Kaplan et al. (2020, OpenAI): Showed that loss decreases as a power law with compute (FLOPs), model size, and data. Suggested that increasing model size is more important than increasing data for a given compute budget.
Chinchilla (2022, DeepMind): Refined the scaling laws, showing that model size and training tokens should be scaled equally for compute-optimal training. For a given FLOPs budget C, optimal parameters P ≈ C^0.5 and optimal tokens D ≈ C^0.5.
Chinchilla-optimal ratio: ~20 training tokens per parameter. A 70B model should be trained on ~1.4T tokens for compute-optimal performance.

From FLOPs to Cost

The economic reality of AI compute is directly derived from FLOPs:

Model	Training FLOPs	Estimated Training Cost	Inference FLOPs/Query
BERT-base	~10^20	~$1,000	~10^10
GPT-3	~3 × 10^23	~$5-10M	~3.5 × 10^11
Llama 3 8B	~7 × 10^23	~$1.5M	~2 × 10^10
Llama 3 70B	~6 × 10^24	~$15M	~1.5 × 10^11
Llama 3 405B	~4 × 10^25	~$80M	~8 × 10^11

The cost formula: Cost ≈ Total_FLOPs / (GPU_FLOPS × Utilization × GPU_Price_per_Second). With H100 GPUs at ~2000 TFLOPS (FP8) and ~50% utilization, and $2/GPU-hour pricing, 10^23 FLOPs costs approximately $28,000 in GPU compute. Real costs are higher due to engineering overhead, failed runs, and infrastructure.

Frequently Asked Questions

Why do FLOPs matter if GPUs already have TFLOPS ratings?

GPU TFLOPS ratings tell you the hardware's theoretical peak performance. FLOPs tell you the model's computational requirements. Together, they let you estimate training time: Time ≈ FLOPs / TFLOPS. However, actual performance is typically 30-70% of theoretical peak due to memory bandwidth limitations, communication overhead, and inefficient kernel implementations. This is why the "Model FLOPs Utilization" (MFU) metric is important—it measures what fraction of theoretical FLOPS a training run actually achieves. Top training runs achieve 40-55% MFU.

How do different numerical precisions affect FLOPs?

FLOPs count is precision-agnostic—a multiply is a multiply whether it's FP32, FP16, or INT8. However, hardware performance varies dramatically by precision. The NVIDIA H100 delivers ~67 TFLOPS in FP64, ~1000 TFLOPS in FP32/TF32, ~2000 TFLOPS in FP16/BF16, and ~4000 TFLOPS in FP8/INT8. This is why mixed-precision training (using FP16 for most operations, FP32 for accumulation) has become standard. The same FLOPs count runs 2-4× faster in lower precision because hardware can process more operations per second in those formats.

What is the difference between MACs and FLOPs?

MAC (Multiply-Accumulate) is a single hardware operation that performs a × b + c. In FLOPs counting, one MAC is typically counted as 2 FLOPs (one multiply + one add). Some papers and tools report MACs instead of FLOPs—a model that requires 10^12 MACs requires 2 × 10^12 FLOPs. Always check which convention is being used. Hardware specifications often report MACs or FMA operations, while research papers more commonly report FLOPs. The 2× difference is important when converting between the two for cost estimates.

How does sparse computation affect FLOPs?

Sparse models (with many zero weights) can reduce effective FLOPs. If 50% of weights are zero, the model theoretically requires only 50% of the FLOPs—but only if the hardware supports sparse computation efficiently. NVIDIA's Ampere/Hopper architectures support 2:4 structured sparsity, which can double throughput for sparse tensors. Unstructured sparsity (random zeros) is much harder to accelerate. Mixture of Experts (MoE) models achieve sparsity by only activating a subset of parameters per token—reducing FLOPs per token while keeping total parameters high. Sparse FLOPs counting is nuanced: the theoretical FLOPs saved by sparsity may not translate to wall-clock speedup without hardware support.

What is the trend in FLOPs efficiency?

AI models are becoming more FLOPs-efficient. Llama 3 8B achieves better performance than GPT-3 175B while using fewer total training FLOPs. This is driven by: better architectures (grouped-query attention, SwiGLU activations), improved training techniques (better learning rate schedules, data curricula), higher-quality training data, and longer training on more data rather than just making models bigger. The trend is toward getting more capability per FLOP, which is economically essential—if every capability improvement requires 10× more FLOPs, progress becomes prohibitively expensive. The AI industry is transitioning from the brute-force scaling era to the efficiency era.

Complete your AI learning journey

You've reached the end of the AI Glossary! Explore more AI topics in our Getting Started guide, or compare models to find the right one for your needs.

Explore Getting Started →