What are parameters in a neural network?

Parameters (also called weights and biases) are the trainable numerical values that define a neural network's behavior. Each parameter is a number that gets adjusted during training to minimize the loss function. In a dense layer, parameters are the weights connecting each input neuron to each output neuron, plus the bias for each output neuron. The parameter count of a model is the total number of these trainable values. For example, a model with 7 billion parameters (7B) has 7 billion individual numbers that define its behavior. Parameters are stored as floating-point numbers, typically in FP16 (2 bytes) or FP32 (4 bytes) format.

How do you calculate parameter count?

Parameter count is calculated by summing the parameters in each layer. For a dense (fully-connected) layer with n_in inputs and n_out outputs: parameters = n_in × n_out + n_out (weights + biases). For a convolutional layer with kernel size k, c_in input channels, and c_out output channels: parameters = k × k × c_in × c_out + c_out (weights + biases). For an attention layer with hidden dimension d: ~4 × d² (for Q, K, V, and output projections). For embedding layers with vocabulary size V and embedding dimension d: parameters = V × d. The total parameter count is the sum across all layers, excluding non-trainable components like activation functions.

How does parameter count affect model capability?

More parameters generally mean more capability, following scaling laws. Larger models can: (1) Store more factual knowledge—more parameters = more memory capacity; (2) Learn more complex patterns and relationships; (3) Handle more nuanced reasoning; (4) Perform better on a wider range of tasks. However, more parameters is not always better: (1) Returns diminish—each doubling of parameters yields a smaller improvement; (2) Training data must scale with parameters—a large model with insufficient data underperforms a smaller model with adequate data; (3) Inference cost scales with parameter count; (4) Smaller, well-trained models often outperform larger, poorly trained ones.

How does parameter count relate to memory requirements?

Memory requirements scale linearly with parameter count and precision. In FP16 (2 bytes per parameter): a 7B parameter model requires ~14 GB for weights alone. In FP32 (4 bytes): ~28 GB. Additional memory is needed for: (1) Activations—temporary values during computation, typically 1-2× the weight memory; (2) Optimizer states—for training, Adam adds 8 bytes per parameter (momentum + variance), making total training memory ~16-20 bytes per parameter; (3) KV-cache—for LLM inference, grows with sequence length. A 7B model in FP16 needs ~14 GB for inference; training requires ~56-112 GB. This is why large models require multiple GPUs.

What is the difference between total parameters and trainable parameters?

Total parameters include all weights and biases in the model. Trainable (or active) parameters are those updated during training. In some architectures, not all parameters are trained: (1) Mixture of Experts (MoE) models have many parameters but only a subset are active for each input—GPT-4 may have ~1.8T total parameters but only ~280B active per forward pass; (2) Frozen layers in transfer learning—some parameters are kept fixed during fine-tuning; (3) Embedding layers may be frozen. When comparing models, clarify whether you're referring to total or active parameters. For most non-MoE models, total = trainable.

What are the parameter counts of major AI models?

Key model sizes: BERT-base (2018): 110M parameters; GPT-2 (2019): 1.5B; GPT-3 (2020): 175B; PaLM (2022): 540B; Llama 2 (2023): 7B, 13B, 70B; Llama 3 (2024): 8B, 70B, 405B; GPT-4 (2023): estimated ~1.8T (MoE, ~280B active); Claude 3 (2024): undisclosed, estimated 200B-2T. The trend has been toward larger models, but recent emphasis is on efficiency—smaller models with better training and data can match or exceed larger predecessors. Llama 3 8B outperforms GPT-3 175B on many benchmarks, demonstrating the importance of training quality over pure parameter count.

🔢Model Architecture

AI Glossary: Parameter Count

Parameter count is the number of trainable weights in a neural network—the fundamental measure of model size. It determines capability, memory requirements, training cost, and whether a model can even run on your hardware. From 110M parameter BERT to trillion-parameter giants, parameter count is the first number you look at when evaluating any AI model.

📅Published: Jan 2024

⏱️12 min read

📊Level: Beginner

Table of Contents

What are Parameters?
How Parameter Count is Calculated
Parameter Count and Model Capability
Memory and Infrastructure
The Parameter Landscape
Frequently Asked Questions

What are Parameters?

Parameters are the trainable numerical values that define a neural network's behavior. Each parameter is a single number—a weight or bias—that gets adjusted during training to minimize the model's loss. Think of parameters as the model's "knowledge": each one stores a tiny piece of what the model has learned. The parameter count is simply the total number of these numbers in the model.

When you hear that "Llama 3 has 70 billion parameters," it means the model contains 70 billion individual numbers. These numbers encode everything the model knows about language—grammar, facts, reasoning patterns, and even biases. Each number is stored in computer memory, typically as a 16-bit floating-point value (2 bytes), meaning a 70B parameter model requires approximately 140 GB just to store its weights.

What They Are

Trainable numerical values (weights and biases) that define the model's behavior. Each is a floating-point number.

How They're Stored

FP16 (2 bytes/param) for inference and modern training. FP32 (4 bytes) for older training. INT8/INT4 for quantized inference.

Why They Matter

Parameter count determines model capability, memory footprint, training cost, and hardware requirements.

How Parameter Count is Calculated

Dense (Fully-Connected) Layer

The most straightforward: parameters = n_in × n_out + n_out (weights + biases). A layer with 768 inputs and 768 outputs has 768 × 768 + 768 = 590,592 parameters.

Attention Layer

In transformer models, self-attention is the dominant parameter consumer. For a transformer with hidden dimension d (e.g., d=4096 for Llama 3 8B): the Q, K, V, and output projection matrices each have d × d parameters, totaling approximately 4 × d² per attention layer. For multi-head attention with h heads, each head's projections are d × (d/h), but the total across all heads equals the single-head count.

Feed-Forward Network (FFN)

The other major parameter consumer in transformers. The FFN has two dense layers: expanding from d to d_ff (typically 4×d or 8×d) and then projecting back to d: parameters = d × d_ff + d_ff × d = 2 × d × d_ff. For Llama 3 8B with d=4096 and d_ff=14336: 2 × 4096 × 14336 ≈ 117M parameters per FFN layer.

Embedding Layer

The embedding matrix maps vocabulary tokens to vectors: parameters = vocabulary_size × embedding_dimension. For Llama 3 8B with a vocabulary of 128K and d=4096: 128,000 × 4,096 ≈ 524M parameters—a significant fraction of the total!

Parameter Count and Model Capability

More parameters generally mean more capability, but the relationship is governed by scaling laws:

Power law relationship: Loss (perplexity) decreases as a power law with parameter count. Doubling parameters reduces loss by a predictable amount.
Diminishing returns: Each doubling yields a smaller absolute improvement. Going from 1B to 2B parameters gives a bigger improvement than going from 100B to 200B.
Data must scale too: The Chinchilla scaling laws show that for optimal compute efficiency, training tokens should scale proportionally with parameter count (~20 tokens per parameter). A 70B model needs ~1.4T training tokens.
Emergent abilities: Some capabilities only appear above certain parameter thresholds. Chain-of-thought reasoning, for example, emerges reliably only in models above ~50-100B parameters.

Size Isn't Everything

The Llama 3 family demonstrates that training quality can outweigh raw parameter count. Llama 3 8B (8 billion parameters) outperforms GPT-3 175B (175 billion parameters, 22× larger) on many benchmarks. This is achieved through better data curation, longer training (15T tokens vs. 300B), and improved architecture. The lesson: parameter count is important, but it's one factor among many. A well-trained small model can beat a poorly-trained large model. The parameter count arms race is giving way to a quality-over-quantity approach, especially for deployment where inference cost matters.

Memory and Infrastructure

Model Size	FP16 Inference (Weights Only)	FP32 Training (Weights + Optimizer)	GPUs Needed (80GB A100)
1B	~2 GB	~16-20 GB	1
7B	~14 GB	~56-112 GB	1-2
13B	~26 GB	~104-208 GB	2-3
70B	~140 GB	~560-1120 GB	2-8 (inference), 8-16 (training)
405B	~810 GB	~3.2-6.5 TB	12+ (inference), 40+ (training)

Memory is the primary constraint for deploying large models. Inference requires ~2× parameters in bytes (weights + KV-cache + activations). Training requires ~8-20× parameters in bytes (weights + gradients + optimizer states + activations). Quantization (INT8, INT4) can reduce inference memory by 2-4×.

The Parameter Landscape

Historical Growth

2018: BERT (110M) — The model that started the pre-training revolution. 110M parameters was considered large at the time.
2019: GPT-2 (1.5B) — OpenAI's first "too dangerous to release" model. Showed that more parameters = better text generation.
2020: GPT-3 (175B) — The model that captured the public imagination. Demonstrated few-shot learning and in-context learning.
2022: PaLM (540B) — Google's largest dense model. Showed emergent chain-of-thought reasoning.
2023: GPT-4 (~1.8T MoE) — Mixture of Experts architecture. ~1.8T total parameters, ~280B active per forward pass.
2024: Llama 3 (8B, 70B, 405B) — Open-source models achieving frontier performance through better training, not just more parameters.

Frequently Asked Questions

What is the difference between total and active parameters?

Total parameters include all weights and biases in the model. Active parameters are those used in a single forward pass. In standard dense models, total = active. In Mixture of Experts (MoE) models, only a subset of "expert" layers are activated for each input, so total parameters can be much larger than active parameters. For example, Mixtral 8x7B has ~47B total parameters but only ~13B active per token. When comparing models, it's important to note whether you're citing total or active parameters, as they can differ by 3-10× in MoE architectures.

How many parameters does the human brain have?

The human brain has approximately 86 billion neurons and an estimated 100-1000 trillion synapses. However, comparing neural network parameters to biological synapses is misleading: (1) Biological synapses are far more complex than a single weight value—they involve complex biochemical processes, temporal dynamics, and structural plasticity; (2) Each biological neuron is itself a complex computational unit; (3) The brain uses vastly different learning algorithms and architectures. The comparison is sometimes used to suggest that AI models are approaching human-level complexity, but it's a rough analogy at best. By synapse count, the largest AI models (~1.8T parameters) are within 1-2 orders of magnitude of the brain's synapse count.

Can I run a 70B parameter model on my laptop?

Not in standard precision. A 70B model in FP16 requires ~140 GB of memory, far exceeding most laptops (typically 8-32 GB RAM). However, with heavy quantization: INT4 quantization reduces the requirement to ~35 GB, which is still beyond most laptops. Apple Silicon Macs with unified memory (M2 Ultra with 192 GB) can run 70B models with quantization. For most laptops, models in the 1-8B parameter range are more practical: Llama 3 8B in INT4 requires ~4 GB, well within laptop memory. The trade-off is quality vs. accessibility—smaller models are improving rapidly, making powerful AI accessible on consumer hardware.

Why do embedding layers have so many parameters?

The embedding layer maps each vocabulary token to a dense vector. Its parameter count is vocabulary_size × embedding_dimension. For large language models, the vocabulary is typically 50K-256K tokens, and the embedding dimension is 1024-8192. This means embedding layers can have hundreds of millions to billions of parameters—a significant fraction of the total model size. For Llama 3 8B with 128K vocabulary and 4096-dim embeddings: 524M parameters, or ~6.5% of the total. Some models use weight tying (sharing embedding weights with the output projection layer) to reduce this overhead. Embedding parameters are necessary for representing the model's entire vocabulary, and they can't be easily reduced without limiting the model's vocabulary size.

Is there a limit to how many parameters a model can usefully have?

There's no known hard limit, but there are practical constraints. The Chinchilla scaling laws suggest that for a given compute budget, there's an optimal parameter count balanced with training data. Beyond that, adding parameters without adding data yields diminishing returns. However, recent "overtrained" models (Llama 3, Phi-3) show that training smaller models on more data can be more efficient than training larger models on less data. The practical limit is economic: training a 100T parameter model would cost billions of dollars and require data that doesn't exist. The frontier is shifting from "how big can we make it?" to "how good can we make it at a given size?"

Continue your AI learning journey

Next, explore FLOPs—the measure of computational work that determines how much processing power is needed to train and run AI models.