Why do AI labs spend so much on bigger training runs? Why did models suddenly get smaller and better at the same time? Why do some small models today beat much larger models from a few years ago? Most of the answers trace back to one body of research: scaling laws.
Scaling laws are empirical relationships between three things you can control when training a language model (its size, the amount of data and the amount of compute) and one thing you care about: how well it predicts text. This guide explains the main results in plain English, how thinking shifted from "bigger models" to "more data," what inference-time scaling adds, and what all of it means if you build with models rather than train them.
The three ingredients
- Parameters (N): the number of learned weights in the model, a rough measure of its capacity. An "8B model" has about 8 billion.
- Data (D): the number of training tokens the model sees. Modern models train on trillions of tokens.
- Compute (C): the total amount of arithmetic used in training, measured in floating-point operations (FLOPs). A widely used approximation for transformer training is C ≈ 6 × N × D.
The approximation shows the basic trade-off. With a fixed compute budget, you can train a bigger model on less data or a smaller model on more data. Scaling laws tell you which choice gives the best model.
What the original scaling laws found
In 2020, Kaplan and colleagues at OpenAI published "Scaling Laws for Neural Language Models." They trained many models of different sizes on different amounts of data and found that the test loss, a measure of how well the model predicts unseen text, falls as a smooth power law in each ingredient, as long as the others are not bottlenecks.
In plain terms:
- every time you multiply compute by a fixed factor, loss improves by a roughly predictable amount;
- improvements continue across many orders of magnitude, with diminishing returns but no sudden wall;
- the shape of the model (depth vs. width) matters much less than its total size.
That predictability changed how labs plan. You can train small models, fit the curve, and estimate how a much larger model will perform before spending the money.
The paper also suggested that, for a growing compute budget, most of the extra compute should go into making the model bigger rather than into more data. Several large models of that era followed this advice.
Chinchilla: data matters as much as size
In 2022, Hoffmann and colleagues at DeepMind revisited the question in "Training Compute-Optimal Large Language Models." With a different experimental setup, including adjusting the learning-rate schedule to each training length, they found that model size and data should grow in roughly equal proportion. Many existing large models were, by this analysis, undertrained: too many parameters, too few tokens.
They tested the claim directly. Chinchilla, a 70-billion-parameter model trained on about 1.4 trillion tokens, outperformed their own 280-billion-parameter Gopher model, which used a similar compute budget but far less data. The widely quoted rule of thumb that came out of this work is about 20 training tokens per parameter for compute-optimal training.
| Question | Kaplan et al. (2020) | Hoffmann et al. (2022) |
|---|---|---|
| How does loss scale? | Smooth power laws in N, D and C | Agrees |
| Extra compute should go to… | Mostly bigger models | Size and data roughly equally |
| Rule of thumb | Large models, relatively less data | ~20 tokens per parameter |
| Practical effect | Very large models | Smaller models trained on far more data |
Why today's small models are trained "too long"
Chinchilla answers a specific question: what is the best model for a fixed *training* budget? But a model is trained once and used millions of times. Every user request costs inference compute, and inference cost grows with model size.
So labs now often train smaller models on far more than 20 tokens per parameter. Meta, for example, reported training the 8B Llama 3 model on over 15 trillion tokens, which is many times the Chinchilla-optimal amount for its size. Training costs more than strictly optimal, but the result is a small model that is cheap to run and much stronger than its size would once have suggested. This is a large part of why 8B-class models can now run usefully on laptops; see how to run an 8B LLM on a laptop.
Distillation pushes the same idea further: train a small model to imitate a larger one, getting more capability per parameter.
Loss is not the same as usefulness
Scaling laws predict loss, a smooth and somewhat abstract number. What people care about is performance on tasks: answering questions, writing code, following instructions. The link between the two is real but messy.
Some task scores seem to jump suddenly at a certain scale, which led to discussion of emergent abilities (Wei et al., 2022). Later work, notably Schaeffer et al. (2023), "Are Emergent Abilities of Large Language Models a Mirage?", argued that many apparent jumps come from the choice of metric: all-or-nothing metrics such as exact match turn smooth underlying improvement into a sudden-looking step. The debate is not fully settled, but the practical lesson is that task performance is harder to forecast than loss.
Benchmarks also saturate as models improve, which makes progress look like it is slowing when the test has simply stopped discriminating; see when a benchmark saturates.
A third axis: inference-time compute
Since 2024, a new kind of scaling has become central: spending more compute when answering, not only when training. Reasoning models generate long internal chains of thought before answering, and their accuracy on hard problems tends to improve as they are allowed to think longer. Techniques like sampling several answers and voting (self-consistency) are another form of the same idea.
This gives builders a dial they did not have before: for a hard question, you can pay for more thinking instead of a bigger model. It also creates new cost patterns, covered in what reasoning tokens actually cost.
Architecture changes the curve's constants, not its existence
Techniques such as mixture of experts let a model have many parameters while using only a fraction of them for each token. That changes how compute relates to parameters, but the broad pattern remains: more effective compute and more good data produce better models, with diminishing returns.
The open question in the field is data. High-quality human-written text is finite. Synthetic data, better filtering and new data sources are all attempts to keep the data side of the curve growing.
What scaling laws mean for builders
You probably will never train a frontier model, but scaling laws still shape your decisions.
- Expect steady improvement, with diminishing returns. Next year's models will likely be better per dollar; avoid locking into long contracts or heavy fine-tuning for one specific model.
- Small models will keep getting more capable. Tasks that need a large model today may fit a small one next year. Re-test regularly; see where small models quietly win.
- Paying for inference compute is a real option. For hard, high-value questions, more reasoning may be cheaper than a larger model.
- Data quality beats data quantity for your own work. The same lesson applies to fine-tuning sets and eval sets: a smaller, cleaner set usually wins.
Frequently asked questions
What are scaling laws in AI?
Scaling laws are empirical relationships showing that language model performance, measured as loss on held-out text, improves predictably as a power law when you increase model size, training data and compute, as long as none of them is a bottleneck.
What is the Chinchilla scaling law?
From Hoffmann et al. (2022), it says that for a fixed training compute budget, model size and training tokens should grow roughly in proportion, at about 20 tokens per parameter. Their 70B Chinchilla model beat the much larger Gopher by training on more data.
Why are small models trained on so much data now?
Because inference costs matter. Training a small model well beyond the compute-optimal point costs more once, but produces a model that is cheaper to run for every request afterwards.
Are scaling laws slowing down?
Loss continues to improve with scale, but with diminishing returns, and high-quality data is becoming a constraint. Progress has also shifted partly towards inference-time compute and better training methods rather than raw size alone.
