Temperature is the most commonly adjusted setting in language model APIs and one of the most misunderstood. It is often described as a "creativity dial," which is roughly right and misleading in the details. Temperature does not make a model smarter or more imaginative. It changes how the model chooses among the words it already considers likely.
This guide explains what temperature does mathematically (in plain language), how it interacts with top-p and top-k sampling, what typically changes as you move from 0 to 1.5, why temperature 0 is not perfectly deterministic, and how to choose settings for different tasks and for evaluations.
How a model chooses the next token
At each step, a language model produces a score, called a logit, for every token in its vocabulary. These scores are converted into probabilities with the softmax function. The model then picks the next token from that probability distribution, a step called sampling.
Temperature divides every logit by a number *T* before the softmax:
- T = 1 leaves the distribution as the model learned it.
- T < 1 sharpens it: likely tokens become even more likely, unlikely ones fade.
- T > 1 flattens it: unlikely tokens get a larger share of the probability.
- T → 0 approaches greedy decoding: always pick the single most likely token.
import numpy as np
def probs(logits, T):
z = np.array(logits) / max(T, 1e-6)
e = np.exp(z - z.max())
return e / e.sum()
logits = [4.0, 3.0, 1.0] # three candidate tokens
for T in (0.2, 0.7, 1.0, 1.5):
print(T, probs(logits, T).round(3))
# 0.2 [0.993 0.007 0. ] almost always the top token
# 1.5 [0.607 0.311 0.082] the third token now appears in ~8% of samplesTemperature never adds new options. It only changes how often lower-ranked options are chosen.
Top-p and top-k: the other two dials
Most APIs also offer truncation settings that remove unlikely tokens before sampling:
- Top-k keeps only the *k* most likely tokens.
- Top-p (nucleus sampling, from Holtzman et al., 2019) keeps the smallest set of tokens whose probabilities add up to *p*, such as 0.9. When the model is confident, the set is small; when it is unsure, the set grows.
Temperature and top-p interact. A high temperature with a tight top-p is far tamer than a high temperature alone, because the flattened tail is cut off anyway. A common recommendation is to adjust one and leave the other at its default, so you know which change caused what.
What changes as you raise temperature
The effects below are typical patterns; their size varies by model and task.
| Temperature | Typical behaviour | Good for |
|---|---|---|
| 0 – 0.3 | Highly repeatable, conservative wording, can repeat itself in long outputs | Extraction, classification, code, evals |
| 0.4 – 0.8 | Natural variation in phrasing, stable content | Chat, summaries, most writing |
| 0.9 – 1.2 | Noticeably varied ideas and structure, more small errors | Brainstorming, creative drafts, multiple candidates |
| 1.3 and above | Increasing incoherence, off-topic tokens, format breaks | Rarely useful without tight top-p |
Three effects are worth knowing in detail.
Format reliability drops first. Structured output, such as JSON or a fixed label, depends on every token being right. A low-probability token in the wrong place breaks the format, so even moderate temperatures increase parse failures. See structured outputs vs. plain prompts.
Accuracy on single-answer tasks usually falls slowly, then quickly. For maths or factual questions, low temperatures generally perform best for a single answer. Moderate temperatures cost little; very high ones cost a lot.
Diversity rises. Higher temperatures produce more varied outputs, which is useful when you want several distinct candidates, for example for self-consistency voting, where you sample several reasoning paths and take the majority answer.
Why temperature 0 is not fully deterministic
Many developers set temperature to 0 and expect identical outputs for identical inputs. Often they get them; sometimes they do not. Reasons include:
- Floating-point arithmetic. GPU computations can add numbers in different orders, which produces tiny differences in results. When two tokens have nearly equal scores, a tiny difference can flip the choice, and everything after that point diverges.
- Batching on the server. Hosted APIs process many users' requests together, and the batch composition can affect numerical results.
- Model updates. Providers may update model versions or infrastructure behind the same name.
- Implementation details. Some APIs treat 0 as a very small positive temperature rather than strict greedy decoding.
Some APIs offer a seed parameter for better reproducibility, usually described as best-effort. For evaluations, the practical response is to assume some variance and measure it.
Temperature and reasoning models
Some reasoning-focused models restrict or ignore temperature settings, because their outputs depend on long internal reasoning that was tuned at particular sampling settings. Check the documentation; setting a parameter the API ignores gives you false confidence about reproducibility.
Choosing temperature for evaluations
Evaluations need special care, because sampling noise can be mistaken for real differences between prompts or models.
- Evaluate at the temperature you use in production. A model tested at 0 and deployed at 0.7 has not really been tested.
- Run several samples per item when temperature is above 0, and report the mean and spread.
- Compute confidence intervals before declaring a winner; see why your 2-point improvement might be noise.
- Keep temperature fixed when comparing prompts, and change one variable at a time.
For building the eval set itself, see build an eval set in an afternoon.
Other sampling settings you may meet
Temperature, top-p and top-k are the common three, but APIs and local inference tools expose a few more. They are worth recognising so you can leave them alone deliberately rather than by accident.
- Min-p keeps only tokens whose probability is at least a set fraction of the top token's probability. Like top-p it adapts to the model's confidence, and it is popular in local inference tools because it stays coherent at higher temperatures.
- Frequency and presence penalties reduce the probability of tokens that have already appeared, to discourage repetition. Small values can help long creative outputs; large values distort word choice and can break structured formats that legitimately repeat keys.
- Repetition penalty in many local tools serves the same purpose and has the same risk for code and JSON.
- Seed fixes the random number generator for sampling, which improves reproducibility when everything else is identical. It does not remove the floating-point and batching effects described above.
- Max tokens is not a sampling setting, but it interacts with them: higher temperatures sometimes produce longer, rambling outputs that hit the limit.
When results change unexpectedly after a library or SDK upgrade, check the defaults for all of these. A changed default is a common and frustrating cause of behaviour shifts.
A practical temperature sweep
If you are unsure what setting to use, run a small sweep on your own task:
- Choose 50 representative inputs with automatic checks (format validity, correctness).
- Run each input five times at temperatures 0, 0.3, 0.7 and 1.0.
- For each temperature, record accuracy, format validity and how often the five runs agree.
- Pick the highest temperature that keeps accuracy and validity within your quality bar, if you want variety; otherwise pick the lowest.
The sweep usually takes under an hour and replaces guesswork with a clear curve for your task.
Frequently asked questions
What does temperature do in an LLM?
Temperature rescales the model's next-token scores before they are turned into probabilities. Lower values make the most likely tokens even more likely, giving consistent output; higher values give less likely tokens more chance, increasing variety and errors.
What temperature should I use?
Use 0 to 0.3 for extraction, classification, code and evaluations; 0.4 to 0.8 for chat and general writing; and around 0.9 to 1.2 for brainstorming or generating diverse candidates. Test on your own task to confirm.
Is temperature 0 deterministic?
Not always. Floating-point effects, server-side batching and model updates can cause small differences that change the output. Treat temperature 0 as highly repeatable but not guaranteed identical.
Should I change temperature or top-p?
Usually one, not both. Many providers recommend adjusting either temperature or top-p and leaving the other at its default, which makes the effect of each change easier to understand.
