What Is Temperature in an LLM? – Definition and How It Works

Learn what temperature does to token sampling, how it interacts with top-p, which values suit which tasks, and what it cannot fix.

Temperature controls how sharply a model's next-token probabilities are peaked before one is sampled. It adjusts variety, not correctness.

1. What It Changes

At each step the model produces a score for every possible next token. Temperature scales those scores before they become probabilities.

• Low temperature sharpens the distribution — likely tokens become much more likely.
• High temperature flattens it — unlikely tokens gain a real chance.
• Zero makes selection greedy: always the highest-scoring token.

2. What Values Mean in Practice

3. Zero Is Not Fully Reproducible

Greedy decoding removes sampling randomness, but identical outputs are still not guaranteed: batching, hardware differences, and floating-point ordering can change results between runs.

Treat temperature zero as "as stable as available", not as a determinism guarantee.

4. Temperature and Top-P

Top-p sampling restricts the candidate set to the smallest group whose probabilities sum to a threshold, then samples within it.

The two interact, and tuning both at once makes effects hard to attribute. The common advice is to adjust one and leave the other at its default.

5. What Temperature Does Not Fix

• Hallucination. Low temperature produces confidently wrong output more consistently, not less often.
• Missing knowledge. Sampling cannot supply facts absent from the context.
• Format compliance. A schema does that; temperature only reduces variety.
• Reasoning quality. It changes selection, not capability.

Lowering temperature to improve accuracy is one of the most common misapplications: it stabilises the answer, whether or not the answer is right.

6. Choosing a Value

1. Extraction, classification, or anything parsed downstream — zero.
2. Factual answering with natural phrasing — low.
3. Drafting and conversation — moderate.
4. Idea generation — higher, and expect to discard some outputs.
5. Evaluating a prompt change — fix temperature so you measure the prompt, not the sampling.

7. A Note on Reasoning Models

Some reasoning-oriented models ignore or restrict temperature, because their sampling is managed internally. Check the provider's documentation rather than assuming the parameter applies.

8. Summary

Temperature scales token probabilities before sampling: lower is more predictable, higher is more varied. It is a control on variety alone.

Use zero where output is parsed, low for factual work, moderate for prose, and remember that reducing it makes wrong answers more consistent rather than less likely.

Potential Related Glossary

• Parsing
• Data Structure
• Unstructured Data
• Data Wrangling