Why the Same Model Almost Never Gives the Same Answer Twice

Why doesn’t a language model give the same answer to the same question every time?

A language model does not compute a single answer at each word. It computes a probability for every possible next word. A separate step, sampling, then draws the actual word from that distribution: a random draw, weighted by those probabilities. Temperature, top-p, and top-k set how that draw happens.

Same prompt, same model, sent twice. Two different answers. No network glitch, no bug.

That is exactly how it is supposed to work.

A distribution, not a single word

A language model does not pick a next word directly. For every word in its vocabularyVocabularyThe fixed list of every word (more precisely: token) a model knows and can output. Usually more than a hundred thousand entries., it first computes a number. The number says how well that word fits next. More than a hundred thousand such numbers, one per possible word.

An example: the model currently has four words in view. All four are in the running as the next one:

Together these numbers form a distributionDistributionA number for every possible next word, all of them together add up to 100 percent. Sampling draws the actual word from it.over every possible next word. A second step then draws the actual word. And this step does not automatically take the frontmost word. It draws as if from a raffle drum with a hundred tickets: 42 of them carry Word A, 28 Word B, 15 Word C. Word A wins most often. Still, more than half of all draws land on a different word.

Above, Word A got drawn: the largest chance, but no guarantee. Here is the same drum across ten draws in a row:

This randomness is not a side effect, it is wanted. A model that only ever takes the most likely word writes surprisingly monotonous text: it repeats itself, reaches for the same phrasings again and again, and quickly sounds mechanical. Natural-sounding text only emerges once the second and third most likely words get their turn regularly. This weighted random draw has a name: sampling.

Under the Hood

The four percentages above are already a result, not a raw value. The last model layer first produces raw numbers, no percentages, one per token in the vocabulary. These numbers are called logitsLogitsThe raw, unnormalized values from the last model layer, one per token in the vocabulary. Not probabilities yet.. A function called softmaxSoftmaxThe function that turns logits into a distribution whose values sum to 100 percent.turns them into exactly the bars shown above.

Temperature scales the logits before softmax runs. A value below 1 sharpens the distribution, a value above 1 flattens it. Here is the example from above at two different temperature values:

At temperature 0, only the single highest bar survives, the same one every time: greedy decodingGreedy decodingAlways picking the single most likely next word, never drawing. The limit case of sampling at temperature 0..

Top-k and top-p both narrow the candidate pool before the draw, in different ways. Top-k always takes exactly k words, no matter how sharp or flat the distribution is. Top-p (nucleus sampling) takes as many words as it needs until their probabilities together cross a threshold p. For a sharp distribution that is a handful of words, for a flat one many. On the same example from above, the difference looks like this:

Top-k = 2 takes exactly the first two words either way, regardless of the threshold.

On top of that come repetition penalties against loops, min-p as a probability floor, and a seed for the draw itself.

Important: temperature 0 is nearly deterministic, not guaranteed. The reason sits in rounding. A computer rounds after every single calculation step, to a fixed number of digits. The result of a long sum therefore depends minimally on its order. An example with four digits of precision: (1000 + 0.4) + 0.4 rounds down twice and stays at 1000. Computing 0.4 + 0.4 first and then adding the 1000 gives 1001 instead. No step was wrong, only the order differed. Sums like these, with thousands of steps, sit behind every number the model computes per word. Their order depends on the batch size: on how many requests the server happens to process together. That fluctuates with load from second to second. Usually this only shifts the last decimal place, and nothing happens. But when two words sit nearly tied, exactly that place decides which one is ahead. And from that one word on, the whole answer takes a different path.

When Reproducibility Is the Wrong Expectation

Creative tasks need spread: ad copy, idea variations, brainstorming. Greedy decoding only ever picks the most likely next word there, not the best answer, and repeats itself fast. Structured tasks need the opposite: extracting an invoice number, classifying a field, forcing a contract into a fixed schema. High temperature is a mistake there, not creativity. Two random variants of the same extraction are barely distinguishable from a bug to the code reading them downstream.

An easy but risky conclusion in German Mittelstand projects: setting temperature to 0 and writing “deterministic” into the spec. For documentation, an audit trail, or a compliance answer, that is not quite enough, because even greedy decoding stays only nearly deterministic. Anyone who needs to verify a request with an exactly reproducible wording also needs a fixed seed and a fixed model version. Even then, a gap remains on a shared API: the provider decides which other requests land in the same batch as one’s own, and exactly that can flip a close call between two words. When it counts, only a separate check helps, rather than the assumption that temperature 0 means exactly identical.

A model holds no single correct answer in its head. Only a distribution of possible answers. Sampling decides which one gets drawn this time. Temperature decides how widely that draw spreads.

Next term: the untrusted function. Why a language model is technically treated like an unreliable function, not a database.

Frequently Asked Questions

What is sampling in a language model? Sampling is the step that draws one word out of the probability distribution the model computed over every possible next word: a random draw, weighted by the probabilities. The most likely word wins most often, but not always.

What is the difference between logits and the finished distribution? Logits are the raw, unnormalized values from the last model layer. Softmax turns them into a distribution that sums to 100 percent.

What does temperature do during sampling? Temperature scales the logits before the softmax function. Low values sharpen the distribution, high values flatten it.

What is the difference between top-k and top-p? Top-k always takes exactly k candidates. Top-p (nucleus sampling) adds up the probabilities from the top until the threshold p is reached. When one word dominates the distribution, that is a handful of candidates; when the probability spreads across many words, correspondingly more.

Does temperature 0 really always give the same answer? Almost, but not guaranteed. Computers round after every calculation step, and the order of those steps depends on how many requests the server happens to process together. When two words sit nearly tied, that tiny shift can decide which one wins.

Part of the series AI Engineering Explained. Related: What Is a Token? and Why the Same AI Request Has Two Different Speeds.