Temperature, Top-p, and the Knobs That Matter

Every LLM generates text one token at a time. Before each token is selected, the model produces a logit — a raw score — for every token in its vocabulary (often 50 000+ tokens). The raw scores are meaningless on their own; the model converts them into a probability distribution through a function called softmax.

The key insight is what happens inside that formula when you divide by T before taking the exponential:

• Divide by a small T → differences between logits are amplified → winner-take-all distribution
• Divide by large T → differences shrink → flatter, more uniform distribution
• T = 1 → raw softmax, the model’s “natural” distribution

You saw this logit-to-probability step described in “How LLMs Work” — now you’re getting the dial that reshapes it.

Temperature is the most-used and most-misunderstood parameter. Move the slider and watch the distribution reshape in real time.

When to lower temperature (toward 0)

• JSON extraction, structured output
• Code that must compile
• Factual Q&A where hallucination risk is high
• Classification tasks

When to raise temperature (toward 1–1.3)

• Brainstorming, creative writing, marketing copy
• Paraphrase generation (you want variety)
• Generating multiple diverse candidates for a downstream ranker

Even at a sensible temperature, the probability distribution has a long tail: hundreds of tokens with tiny but non-zero probabilities. If the model samples that tail too often, outputs drift toward incoherence. Top-p sampling (also called nucleus sampling) fixes this.

The elegant property: the nucleus automatically shrinks when the model is confident and expands when it’s uncertain. If the top token has 90% probability, the nucleus at p=0.9 contains just that one token. If probability is spread across 30 tokens, the nucleus contains all 30.

Practical guidance

• p = 0.9–0.95 is the community default for conversational tasks
• p = 1.0 disables top-p filtering; temperature alone governs selection
• p = 0.1–0.5 for very structured outputs where you want a tight nucleus

Top-k is a blunter instrument than top-p:

• Advantage: fully predictable computational cost — always exactly k candidates
• Disadvantage: fixed k=40 treats a near-uniform distribution (where 40 tokens are plausible) the same as a peaked distribution (where 40 tokens includes a lot of garbage)

Top-k and top-p are usually applied together. The order is: apply top-k first (drop everything outside the k best), then apply top-p (drop everything outside the nucleus), then sample with temperature-scaled probabilities. This means each constraint can only make the effective vocabulary smaller, never larger.

When top-k matters most

• When your API exposes top-k as the primary diversity knob (some local model runtimes)
• As a safety net paired with a loose top-p — for example, top-k=100, top-p=0.95
• k=1 means greedy decoding — same effect as temperature=0

Two parameters control when the model stops rather than what it samples:

max_tokens

The hard ceiling on output length. The model cannot exceed it. Setting this too low truncates answers mid-sentence; setting it too high wastes compute and (in pay-per-token APIs) money.

Stop sequences

An array of strings that, when generated, cause the model to stop immediately (the stop string itself is discarded). This is how you enforce output fences from the previous lesson’s prompt templates:

const response = await client.messages.create({
  model: "claude-opus-4-5",
  max_tokens: 256,
  stop_sequences: ["```"],   // stop when the model tries to close the code fence
  messages: [{ role: "user", content: prompt }],
});

\`\`\`json
{ "name": "Alice", "age": 30 }
\`\`\`

The interactions between parameters are where practitioners get confused. This widget lets you explore four canonical presets and then build your own combination.