Sampling Parameters (Temperature, Top-p, Min-p)

1. Concept Overview & Systemic Problem

A language model does not generate text directly. At each autoregressive step, the output layer of the transformer produces an array of raw unnormalized numbers — logits for each word from a vocabulary of over 100,000 elements. The transformation of these raw numbers into a specific selected token is regulated by a decoding algorithm.

If the token with the absolute highest probability is always chosen (Greedy Search), the model becomes predictable, dry, and often falls into infinite loops of repeating the same phrases. Conversely, if words are chosen completely at random, the generation devolves into nonsensical ramblings and syntactical errors.

Sampling Parameters are a set of mathematical regulators that alter the shape of the probability distribution before the selection of a symbol. Understanding their internal mechanics allows engineers to finely balance between strict determinism of the compiler and creative heuristics in solution searching.

2. Architectural Taxonomy & Mental Model

The logits processing pipeline consists of sequential mathematical filters:

┌─────────────────────────────────────────────────────────────┐
│                 TOKEN DECODING PIPELINE                     │
├─────────────────────────────────────────────────────────────┤
│ 1. Raw Output Logits z_i from Transformer Linear Head       │
├─────────────────────────────────────────────────────────────┤
│ 2. Penalties Modification:                                  │
│    • Frequency Penalty (Weight reduction for repeated tokens)│
│    • Presence Penalty (One-time penalty for already mentioned words)│
├─────────────────────────────────────────────────────────────┤
│ 3. Temperature Scaling: z'_i = z_i / T                      │
│    T ➔ 0: peaks sharpen (Argmax determinism)               │
│    T > 1: distribution smooths (high entropy)               │
├─────────────────────────────────────────────────────────────┤
│ 4. Softmax Normalization: P(w_i) = exp(z'_i) / Sum(exp(z'_j))│
├─────────────────────────────────────────────────────────────┤
│ 5. Truncation Filters (Top-K ➔ Top-P ➔ Min-P)                │
│    Truncating the long "tail" of low-probability junk tokens │
├─────────────────────────────────────────────────────────────┤
│ 6. Stochastic Multinomial Draw (Final token selection)      │
└─────────────────────────────────────────────────────────────┘

1. Temperature ($T$):
   • A scaling divisor for the logits vector before the Softmax function. As $T \to 0$, the difference between the most probable token and others approaches infinity (determinism). At high temperatures ($T = 1.0–1.5$), low-probability tokens gain a real chance of being selected.
2. Top-P (Nucleus Sampling):
   • A dynamic threshold of cumulative probability. The algorithm sorts tokens in descending order and retains only the minimal set whose cumulative probability reaches $P$ (e.g., 0.9 = 90% of the distribution mass).
3. Min-P (Dynamic Thresholding):
   • A modern advanced filter. It truncates all tokens whose individual probability is less than $P_{\min} = \text{Min-P} \times P_{\max}$, where $P_{\max}$ is the probability of the absolute leader. If the leader has 80% and Min-P = 0.05, all options with less than 4% chance are truncated.
4. Repetition Penalties (Presence & Frequency Penalties):
   • Reduce the logits of tokens that have already appeared in the generated text, preventing looping.

3. Technical Pipeline & Internal Mechanics

The lifecycle of a single probabilistic token selection step:

1. Logits Retrieval: The last layer of the network outputs a raw logits vector $z = [12.4, 8.1, 15.6, -3.2, \dots]$.
2. Application of Repetition Penalties: If a word has appeared three times, its logit is reduced by $3 \times \text{frequency_penalty}$.
3. Temperature Division: Each value in the vector is divided by the temperature: $z'_i = z_i / T$.
4. Probability Calculation via Softmax: Logits are converted into normalized probabilities ranging from 0 to 1, summing strictly to 1.0.
5. Filtering via Min-P: The highest score $P_{\max} = 0.70$ is found. With $\text{Min-P} = 0.1$, the threshold is $0.07$. All tokens with probabilities below 7% are removed from the pool.
6. Stochastic Draw (Multinomial Sampling): A pseudorandom number generator (PRNG) samples from the retained pool of probabilities. The selected token is passed to the output.

4. Production Engineering Scenarios

01. Error-Free SQL Query Generation and Validation Schemas

A backend service generates migration code based on user prompts:

• Settings: temperature: 0.0 (Greedy Search).
• The model reliably selects the most standard syntactical constructs, excluding random comma errors or nonexistent keywords.

02. Using Min-P for High-Quality Refactoring

An agent refactors a complex algorithmic module:

• Settings: temperature: 0.7, min_p: 0.05.
• The model gains creative freedom in selecting architectural patterns, but the Min-P mechanism reliably protects against choosing random hallucinated functions.

03. Research Brainstorming for Architecture (High Entropy Search)

Generating non-trivial business ideas and testing scenarios:

• Settings: temperature: 0.9, top_p: 0.95.
• The model proposes unexpected edge cases, unconventional load scenarios, and alternative failure hypotheses.

5. Pitfalls, Common Mistakes & Security

• Repetition penalties break code syntax: A high frequency_penalty (e.g., > 0.5) in programming tasks leads to disaster: the model tries to avoid natural repetitions of keywords (return, const, closing braces }}), which breaks compilation. For code, penalties should be set to 0.
• Fixed temperature in new reasoning models: In models like OpenAI o1 or DeepSeek-R1, the internal thought chain was trained under a strictly fixed temperature (usually $T=1.0$ or $0.6$). Attempting to forcibly set $T=0$ via API is often prohibited or disrupts the model's self-correction process.
• Conflict of overly aggressive filters: Simultaneously setting top_p: 0.1 and a high min_p can reduce the sample to a single or zero token, causing inference engine failure.
• Illusion of reproducibility through Seed: The seed parameter fixes the initial state of the random number generator but does not guarantee identical text when changing the version of the CUDA driver or the number of parallel threads in a batch.