Next-Token Prediction

1. Concept Overview & Systemic Problem

When observing AI design a microservices architecture or compose a poem, it’s easy to believe that consciousness with its own internal monologue has emerged within the computer.

However, at the computational level, any autoregressive LLM (GPT-4o, Claude 3.7, Llama 3.3, DeepSeek R1) executes a single iterative cycle:

$$P(W) = \prod_{t=1}^{T} P(w_t \mid w_1, w_2, \dots, w_{t-1})$$

1. Read the input token sequence $w_1, \dots, w_{t-1}$.
2. Pass the vectors through Transformer layers (Self-Attention + MLP).
3. Calculate the numeric score (Logit) for each of the 128,000 tokens in the vocabulary.
4. Apply Softmax and sampling algorithms (Temperature, Top-P), selecting one token $w_t$.
5. Add $w_t$ to the context and repeat the step.

Mental model: this is not an oracle that knows the final thought in advance, but a statistical trajectory generator that takes a step and only then sees where to step next.

┌─────────────────────────────────────────────────────────────┐
│              MATHEMATICAL CONVEYOR OF ONE STEP            │
├─────────────────────────────────────────────────────────────┤
│ 1. Input context: "Kyiv is the capital"                    │
│    Token vectorization ➔ Pass through transformer layers    │
├─────────────────────────────────────────────────────────────┤
│ 2. Output layer (Unembedding): 128,000 logits               │
│    • Token "Ukraine"   ➔ Logit: +14.2                       │
│    • Token "ancient" ➔ Logit: +8.1                          │
│    • Token "France"   ➔ Logit: -4.5                         │
├─────────────────────────────────────────────────────────────┤
│ 3. Softmax normalization (Logits / Temperature):            │
│    • "Ukraine"   ➔ 94.2%                                    │
│    • "ancient" ➔ 5.1%                                      │
│    • "France"   ➔ 0.0001%                                  │
├─────────────────────────────────────────────────────────────┤
│ 4. Sampling: selecting token "Ukraine"                      │
│    New context: "Kyiv is the capital of Ukraine" ➔ Next step│
└─────────────────────────────────────────────────────────────┘

2. Sampling Implementation in Python and Inspection via CLI

Here’s how the calculation of the next token probabilities looks in Python, considering temperature:

import numpy as np

def compute_next_token_probs(logits: np.ndarray, temperature: float = 0.7) -> np.ndarray:
    # 1. Scale by temperature
    scaled = logits / max(temperature, 1e-4)
    # 2. Stable Softmax (subtracting max to prevent overflow)
    exp_logits = np.exp(scaled - np.max(scaled))
    probs = exp_logits / np.sum(exp_logits)
    return probs

# Example call: 4 possible tokens
mock_logits = np.array([12.5, 9.1, 4.0, 1.2])
print("Token probabilities:", np.round(compute_next_token_probs(mock_logits, temperature=0.7), 4))

How to View Raw Token Probabilities via cURL in Terminal:

You can request the API to return exact log probabilities (logprobs) to see the alternatives the model considered:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "The capital of France is"}],
    "max_tokens": 1,
    "logprobs": true,
    "top_logprobs": 3
  }'

In the JSON response, the top_logprobs parameter will show the exact mathematical scores for options like Paris, city, etc.

3. Why Next-Token Leads to Hallucinations and How to Mitigate It

As the model generates text locally "here and now," it is prone to statistical traps:

1. Early Commitment Error: If the model randomly selects a poor token in the first two words, all subsequent tokens will attempt to justify this erroneous choice instead of acknowledging the mistake.
2. No Backtracking: Standard inference cannot erase an already generated word.
3. Why Reasoning Models (Reasoning Models / DeepSeek R1 / o3) Perform Better: They utilize chains of thought (<thought>...</thought>), generating hundreds of internal analysis and self-checking tokens before providing the final answer.

4. Production Engineering Scenarios

01. Debugging Token Sequences

Utilize logging to capture the logits and probabilities at each generation step, allowing for fine-tuning and error analysis.

02. Implementing Temperature Control

Experiment with different temperature settings in production to balance creativity and coherence in generated outputs.

03. Enhancing Prompt Engineering

Design prompts that encourage the model to generate reasoning tokens, improving the quality of the final output by providing better context.

5. Pitfalls, Common Mistakes & Security

• Over-reliance on Initial Tokens: Avoid designing prompts that lead to early commitment errors; ensure diversity in initial token selection.
• Ignoring Logits Analysis: Regularly inspect logits and probabilities to understand model behavior and prevent hallucinations.
• Security Risks with API Exposure: Ensure that API keys are securely managed and not exposed in client-side code to prevent unauthorized access.