Generation Speed (TPS / TTFT / Latency)

1. Concept Overview & Systemic Problem

In engineering discussions, model performance is often reduced to vague definitions of "fast" or "slow." This amateur approach leads to failures in real-time systems: voice bot interfaces lag for several seconds before responding, code autocompletion in IDEs frustrates with delayed character appearances, and background agent cycles get stuck in multi-hour queues.

The actual inference of large language models adheres to the laws of semiconductor physics and is divided into two distinct hardware phases:

1. Prefill Phase: Parallel processing of the input prompt.
2. Decode Phase: Sequential generation of output symbols token by token.

Metrics TTFT (Time to First Token) and TPS (Tokens Per Second) serve as the systemic compass for engineers, determining queue architecture, hardware selection, and user experience in interactive applications.

2. Architectural Taxonomy & Mental Model

Latency and throughput metrics are classified according to the stages of the request lifecycle:

┌─────────────────────────────────────────────────────────────┐
│                 INFERENCE LATENCY TAXONOMY                  │
├─────────────────────────────────────────────────────────────┤
│ 1. Queue Time (Waiting time for free GPU resources)         │
├─────────────────────────────────────────────────────────────┤
│ 2. Prefill Phase ➔ TTFT (Time to First Token)               │
│    • Compute-Bound (TFLOPS Tensor Cores)                    │
│    • Direct computation of KV Cache for the entire input text│
├─────────────────────────────────────────────────────────────┤
│ 3. Decode Phase ➔ TPS / ITL (Inter-Token Latency)           │
│    • Memory-Bandwidth Bound (GB/s VRAM bus)                  │
│    • Sequential reading of weights for each token            │
├─────────────────────────────────────────────────────────────┤
│ 4. Total E2E Latency = TTFT + (N_generated_tokens * ITL)    │
└─────────────────────────────────────────────────────────────┘

1. TTFT (Time to First Token):
   • The number of milliseconds from the moment the HTTP request is sent by the client to the receipt of the first byte of the streaming response. It consists of network latency, queue waiting time, and prefill computation time.
2. TPS (Tokens Per Second):
   • The number of tokens the system outputs in one second. For a single stream, it is $1 / \text{ITL}$. For the entire server (Aggregate Throughput), it is the total number of tokens generated by all clients simultaneously.
3. ITL (Inter-Token Latency):
   • The time between the appearance of adjacent tokens during streaming. A comfortable threshold for the human eye is less than 30 ms per token (equating to >33 TPS).
4. Speculative Decoding:
   • An algorithmic trick that transforms part of the Decode phase operations into parallel verification, increasing net TPS by 2–3 times without sacrificing accuracy.

3. Technical Pipeline & Internal Mechanics

The lifecycle of inference under the microscope of timings:

1. Request Reception and Batching: A request with 4000 tokens of code enters the inference engine (e.g., vLLM).
2. Prefill Phase: The GPU loads weight matrices and simultaneously multiplies all 4000 tokens on Tensor Cores. Vectors Q, K, V are computed, and KV cache memory is initialized.
3. First Token Emission (TTFT Fixation): The first token is sent to the client via Server-Sent Events. If the prompt is cached (Prompt Caching), this stage takes 50 ms; if not, it ranges from 500 ms to 3 seconds.
4. Sequential Decode Loop:
   • To generate one token, the GPU must read all 70 billion model parameters from VRAM into the chip cache.
   • At a memory bandwidth of 1000 GB/s (RTX 4090), reading 35 GB of the quantized model takes: $$35 \text{ GB} / 1000 \text{ GB/s} = 35 \text{ ms per token} \approx 28 \text{ TPS}$$
5. Request Completion: Generation ends upon reaching the token <|im_end|> or the length limit, and KV cache resources are freed for other requests.

4. Production Engineering Scenarios

01. Developing a Voice Assistant with Human-Like Latency

A human notices a pause in conversation if the response is delayed beyond 500 ms:

• Engineers choose the Gemini 2.0 Flash or Mistral NeMo model with local deployment.
• By optimizing the TTFT stack to 180 ms, text-to-speech (TTS) takes an additional 120 ms. A total response time of 300 ms creates the effect of a live dialogue.

02. High-Speed Autocompletion in IDE (Ghost Text)

A developer types code in Cursor at a speed of 5 characters per second:

• Any delay exceeding 150 ms results in the suggestion appearing only after the developer has typed a new character.
• For such tasks, lightweight models (3B–7B parameters) or speculative engines that emit the first characters in 40–80 ms are utilized.

03. Batch Processing Large Arrays of Documents in the Backend

Nightly processing of 100,000 user PDF files:

• Individual TTFT for each file is irrelevant.
• The server is optimized for Aggregate Throughput: through continuous batching in vLLM, the cluster simultaneously handles 256 streams, achieving a total speed of 4,000 tokens per second.

5. Pitfalls, Common Mistakes & Security

• TPS Drop During KV Cache Exhaustion: When video memory fills to 95%, the engine begins offloading the cache to SSD or CPU. The generation speed drops drastically, paralyzing the system.
• Blind Pursuit of TPS Without Quality Control: A model with high speed (200 tokens/s) that generates non-functional code with syntax errors only increases development time for manual fixes.
• Illusion of Speed Due to Network Buffering: If a proxy server (Nginx or Cloudflare) buffers the response instead of streaming it (missing header X-Accel-Buffering: no), the client will not see any tokens until the model generates the entire text completely.
• Impact of Context Length on TTFT: Increasing the input prompt from 2,000 to 100,000 tokens disproportionately raises prefill time without caching enabled, turning a "fast" model into a slow one.