TTFT vs TPS (Inference Latency Metrics)
Two key engineering metrics for inference performance: Time To First Token (latency to response initiation) and Tokens Per Second (throughput of code generation).
1. Concept Overview & Systemic Problem
When evaluating the performance of an AI model, novice developers often look at a single overall speed metric: "It generates quickly." However, in real production, latency is divided into two distinct hardware phases:
- Prefill Phase: The model must "read" an input prompt of 50,000 tokens of code. This is a compute-intensive operation that determines TTFT (Time To First Token).
- Decode Phase: The model generates new code sequentially, token by token. This is a memory-bound operation that determines TPS (Tokens Per Second).
2. Architectural Taxonomy & Mental Model
[ USER SENDS REQUEST ]
│
▼
┌───────────────────────────────────────────────────────────────────────┐
│ 1. PREFILL PHASE (Parallel processing of the entire input prompt) │
│ • GPU FLOPs utilized at 100% │
│ • Time to first character: ➔ [ TTFT = 420 ms ] │
└───────────────────────────────────┬───────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────────┐
│ 2. DECODE PHASE (Sequential generation of tokens) │
│ • Memory Bandwidth utilized at 100% │
│ • Streaming rate: ➔ [ TPS = 85 tokens/sec ] │
└───────────────────────────────────┬───────────────────────────────────┘
│
▼
[ FULL RESPONSE COMPLETED ]
3. Technical Pipeline & Internal Mechanics
01. Optimizing Interactive Assistant in IDE
Developers expect real-time code suggestions (Inline Completion). If TTFT exceeds 300 ms, developers continue typing themselves, rendering the suggestion useless. For this role, models with minimal TTFT (Gemini Flash or local 3B models) are selected.
02. Choosing a Provider for Codebase Refactoring
For the task of writing 500 lines of code in the background, a model with TTFT = 5 seconds and TPS = 120 tokens/sec will complete in 9 seconds, while a model with instant start (TTFT = 0.2s) but slow TPS = 25 tokens/sec will require 20 seconds.
4. Production Engineering Scenarios
01. Misleading Average TPS
Providers often advertise peak TPS for short responses. As the output text lengthens (KV cache grows), TPS can drop by 30–40%.
02. Impact of Queue Latency
If the server is overloaded with requests, high TTFT may be caused not by model slowness but by a request waiting 2 seconds in the scheduler's queue. Always separate Queue Time from pure GPU Prefill Time.
5. Pitfalls, Common Mistakes & Security
TTFT and TPS are the engineering coordinates for model selection architecture. Understanding the difference between Prefill and Decode phases allows for building systems that feel instantaneous to users in the interface while maximizing generation speed for autonomous agents.
FAQ: TTFT vs TPS (Inference Latency Metrics)
Related terms
Generation Speed (TPS / TTFT / Latency)
Key engineering performance metrics for language models: Time to First Token (response time to input context) and Tokens Per Second (streaming output text generation speed).
vLLM (High-Performance Inference Engine)
Leading open-source inference engine and LLM servicing framework that revolutionizes throughput with the PagedAttention memory virtualization algorithm and continuous batching.
Gemini Flash & Pro (Google Gemini)
A family of multimodal models from Google DeepMind that combines a record context window (up to 2 million tokens), extreme generation speed (over 150 tokens/sec), and native perception of video and audio.
Sampling Parameters (Temperature, Top-p, Min-p)
Mathematical hyperparameters of stochastic decoding (Temperature, Top-P, Min-P, Penalties) that govern the probability distribution for selecting the next token, defining the model's level of determinism, accuracy, and creativity.