KV-Cache Offloading & Compression

1. Concept Overview & Systemic Problem

With the emergence of models featuring context windows of 1–2 million tokens (Gemini 2.0, Claude 3.5/3.7), engineers face a new infrastructural crisis:

• A 70B model weighs a static 38 GB of VRAM.
• However, if 4 users open sessions with long code contexts of 200,000 tokens, their total KV-cache size exceeds 60 Gigabytes!
• When one user pauses for 5 minutes to think, their gigabyte cache continues to unnecessarily block VRAM, preventing other users from sending requests.

KV-Cache Offloading & Compression implements dynamic two-tier or three-tier inference memory management: hot cache remains in VRAM, warm cache resides in server RAM, and cold cache is offloaded to NVMe storage.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 KV-CACHE HIERARCHY & TIERS                  │
├─────────────────────────────────────────────────────────────┤
│ TIER 1: HOT VRAM (HBM3 on GPU)                              │
│ • Latency: < 100 ns | Bandwidth: 2–3 TB/s                   │
│ • Stores active tokens for the current generation cycle      │
├─────────────────────────────────────────────────────────────┤
│                          ▲                                  │
│                 Offload  │  Prefetch                        │
│                          ▼                                  │
├─────────────────────────────────────────────────────────────┤
│ TIER 2: WARM SYSTEM RAM (DDR5 on Host Server)               │
│ • Latency: ~100 µs  | Bandwidth: 100–300 GB/s               │
│ • Stores user sessions waiting in queue                      │
├─────────────────────────────────────────────────────────────┤
│                          ▲                                  │
│                 Swap     │  Restore                         │
│                          ▼                                  │
├─────────────────────────────────────────────────────────────┤
│ TIER 3: COLD NVMe STORAGE (PCIe 5.0 SSD)                    │
│ • Latency: ~10 ms   | Bandwidth: 10–14 GB/s                 │
│ • Long-term storage for massive agent sessions                │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Saving the State of an Agent's Night Session

An agent completes a long task analyzing 150 files. Their context state (15 GB KV-cache) is offloaded to a fast NVMe disk via Litestream/vLLM offload. When an engineer asks a clarifying question in the morning, the cache loads in 1 second without redoing the costly prefill of 150 files.

02. Implementing FP8 KV-Caching in vLLM

Enabling hardware cache compression with a single flag:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90

This allows doubling the number of concurrent sessions without allocating additional GPUs.

4. Production Engineering Scenarios

• PCIe Bottleneck: If the server has slow PCIe lanes (e.g., PCIe 3.0 or x4 lanes instead of x16), the time to offload and load the cache may exceed the time required for recomputation.
• Quantization Noise in Overlong Contexts: When compressing the cache to 4 bits (INT4) in contexts exceeding 100k tokens, error accumulates, leading to a loss of the model's ability to accurately cite information.

5. Pitfalls, Common Mistakes & Security

KV-Cache offloading and compression have transformed massive context windows from an expensive exotic into an accessible engineering tool. Understanding the memory hierarchy enables the design of systems with million-token contexts on moderate servers.