Fast NVMe Scratch Volumes for AI Models

1. Concept Overview & Systemic Problem

When working with large language models and vector databases, the disk subsystem often becomes an unnoticed bottleneck:

• The server has a powerful CPU and fast GPU, but each switch or restart of the model (e.g., switching from a coding model to a reasoning model) causes the system to hang for 3 minutes waiting for disk reads.
• A vector database (Qdrant / Milvus) begins reading index files from disk when it exceeds memory: if the disk is slow, search times jump from 5 milliseconds to 800 milliseconds.

Fast NVMe Scratch Volumes represent an infrastructure pattern for proper storage separation: the operating system and backups reside on reliable disks, while all "hot" model weights and temporary caches are offloaded to a dedicated ultra-fast NVMe drive.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 TIERED SERVER STORAGE TOPOLOGY              │
├─────────────────────────────────────────────────────────────┤
│ 1. ROOT OS DISK (/dev/sda - Standard SSD / RAID1):          │
│    • Ubuntu OS, System Services, Docker Daemon              │
│    • High reliability & automated cloud snapshots           │
├─────────────────────────────────────────────────────────────┤
│ 2. DEDICATED NVME SCRATCH DISK (/dev/nvme0n1 mounted at /mnt/scratch):
│    • Direct PCIe 5.0 x4 connection (up to 14 GB/s read)     │
│    • Directory: `/mnt/scratch/huggingface` (LLM Models Cache)│
│    • Directory: `/mnt/scratch/vector_indexes` (Qdrant mmap) │
│    • Directory: `/mnt/scratch/docker_build_cache`           │
│    ➔ Model Llama-70B loads into VRAM in ~4 seconds!         │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Fast Cache Configuration for Ollama / Hugging Face

Redirecting model loading paths to the dedicated NVMe disk:

export HF_HOME=/mnt/scratch/hf_cache
export OLLAMA_MODELS=/mnt/scratch/ollama_models

The command ollama run deepseek-r1:14b starts instantly, as binary layers are read through the Linux kernel at maximum PCIe bus throughput.

02. Using mmap in Vector Databases (Memory-Mapped Files)

The vector database mounts a 100GB index from the NVMe disk. Thanks to the high IOPS rate, the database accesses vectors on disk almost as quickly as it does RAM, saving thousands of dollars on server RAM.

4. Production Engineering Scenarios

01. Data Ephemerality on Scratch Disks

Some cloud providers (AWS EC2 Instance Store) wipe the local NVMe disk upon each instance stop (Stop/Start). Never store the primary PostgreSQL database there—use it strictly for cached artifacts that can be automatically reloaded.

02. NVMe Thermal Throttling

When continuously reading terabytes of models, the NVMe drive may heat up above 75°C and automatically throttle speed by three times. Ensure the server in the data center has quality radiator cooling.

5. Pitfalls, Common Mistakes & Security

Proper disk configuration is foundational for high responsiveness in local agent platforms. Investing in fast local NVMe drives removes disk I/O bottlenecks, enabling instant loading of modern language models.