Vector Databases (Vector DBs & ANN Search)

1. Concept Overview & Systemic Problem

With the emergence of embeddings, engineers faced a new challenge: how to store and quickly search data in vector space, where the dimensionality of each record consists of hundreds or thousands of coordinates:

1. Inadequacy of Classical B-Tree Indexes: Classical B-Tree indexes effectively sort one-dimensional numbers or strings but are mathematically powerless against 1536-dimensional geometric vectors (the "Curse of Dimensionality").
2. Massive Memory Requirements: Storing arrays of float32 numbers requires gigabytes of RAM and specialized hardware acceleration (AVX-512 instructions, SIMD, or CUDA).
3. Need for Metadata Coupling: Finding a vector is insufficient—the system must instantly return associated text, author, date, source URL, and user access level.

Vector Databases have transformed vector semantic search into a reliable infrastructural primitive capable of finding the most similar entities among billions of records in mere milliseconds.

2. Architectural Taxonomy & Mental Model

In the realm of vector storage, two conceptual models dominate the organization of data and their corresponding index structures:

• 1. HNSW Index (Hierarchical Navigable Small World): The gold standard for vector search. It builds a multi-layered graph similar to the Skip-List algorithm: the upper layers contain long links for quick jumps to the required space cluster, while the bottom layer (Layer 0) performs detailed navigation among nearest neighbors. It provides the best balance between speed and search completeness (Recall > 98%).
• 2. IVF Index (Inverted File Index): The space is divided into Voronoi cells through K-Means clustering. The query first identifies a few nearest centroids, then scans only the vectors within those clusters. It requires less RAM than HNSW but has lower accuracy.
• 3. Architectural Formats of Databases:
  • Dedicated Vector DBs: Qdrant (Rust), Milvus (Go/C++), Chroma, Pinecone. Optimized for scale, sharding, and parallel GPU computations.
  • Integrated Vector Extensions: pgvector for PostgreSQL, sqlite-vec / Turso for SQLite. Provide ACID transaction support and familiar SQL syntax.
• 4. Quantization and Memory Compression: Utilizing Scalar Quantization (SQ) or Product Quantization (PQ) to compress vectors in memory to 8-bit or 1-bit representations.

3. Technical Pipeline & Internal Mechanics

The lifecycle of storage and search in a vector database:

1. Ingestion & Payload Attachment: The client submits a vector along with JSON metadata (text, document_id, created_at, tenant_id).
2. Graph Insertion & Edge Linking: The algorithm finds nearest neighbors for the new vector at each layer of the graph and creates bidirectional edges considering the vertex degree limit $M$.
3. Query Ingestion & Multi-layer Traversal: Upon receiving a query vector, the algorithm begins a greedy search from the top layer, descending to lower levels as the cluster is localized.
4. Single-Stage Filtered Retrieval: If the query contains SQL/JSON filters, metadata matching occurs directly during graph traversal (Filtered HNSW), ensuring the return of strictly relevant records.

4. Production Engineering Scenarios

01. Agent Memory Storage

An autonomous agent stores facts about the developer in a Qdrant or pgvector collection: [vector, payload: { user_id: 104, fact: "prefers bun over npm" }]. Before starting a session, the agent retrieves the 5 most relevant facts.

02. Production RAG for Technical Support

The vector database stores 500,000 chunks of documentation. A client query finds 20 most relevant instruction fragments in 12 milliseconds, which are then passed to a reranker.

03. Semantic E-Commerce Catalog with Faceted Filtering

Searching for clothing with the query: “light running jacket for fall” with mandatory pre-filter price <= 3000 AND in_stock = true AND size = 'L'.

5. Pitfalls, Common Mistakes & Security

• Dimension Mismatch Error: Attempting to search with an OpenAI model vector (1536 dimensions) in an index created for a Cohere model (1024 dimensions) results in a fatal runtime error in the database.
• HNSW Build RAM Spike: Building an HNSW index over 5 million vectors requires 2–3 times more RAM during construction than for final storage. Build indexes with RAM buffer considerations or use external disk quantization.
• Neglecting Vacuuming and Defragmentation: Frequent UPDATE and DELETE operations create orphaned empty nodes in vector graphs. Regularly run index optimization (Vacuum / Segment Compaction).