An inverted index reverses the usual relationship between records and their contents. Instead of reading each document to discover its words, a search engine maintains a dictionary that maps each searchable term to a posting list of matching document identifiers. For example, the term “timeout” might point to log records 12, 47, and 93. A query can retrieve those identifiers directly rather than scan every log record. Posting lists can also store term frequencies and positions to support relevance scoring and phrase matching.
For cloud data-store selection, distinguish a traditional text inverted index from an inverted file vector index, commonly called IVF. They share the principle of organizing candidates into searchable lists, but their keys and matching behavior differ. Text indexes use terms; IVF uses partitions of an embedding space, commonly represented by trained cluster centroids. A text inverted index does not automatically provide semantic similarity search, and an IVF index does not perform keyword matching merely because its name contains “inverted.”
To build a typical IVF index, the engine trains centroids using representative vectors and assigns each stored vector to a partition associated with a nearby centroid. At query time, it identifies nearby centroids, searches the corresponding inverted lists, and ranks candidate vectors by the configured distance or similarity measure. For example, a product-search application might partition millions of product embeddings into thousands of lists and probe only a small subset for each query.
The number of partitions and the number searched per query control important trade-offs. More partitions generally produce smaller candidate lists, but increase centroid-selection work and training requirements. Probing more lists generally improves recall while increasing latency and computation. Searching too few lists can miss a true nearest neighbor located across a partition boundary. Parameter names vary: implementations commonly expose settings such as nlist and nprobe, while pgvector uses lists and probes.
IVFFlat stores uncompressed vectors within the partitioned structure and evaluates candidate distances using those vectors. IVFPQ combines IVF partitioning with product quantization to reduce vector storage and distance-computation cost, at the expense of additional approximation. Neither partitioning nor quantization should be confused with the embedding model itself: the model creates vectors, while the index organizes them for retrieval.
HNSW, or Hierarchical Navigable Small World, is a different approximate nearest-neighbor index family. It navigates a multilayer proximity graph rather than searching centroid-based inverted lists. HNSW commonly offers strong recall and latency but can require substantial graph memory and build effort. IVF typically requires representative training data, whereas HNSW does not require centroid training. Actual performance depends on the implementation, dataset, hardware, filters, and query workload.
In managed cloud search and database systems, engineers must also account for index storage, ingestion overhead, refresh visibility, replication, and maintenance. Changing embedding models or substantial shifts in the data distribution may require rebuilding or retraining a vector index. Validate approximate-search recall against an exact-search baseline, test filtered queries with realistic tenant boundaries, and measure latency under concurrent ingestion. Choose text indexing for lexical retrieval, vector indexing for embedding similarity, and hybrid retrieval when both exact terminology and semantic meaning matter.
Amazon OpenSearch Service