Vector Search architecture

Applies to VAST Cluster 5.5.

This page explains how a vector-indexed table maps onto the VAST cluster: where the index lives, how it's built in the background, and what happens when you search. It describes the mechanism, so you can reason about speed and recall. For what you choose when you create an index, see the vector index.

Design goals

Goal	What it means	How the index meets it
Scale to trillions of vectors	The index can't depend on fitting in memory.	The index is a hierarchy of clusters of quantized vectors, kept on the cluster's flash storage. A search reads only the clusters it opens.
Fast searches	Search latency close to the time a model takes to run.	Each level of the hierarchy rules out most of the space, and searches run on the CNodes behind the address you connect to.
New vectors searchable at once	Data keeps arriving.	New rows can be searched as soon as they commit. Indexing runs in the background and doesn't hold up inserts.
Permissions that match the data	Access to vectors can't drift from access to what they describe.	Searches run inside VAST DataBase, so identity policies and row and column security apply to every result.

How it maps onto the cluster

Every CNode reads the same table and index from shared storage; nothing is sharded.

A vector-indexed table is stored like any other VAST DataBase table. Next to the table's rows, the cluster keeps internal structures for the index: the cluster levels, the quantized vectors, and the recent rows that aren't indexed yet. You don't see or query these structures directly. For how VAST DataBase stores tables in general, see how tables are stored.

Searches run in the VAST Query Engine, which runs inside the cluster's CNodes (see VAST Query Engine). Your client opens an ADBC connection to the Query Engine's virtual IP pool, which is dedicated to the Query Engine. Like everything on the cluster, it's built on the DASE architecture: the CNodes hold no data of their own and read the same shared storage, so the index and its table aren't split into shards owned by particular servers.

The hierarchy

The index is a tree of clusters. Each cluster holds up to 1,000 members of the level below, so each level holds about 1,000 times as many vectors as the one below it:

Level	Holds	Vectors under one cluster
L1	up to 1,000 vectors	1,000
L2	up to 1,000 L1 clusters	about a million
L3	up to 1,000 L2 clusters	about a billion
L4	up to 1,000 L3 clusters	about a trillion
L5	the root, where every search starts

The index keeps quantized copies of the vectors. The full-precision vectors (FLOAT or FLOAT16) stay in the table, and a search reads them only at the end, for the candidates it has picked.

Each cluster above L1 has a centre (a centroid), stored quantized. A search compares the query vector with centroids, level by level, and only opens the clusters whose centres are nearest.

New rows don't join the tree straight away. They wait with the other recent rows until the background indexing places them. Until then, every search reads the recent rows directly and computes their distances one by one, alongside its walk of the tree. So recent rows are always found, but a large backlog of them makes every search do more work. After a big load, give indexing time to catch up before you measure search speed. See ingest.

Quantization in the walk

cluster_quantization is the encoding the index uses when it builds and stores its clusters, and search_quantization the encoding a search uses for distances while it walks them. Both are set when you create the table; quantization has the allowed values and pairs, and how to choose.

What happens when you search

A search opens only the nearest clusters at each level, and reads recent rows directly.

Your ADBC statement reaches the Query Engine.
The planner checks the query's shape. It uses the index only for ORDER BY the distance function that matches the index's metric, ascending, with a LIMIT and no OFFSET, on a vector-indexed table. Any other shape runs as an ordinary scan. EXPLAIN shows the decision: an index search plans a VAST_VEC_INDEX_SCAN step, an ordinary scan only a VAST_SCAN. See rules for a search that uses the index.
The search walks the tree level by level, from the root. At each level it compares the query vector with the centres of the clusters below and opens the nearest ones: at least a minimum and at most a maximum number for that level. Between the two, it opens the next-nearest cluster only while that cluster is less than a set ratio as far away as the nearest one. So a query whose nearest clusters are clearly closest opens few, and an ambiguous one opens more. By default the budget is 4 to 8 clusters at L4 and L3, 30 to 200 at L2 and 150 to 3,000 at L1, with a ratio of 1.9. Three session settings change it: vector_search_min_prob, vector_search_max_prob and vector_search_pruning_distance_ratio; see search session settings.
In the L1 clusters it opens, it compares the query vector with every quantized vector and keeps more candidates than the LIMIT asks for. Every comparison during the walk uses quantized vectors, in the table's search_quantization.
It reads the full-precision vectors of those candidates once, computes their exact distances, and merges in the recent rows that aren't indexed yet, which it reads directly. The k nearest are returned with the columns you asked for.
SQL filters are applied during the walk, not afterwards, so the k rows you get all pass them. If a filter rejects many rows, the search keeps opening L1 clusters until enough rows have passed it to fill the LIMIT. See filters.
The results stream back to your client as Arrow data.

Background work

Two kinds of background work keep the index in shape:

Initial indexing. Once enough recent rows have built up, a k-means pass turns a batch of them into new L1 clusters of about 1,000 vectors each, and places them in the L2 cluster that's currently open. An L2 cluster closes when it holds about 1,000 L1 clusters, about a million vectors. Higher levels are built the same way, from the centres below them.
Reclustering. Clusters that are poorly organized, or emptied by deletes, are rebuilt. For an L2 cluster, all its vectors are redistributed into new L1 clusters; at higher levels, the centres below are redistributed. Reclustering runs k-means on a sample of the vectors, keeps the new clusters evenly sized, assigns every vector to its nearest new centre, and removes deleted rows.

Reclustering re-places the same vectors into compact, evenly sized clusters.

Both run on the cluster without any action from you, and searches keep running throughout: only the cluster boundaries move. The cluster shares the work fairly between buckets, so one busy table can't hold up the others. Clustering has its own memory budget on the CNodes, but it uses their CPU, so a large load can slow searches until it's done.

The index is a data structure, not a trained model. Nothing needs retraining when your data changes: clusters that no longer fit are reclustered. The cluster tracks each table's progress as its clustering grade (A+, A, B or C), which can jump straight from C to A+ during initial indexing. If it stays low long after a load, contact VAST Support. See developer observability and performance and sizing.

You can't replicate a vector-indexed table on VAST Cluster 5.5; see limits.