Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Measure recall and latency

Prev Next

Applies to VAST Cluster 5.5.

This guide covers what you can measure from the application side when vector search is slow or returns the wrong neighbours: where query time goes, how to measure recall, how to tune the probe settings against it, and what to send support. Cluster-wide monitoring is a platform-team task; see performance and sizing.

Run the code blocks in order in one Python session. Below each block is the output you should see.

Check the index first

A latency or recall number means little while the index is still catching up with a load: until the cluster has indexed new rows, searches compare the query with each of them, and they're slower (5.5.0 release notes). Before you measure, check how far the index has got; performance shows how.

Then check that your search uses the index at all. Run EXPLAIN on it: a search that uses the index has a VAST_VEC_INDEX_SCAN step in its plan; one that scans the table has only a VAST_SCAN (query design). For a whole load test, your platform team can read the cluster's TopN counters: a search that compares about as many rows as the table holds isn't using the index (what your platform team can see).

Set up an example

The rest of this guide uses two tables with the same 200 rows: one with a vector index, and one without, to compare against.

import os
import re
import time
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/vs_guide_observe"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def sql(query):
    with conn.cursor() as cur:
        cur.execute(query)
        return cur.fetchall()

def vec(i):
    return [round(((i * 37 + k * 11) % 101) / 100, 2) for k in range(8)]

values = ", ".join(f"({i}, {vec(i)}::FLOAT[8])" for i in range(200))
sql(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
sql(f"CREATE TABLE IF NOT EXISTS {SCHEMA}.indexed (id BIGINT, embedding VECTOR(FLOAT, 8)) "
    "VECTOR CLUSTER BY embedding (metric = 'l2sq')")
sql(f"CREATE TABLE IF NOT EXISTS {SCHEMA}.exact (id BIGINT, embedding VECTOR(FLOAT, 8))")
for name in ("indexed", "exact"):
    sql(f"INSERT INTO {SCHEMA}.{name} VALUES {values}")
print(sql(f"SELECT count(*) FROM {SCHEMA}.indexed"), sql(f"SELECT count(*) FROM {SCHEMA}.exact"))

[(200,)] [(200,)]

Time the query, not the pipeline

A search request in an application has four parts, timed separately: embed the query with your model; connect, which is often the biggest cost in a short request, so reuse the connection; execute the query (execute_s); and fetch the results (fetch_s). Execute returns as soon as the query is accepted, and the search runs while you fetch, so most of the time shows in fetch_s. To check whether you return too much, compare fetch_s with and without the embedding column and any payload you don't need.

Time the four parts of a search request separately.

A search request in an application usually has four parts: computing the query embedding, connecting, running the query, and fetching the results. Time them separately, so you know which one to work on. Reuse connections: connecting is often the biggest cost in a short request.

def search(table, q, k=10):
    timings = {}
    t0 = time.perf_counter()
    with conn.cursor() as cur:
        cur.execute(f"SELECT id, array_distance(embedding::FLOAT[8], {q}::FLOAT[8]) AS d "
                    f"FROM {SCHEMA}.{table} ORDER BY d LIMIT {k}")
        t1 = time.perf_counter()
        result = cur.fetch_arrow_table()
        t2 = time.perf_counter()
    timings["execute_s"] = t1 - t0
    timings["fetch_s"] = t2 - t1
    return result, timings

result, timings = search("indexed", vec(7))
print(result.num_rows, sorted(timings))

10 ['execute_s', 'fetch_s']

With the VAST ADBC driver, execute() returns as soon as the query is accepted, and the search runs while you fetch, so most of the time shows in fetch_s, even for a small result. To check whether you're returning too much, compare fetch_s with and without the embedding column and any payload you don't need.

Before you change any search settings, record the values you start from; see tune the probes.

Measure recall

Recall@K is the share of the true K nearest neighbours that the indexed search returns. Measure it against an exact search over the same rows:

  • Use the same query vectors, metric, K and ID column for both.
  • Compare IDs, not distances. Floating-point distances can differ in the last digits.
  • Use a set of query vectors that look like your real queries, not one.
  • For the exact side, use a table without a vector index (as here), or a none / none quantization copy.
def ids(table, q, k):
    return {row[0] for row in sql(
        f"SELECT id, array_distance(embedding::FLOAT[8], {q}::FLOAT[8]) AS d "
        f"FROM {SCHEMA}.{table} ORDER BY d LIMIT {k}")}

K = 10
queries = [vec(i) for i in (3, 50, 120, 199)]
recalls = [len(ids("indexed", q, K) & ids("exact", q, K)) / K for q in queries]
print(f"Recall@{K}: {sum(recalls) / len(recalls):.2f}")

Recall@10: 1.00

On a table this small the search sees every row, so recall is perfect. On real data, run the same comparison on a sample of your table and your queries, and repeat it after each change to quantization or search settings.

Tune the probes

The session settings vector_search_min_prob, vector_search_max_prob and vector_search_pruning_distance_ratio decide how many clusters a search reads at each level of the index. More clusters means better recall and slower searches. Query design shows how to set them on a connection, and the configuration reference describes each one.

  • Record the values you start from. The cluster's defaults can't be read with SQL. Ask your platform team for the values on your cluster, and write them down before you change anything, so you can go back.
  • Measure recall first, against an exact baseline, as in the previous section.
  • Pin the minimum and the maximum to the same values for a repeatable trial: the search then reads exactly that many clusters at each level.
  • Start with the first value in each list. It's for the lowest level of the index, and it has the most effect on recall (configuration).
  • Change one setting at a time, and measure recall and latency again after each change.
  • Don't ship a very large minimum. A very large minimum on every level approaches an exact search. That's a way to check recall in a test, not a production setting.

Compare quantization

To see how much quantization costs you in recall, compare the table with a copy that uses none / none, loaded with the same data. Quantization describes the steps.

Give support the query ID

When you report a slow or failing query, include its query ID. After execute(), read it from the ADBC statement with the option vast.db.last_query_id:

with conn.cursor() as cur:
    cur.execute(f"SELECT count(*) FROM {SCHEMA}.indexed")
    cur.fetchall()
    query_id = cur.adbc_statement.get_option("vast.db.last_query_id")
print(re.sub(r"0x[0-9a-f]+|\d+", "N", query_id))

QueryId(req_id = N, vip_id = N)

The example masks the numbers; log the real value. The option is on the statement (cursor), not the connection.

Clean up

for name in ("indexed", "exact"):
    sql(f"DROP TABLE IF EXISTS {SCHEMA}.{name}")
sql(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()
print("cleaned up")

cleaned up

Next steps

Related articles