Quickstart

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

In this quickstart you'll create a table with a vector column and a vector index, insert a few rows, and find the nearest neighbours of a query vector. You'll create the table and search it with SQL through the VAST Query Engine, then do the same with the Python SDK. Finally you'll remove what you created. It takes about 10 minutes once your platform team has set up access; the code itself runs in seconds.

Run each code block in order in the same Python session. Below each block is the output you should see.

Before you start

You need these from your platform team:

Admin task: send to your platform team. See Accessing VAST DataBases using the Query Engine, and the checklist in setup prerequisites. Setup covers what Vector Search needs on top.

You also need Python 3.10 or later on a Linux x86_64 client that can reach the endpoint. The ADBC driver runs on Linux x86_64.

What your platform team sets up for Vector Search, in order, before your ADBC application connects. See setup for each item.

1. Install the drivers

pip install "vastdb[adbc]==2.1.0" adbc-driver-manager

The adbc extra installs the VAST ADBC driver package, adbc-driver-vastdb. It ships the driver library (libadbc_driver_vastdb.so), and adbc_driver_vastdb.get_driver_path() returns its path. adbc-driver-manager is the Python driver manager that loads it.

2. Connect to the Query Engine

Put your connection details in environment variables:

export QE_ENDPOINT="http://query-engine.example.com:9090"   # your Query Engine endpoint, with its port
export AWS_ACCESS_KEY_ID="<access key>"
export AWS_SECRET_ACCESS_KEY="<secret key>"
export VASTDB_BUCKET="<database name>"

These are the section's standard variables. The guides that load data also set AWS_S3_ENDPOINT_URL, the database endpoint; see setup. The quickstart doesn't need it.

Start Python in that shell and connect:

import os
import adbc_driver_vastdb
from adbc_driver_manager import dbapi

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/vs_quickstart"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)
cur = conn.cursor()

cur.execute("SELECT 1")
print(cur.fetchall())

[(1,)]

The endpoint must be a Query Engine endpoint, not a protocols VIP, and it needs the port: a URL on the wrong port fails with Connection refused (os error 111).

Pass autocommit=True. Without it, dbapi.connect() turns autocommit off, and DDL you don't commit is lost when the connection closes: a table you created isn't there on the next connection.

Table names have the form "<bucket>/<schema>".<table>. SCHEMA holds the quoted part.

3. Create a schema and an indexed table

A vector column has the type VECTOR(<element type>, <dimension>). VECTOR CLUSTER BY gives that column a vector index, with the distance metric you choose. Here it's l2sq, which ranks by Euclidean distance:

cur.execute(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
cur.execute(f"""
CREATE TABLE IF NOT EXISTS {SCHEMA}.docs (
    id INTEGER,
    title VARCHAR,
    embedding VECTOR(FLOAT, 8)
) VECTOR CLUSTER BY embedding (metric = 'l2sq')
""")

cur.execute(f"DESCRIBE {SCHEMA}.docs")
for row in cur.fetchall():
    print(row[0], row[1])

id INTEGER
title VARCHAR
embedding VECTOR(FLOAT, 8)

You must give the metric. You can also set the index's quantization, but if you leave it out, both cluster_quantization and search_quantization are scalar_8bit. You can't change the metric or add the index later; see the vector index.

DESCRIBE shows the vector column's type, but not the index.

4. Insert a few rows

cur.execute(f"""
INSERT INTO {SCHEMA}.docs VALUES
    (10, 'a', [1.1, 1.1, 1.1, 1.1, 1.1, 1.1, 1.1, 1.1]::FLOAT[8]),
    (20, 'b', [1.2, 1.2, 1.2, 1.2, 1.2, 1.2, 1.2, 1.2]::FLOAT[8]),
    (30, 'c', [1.3, 1.3, 1.3, 1.3, 1.3, 1.3, 1.3, 1.3]::FLOAT[8]),
    (60, 'd', [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]::FLOAT[8])
""")
print(cur.fetchall())

[(4,)]

The result is the number of rows inserted. Run this block only once, or you'll insert the rows again.

New rows can be found by search straight away. Adding them to the index takes some time, but until then the search reads them directly. With a large load, wait for the cluster to finish indexing before you judge search speed or accuracy. The table's clustering grade, from C up to A+, shows how far indexing has got; see performance and sizing and ingest.

5. Search

A nearest-neighbour search computes the distance to a query vector, sorts by it and keeps the first k rows. This one also filters on id:

cur.execute(f"""
SELECT id, title,
       array_distance(
           embedding::FLOAT[8],
           [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]::FLOAT[8]
       ) AS dist
FROM {SCHEMA}.docs
WHERE id > 15
ORDER BY dist
LIMIT 2
""")
for id_, title, dist in cur.fetchall():
    print(id_, title, round(dist, 4))

20 b 0.5657
30 c 0.8485

Row 10 is the nearest to the query vector, but the filter excludes it. Smaller distances are nearer. array_distance returns the Euclidean distance, for example 0.5657 for row 20: the square root of 8 × 0.2².

For the index to be used, write every search like this:

• Use the distance function that matches the index's metric: array_distance for l2sq. The others are in query design.
• Cast both sides to the column's dimension, here FLOAT[8].
• ORDER BY the distance ascending, and LIMIT k.

6. Do the same with the Python SDK

The SDK creates the same kind of table with VectorIndexSpec, and searches it with table.vector_search(). The search runs as SQL through the Query Engine, so the session needs the ADBC driver and a Query Engine endpoint:

import pyarrow as pa
import vastdb
from vastdb import VectorIndexSpec

session = vastdb.connect(
    endpoint=os.environ["QE_ENDPOINT"],
    adbc_driver_path=adbc_driver_vastdb.get_driver_path(),
)

dim = 8
columns = pa.schema([
    ("id", pa.int32()),
    ("embedding", pa.list_(pa.field("item", pa.float32(), nullable=False), dim)),
])

with session.transaction() as tx:
    s = tx.bucket(BUCKET).create_schema("vs_quickstart", fail_if_exists=False)
    t = s.create_table(
        "docs_sdk",
        columns,
        vector_index=VectorIndexSpec(column="embedding", distance_metric="l2sq"),
        fail_if_exists=False,
    )
    t.insert(pa.table(schema=columns, data=[[10, 20], [[1.1] * dim, [5.0] * dim]]))
    print(t.vector_index)

VectorIndex(column='embedding', distance_metric='l2sq', sql_distance_function='array_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')

vastdb.connect() reads the access keys from the AWS_* variables. A vector column in the SDK is a fixed-size Arrow list of float32 whose items aren't nullable. The table reports its index: the metric, the SQL distance function the SDK will use, and the default quantization.

Now search it, in a new transaction:

with session.transaction() as tx:
    t = tx.bucket(BUCKET).schema("vs_quickstart").table("docs_sdk")
    t.reload_schema()
    result = t.vector_search(vec=[1.0] * dim, columns=["id"], limit=2).read_all()

print(result["id"].to_pylist())
print([round(d, 4) for d in result["vast_pysdk_vector_dist"].to_pylist()])

[10, 20]
[0.2828, 11.3137]

vector_search() returns the columns you ask for, plus the distance in a column named vast_pysdk_vector_dist. Call read_all() before the transaction ends.

When the SDK opens a transaction with the ADBC driver, the driver manager may print Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant on stderr. You can ignore it: the SDK's ADBC connection runs inside the SDK transaction.

7. Clean up

Drop both tables, then the schema:

cur.execute(f"DROP TABLE IF EXISTS {SCHEMA}.docs")
cur.execute(f"DROP TABLE IF EXISTS {SCHEMA}.docs_sdk")
cur.execute(f"DROP SCHEMA IF EXISTS {SCHEMA}")
cur.close()
conn.close()
print("Dropped vs_quickstart")

Dropped vs_quickstart

Next steps

• Quantization: what cluster_quantization and search_quantization trade off.
• Query design: writing searches that use the index, with filters.
• Security: the identity policies search needs.
• ADBC and Python SDK: the two clients in more depth.