CI/CD and automation

Applies to VAST Cluster 5.5.

A vector-indexed table is a VAST DataBase table, so automating VAST Vector Search is mostly automating VAST DataBase: the same view, view policy, identity policy, user and access keys. This page covers only what's specific to vectors: creating a table with a vector index from a pipeline, through the VMS REST API, vastpy, and Terraform with the Python SDK. It also says what the VAST admin MCP server can and can't do here.

For the shared setup (the database view, its policies, the application user and its keys, API tokens, TLS, and the Terraform module this page reuses), see CI/CD and automation for VAST DataBase. To choose between the tools, see automation tools compared.

What each tool can do for vectors

Tool	Vector-specific support	How to get a vector-indexed table
VMS REST API	`POST /tables/` takes a `vector_indexes` array	Create the table with its index in one call
`vastpy` / `vastpy-cli`	The same endpoint	`vastpy-cli post tables … vector_indexes='[…]'`
Terraform provider (4.0.2)	None: there's no table resource	Create the database with Terraform, then the table with the Python SDK
VAST admin MCP server	None: no table, schema or index tools	Not possible; it can only list the database views

A vector index can only be declared when the table is created: you can't add one later, remove it, or change its metric. So a pipeline that "ensures" a vector table should create it once with the right index and otherwise leave it alone. See create an index for the choices.

Admin task: send to your platform team. The automation identity on this page is a tenant admin with a VMS API token. Ask your platform team for one, or for the database objects themselves; see CI/CD and automation for VAST DataBase and Managing VMS API tokens in the KB.

API documentation

The KB's VAST Cluster 5.5 REST API reference (getting started), in particular Create a Database Table (POST /tables/), Show a Database Table (GET /tables/show/) and Add Columns to a Database Table (PATCH /tables/add_columns/).
Your cluster's own copy at https://<VMS address>/docs/ (VMS REST API documentation). The Swagger UI at https://<VMS address>/api/ lists only a few endpoints, not the tables API.

Use the vector_indexes array shown below, as the REST API reference and the cluster do; the field names vector_index_column_name and vector_index_distance_metric aren't used. Show a Database Table returns the same vector_indexes array (check the index).

REST API

The examples use two environment variables: VMS, the VMS address, and VMS_TOKEN, a tenant admin's API token. The database (my-vectors) and schema (vec) already exist (here they also hold an SDK-created table, sdk_ref). -k skips TLS verification for a lab cluster with a self-signed certificate; in a pipeline, pass the cluster's CA with --cacert instead.

Create a table with an id column and a 3-dimensional embedding vector column, indexed for cosine distance without quantization:

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" \
  -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{
  "database_name": "my-vectors", "schema_name": "vec", "name": "items_cosine_none",
  "arrow_schema": [
    {"name": "id", "field": {"column_type": "int64"}},
    {"name": "embedding", "field": {"column_type": "vector", "item_type": {"column_type": "float"}, "size": 3}}
  ],
  "vector_indexes": [{"column_name": "embedding", "distance_metric": "COSINE",
                      "cluster_quantization": "NONE", "search_quantization": "NONE"}]
}'


HTTP 201

The call returns 201 with an empty body. The vector column is a vector of float items, and size is the dimension.

The same call with Euclidean distance and 8-bit scalar quantization, for both clustering and search:

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" \
  -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{
  "database_name": "my-vectors", "schema_name": "vec", "name": "items_l2_sq8",
  "arrow_schema": [
    {"name": "id", "field": {"column_type": "int64"}},
    {"name": "embedding", "field": {"column_type": "vector", "item_type": {"column_type": "float"}, "size": 3}}
  ],
  "vector_indexes": [{"column_name": "embedding", "distance_metric": "EUCLIDEAN",
                      "cluster_quantization": "SCALAR_8BIT", "search_quantization": "SCALAR_8BIT"}]
}'


HTTP 201

Fields and the errors you get

Field	Values
`column_name`	The vector column. One index per table.
`distance_metric`	`COSINE`, `EUCLIDEAN` or `NEGATIVE_INNER_PRODUCT` (`cosine`, `l2sq` and `ip` in SQL and the SDK)
`cluster_quantization`	`NONE` or `SCALAR_8BIT`. Required.
`search_quantization`	`NONE` or `SCALAR_8BIT`. Required. `NONE` needs `cluster_quantization` `NONE`.

Unlike SQL and the SDK, the REST API has no default quantization: leave out both fields and the request fails.

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" \
  -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{
  "database_name": "my-vectors", "schema_name": "vec", "name": "items_default",
  "arrow_schema": [
    {"name": "id", "field": {"column_type": "int64"}},
    {"name": "embedding", "field": {"column_type": "vector", "item_type": {"column_type": "float"}, "size": 3}}
  ],
  "vector_indexes": [{"column_name": "embedding", "distance_metric": "COSINE"}]
}'

{"detail":"2 validation errors for TableInputSchema\nvector_indexes -> 0 -> cluster_quantization\n  field required (type=value_error.missing)\nvector_indexes -> 0 -> search_quantization\n  field required (type=value_error.missing)"}
HTTP 400

Quantized clustering with unquantized search isn't allowed:

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" \
  -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{
  "database_name": "my-vectors", "schema_name": "vec", "name": "items_ip_sq8_none",
  "arrow_schema": [
    {"name": "id", "field": {"column_type": "int64"}},
    {"name": "embedding", "field": {"column_type": "vector", "item_type": {"column_type": "float"}, "size": 3}}
  ],
  "vector_indexes": [{"column_name": "embedding", "distance_metric": "NEGATIVE_INNER_PRODUCT",
                      "cluster_quantization": "SCALAR_8BIT", "search_quantization": "NONE"}]
}'

{"detail":"1 validation error for TableInputSchema\n__root__\n  search_quantization=NONE requires cluster_quantization=NONE (type=value_error)"}
HTTP 400

The other way round (NONE for clustering, SCALAR_8BIT for search) is accepted. For what each setting trades off, see quantization.

Check the index

The table list doesn't show indexes: GET /tables/ returns vector_indexes: null for every table, including indexed ones.

curl -sk -H "Authorization: Api-Token $VMS_TOKEN" \
  "https://$VMS/api/v8/tables/?database_name=my-vectors&schema_name=vec" | jq -c ".results[] | {name, vector_indexes}"

{"name":"sdk_ref","vector_indexes":null}
{"name":"items_cosine_none","vector_indexes":null}
{"name":"items_l2_sq8","vector_indexes":null}
{"name":"items_ip_none_sq8","vector_indexes":null}

To check a table's index, use GET /tables/show/:

curl -sk -H "Authorization: Api-Token $VMS_TOKEN" \
  "https://$VMS/api/v8/tables/show/?database_name=my-vectors&schema_name=vec&name=items_l2_sq8" | jq -c "{name, vector_indexes}"

{"name":"items_l2_sq8","vector_indexes":[{"column_name":"embedding","distance_metric":"EUCLIDEAN","cluster_quantization":"SCALAR_8BIT","search_quantization":"SCALAR_8BIT","clustering_progress":0,"clustering_progress_label":"C"}]}

clustering_progress and its grade clustering_progress_label describe background clustering; a new, empty table starts at 0 and C. See check the index first.

You can't add an index to an existing table

Adding a vector index to an existing table isn't supported, so create the index with the table. PATCH /tables/add_columns/ adds ordinary columns; it isn't a way to add an index.

What the cluster does with vector fields on PATCH /tables/add_columns/:

On a vector-indexed table, any add-columns request fails with HTTP 400. With the field names vector_index_column_name and vector_index_distance_metric (vector_index_column_name, vector_index_distance_metric), the response is {"detail":"Adding columns failed. @handle=0x<handle> adding columns on indexed tables is forbidden","code":"bad_request"}.
On a table without an index, a request that adds a vector column with a vector_indexes entry returns HTTP 204 and adds the column, but /tables/show/ then reports "vector_indexes":[]: no index is created. Don't treat the 204 as success; check the table with /tables/show/.

The same goes for quantization: a PATCH /tables/ with different quantization values in vector_indexes returns HTTP 204, and /tables/show/ still reports the old values. Quantization is fixed when you create the table.

The VAST CLI's table create in 5.5 has no vector-index option, and that command doesn't support complex data types (see table create). Use the REST API, vastpy, SQL or the Python SDK instead.

Check it from the SDK

A pipeline should end by checking the table from the application's side. This script opens each table with the Python SDK and prints its columns and index. Save it as vec_show.py:

"""Show how the SDK sees vector tables created through VMS.
usage: python vec_show.py <endpoint> <database> <schema> <table>..."""
import os, sys
import vastdb

endpoint, database, schema, *tables = sys.argv[1:]
session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"],
                         secret=os.environ["AWS_SECRET_ACCESS_KEY"])
with session.transaction() as tx:
    s = tx.bucket(database).schema(schema)
    for name in tables:
        t = s.table(name)
        print(name, [(f.name, str(f.type)) for f in t.columns()])
        print("   ", t.vector_index)

Run it with the application's key pair and the database endpoint:

AWS_ACCESS_KEY_ID=<access key> AWS_SECRET_ACCESS_KEY=<secret key> \
  python vec_show.py http://<database VIP>:9090 my-vectors vec items_cosine_none items_l2_sq8 items_ip_none_sq8

items_cosine_none [('id', 'int64'), ('embedding', 'fixed_size_list<item: float not null>[3]')]
    VectorIndex(column='embedding', distance_metric='cosine', sql_distance_function='array_cosine_distance', cluster_quantization='none', search_quantization='none')
items_l2_sq8 [('id', 'int64'), ('embedding', 'fixed_size_list<item: float not null>[3]')]
    VectorIndex(column='embedding', distance_metric='l2sq', sql_distance_function='array_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')
items_ip_none_sq8 [('id', 'int64'), ('embedding', 'fixed_size_list<item: float not null>[3]')]
    VectorIndex(column='embedding', distance_metric='ip', sql_distance_function='array_inner_product', cluster_quantization='none', search_quantization='scalar_8bit')

The SDK sees a REST vector column as a fixed-size Arrow list of non-nullable float32, and maps the REST metric names to cosine, l2sq and ip.

The key's user needs an identity policy with the s3:Tabular* actions on the database. Owning the bucket isn't enough for the SDK to read a table created through VMS; without the policy the SDK gets 403 AccessDenied. See security and CI/CD and automation for VAST DataBase.

vastpy

vastpy-cli calls the same endpoints. It reads the VMS address and token from VMS_ADDRESS and VMS_TOKEN, and doesn't verify TLS unless you set VMS_CERT_FILE (details).

First, the database itself. The DataBase page's re-runnable ensure_database.py creates the identity policy, the application user, its key pair and the view, only where they're missing:

python ensure_database.py vectors-dev vectors-dev-app 8203 my-db-policy

created identity policy vectors-dev-tabular (id 10)
created user vectors-dev-app (id 11)
created key <access key> (secret written to vectors-dev-app.key.json)
created view /vectors-dev (id 17, protocols ['S3', 'DATABASE'])

Then the schema and the vector-indexed table. Pass the column list and the index as JSON:

vastpy-cli post schemas database_name=vectors-dev name=embeddings
vastpy-cli post tables database_name=vectors-dev schema_name=embeddings name=items \
  arrow_schema='[{"name": "id", "field": {"column_type": "int64"}}, {"name": "embedding", "field": {"column_type": "vector", "item_type": {"column_type": "float"}, "size": 3}}]' \
  vector_indexes='[{"column_name": "embedding", "distance_metric": "COSINE", "cluster_quantization": "SCALAR_8BIT", "search_quantization": "SCALAR_8BIT"}]'

Both commands print only a blank line when they succeed. Check the index:

vastpy-cli get tables/show database_name=vectors-dev schema_name=embeddings name=items --json | jq -c "{name, vector_indexes}"

{"name":"items","vector_indexes":[{"column_name":"embedding","distance_metric":"COSINE","cluster_quantization":"SCALAR_8BIT","search_quantization":"SCALAR_8BIT","clustering_progress":0,"clustering_progress_label":"C"}]}

And from the SDK, with the key that ensure_database.py wrote:

AWS_ACCESS_KEY_ID=$(jq -r .access_key vectors-dev-app.key.json) AWS_SECRET_ACCESS_KEY=$(jq -r .secret_key vectors-dev-app.key.json) \
  python vec_show.py http://<database VIP>:9090 vectors-dev embeddings items

items [('id', 'int64'), ('embedding', 'fixed_size_list<item: float not null>[3]')]
    VectorIndex(column='embedding', distance_metric='cosine', sql_distance_function='array_cosine_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')

Unlike ensure_database.py, these two calls aren't written to be re-run. A pipeline that runs on every commit should check for the table with get tables/show first, or create it with the SDK and fail_if_exists=False, as in the Terraform section below.

A repeated POST /tables/ for a table that exists fails with HTTP 400 and {"detail":"Table already exists.","code":"bad_request"}. Your pipeline can treat that response as "already created", then check the index with /tables/show/.

Terraform

The VAST Terraform provider has no resource for schemas or tables, so nothing in it is vector-specific. Use Terraform for the database (view, policies, user and key) and let the application create the vector-indexed table with the Python SDK.

This root configuration reuses the DataBase configuration from CI/CD and automation for VAST DataBase as a module, from the directory ../database:

# Vector Search: no Terraform resource is vector-specific. A vector table lives in a
# VAST DataBase database, so CI provisions the same database objects (reusing ../database
# as a module) and the application creates the vector-indexed table with the SDK.
terraform {
  required_version = ">= 1.5"
  required_providers {
    vastdata = {
      source  = "vast-data/vastdata"
      version = "4.0.2"
    }
  }
}

variable "vast_host" {
  type = string
}

variable "tenant_name" {
  type = string
}

variable "pgp_public_key" {
  type = string
}

module "vector_db" {
  source         = "../database"
  vast_host      = var.vast_host
  tenant_name    = var.tenant_name
  pgp_public_key = var.pgp_public_key
  database       = "vectors-ci"
  app_user       = "vectors-ci-app"
  app_uid        = 8204
}

output "database" {
  value = module.vector_db.database
}

output "access_key" {
  value = module.vector_db.access_key
}

output "encrypted_secret_key" {
  value = module.vector_db.encrypted_secret_key
}

output "db_endpoints" {
  value = module.vector_db.db_endpoints
}

The provider reads the token from VASTDATA_API_TOKEN. Provider 4.0.2 needs host in the configuration, so pass the VMS address as TF_VAR_vast_host. The key's secret is encrypted with a PGP public key (TF_VAR_pgp_public_key, ASCII-armored).

terraform init -no-color | grep -E "Initializing modules|- vector_db|Installed|successfully"

Initializing modules...
- vector_db in ../database
- Installed vast-data/vastdata v4.0.2 (self-signed, key ID FADBA165459C78A3)
Terraform has been successfully initialized!

VASTDATA_API_TOKEN=$VMS_TOKEN TF_VAR_pgp_public_key="$(cat ci.pub.asc)" terraform apply -no-color -auto-approve \
  | grep -E "Creat|Apply|Error|error|^Outputs|^[a-z_]+ = "

module.vector_db.vastdata_s3_policy.tabular: Creating...
module.vector_db.vastdata_view_policy.db: Creating...
module.vector_db.vastdata_s3_policy.tabular: Creation complete after 1s [name=vectors-ci-tabular]
module.vector_db.vastdata_user.app: Creating...
module.vector_db.vastdata_view_policy.db: Creation complete after 4s [name=vectors-ci-policy]
module.vector_db.vastdata_user.app: Creation complete after 4s [name=vectors-ci-app]
module.vector_db.vastdata_user_key.app: Creating...
module.vector_db.vastdata_view.db: Creating...
module.vector_db.vastdata_view.legacy: Creating...
module.vector_db.vastdata_user_key.app: Creation complete after 1s
module.vector_db.vastdata_view.db: Creation complete after 4s [name=view-18]
module.vector_db.vastdata_view.legacy: Creation complete after 6s [name=view-13]
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
Outputs:
access_key = "<access key>"
database = "vectors-ci"
db_endpoints = tolist([
encrypted_secret_key = <<EOT

Five of these resources are the database. The sixth, vastdata_view.legacy, is a lesson: Terraform reads every .tf file in a module's directory, and in this run ../database still held the import.tf from the DataBase page's import example. The module took over that existing view (/legacy-db) and changed its owner and view policy to its own, without a warning. Keep a directory you use as a module free of anything else. In this run the extra resource was removed from the state with terraform state rm module.vector_db.vastdata_view.legacy, after which the plan was clean:

VASTDATA_API_TOKEN=$VMS_TOKEN TF_VAR_pgp_public_key="$(cat ci.pub.asc)" terraform plan -no-color | tail -3


Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

The db_endpoints output lists the database VIPs the cluster reports:

terraform output -no-color db_endpoints

tolist([
  {
    "port" = 9090
    "tenant_id" = tonumber(null)
    "vip" = "<database VIP>"
  },
])

Check it against the endpoint your platform team gave you before you use it in a pipeline.

Then the application creates the vector-indexed table. fail_if_exists=False makes the step safe to re-run. Save as create_vector_table.py:

"""After `terraform apply`: create the vector-indexed table with the SDK.
usage: python create_vector_table.py <endpoint> <database>; env AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY"""
import os, sys
import pyarrow as pa
import vastdb
from vastdb import VectorIndexSpec

endpoint, database = sys.argv[1], sys.argv[2]
dim = 3
cols = pa.schema([("id", pa.int64()),
                  ("embedding", pa.list_(pa.field("item", pa.float32(), nullable=False), dim))])
session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"],
                         secret=os.environ["AWS_SECRET_ACCESS_KEY"])
with session.transaction() as tx:
    s = tx.bucket(database).create_schema("embeddings", fail_if_exists=False)
    t = s.create_table("items", cols, fail_if_exists=False,
                       vector_index=VectorIndexSpec(column="embedding", distance_metric="l2sq"))
    print(t.vector_index)

Run it with the key from the Terraform outputs. The secret key is decrypted with the CI's PGP private key:

AWS_ACCESS_KEY_ID=$(terraform output -raw access_key) \
AWS_SECRET_ACCESS_KEY=$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt) \
  python create_vector_table.py http://<database VIP>:9090 $(terraform output -raw database)

VectorIndex(column='embedding', distance_metric='l2sq', sql_distance_function='array_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')

The SDK applies the default quantization, scalar_8bit for both settings. To choose another, pass cluster_quantization and search_quantization to VectorIndexSpec; see create an index.

Tear down

Terraform can't delete a view that still holds schemas or tables, so drop them first. With vastpy-cli:

vastpy-cli delete tables database_name=vectors-ci schema_name=embeddings name=items && \
  vastpy-cli delete schemas database_name=vectors-ci name=embeddings && echo "schema dropped"


schema dropped

VASTDATA_API_TOKEN=$VMS_TOKEN TF_VAR_pgp_public_key="$(cat ci.pub.asc)" terraform destroy -no-color -auto-approve \
  | grep -E "Destroy complete|Error|error|│|detail"

Destroy complete! Resources: 5 destroyed.

Dropping a table deletes its vectors. In a shared environment, give destroy jobs a manual approval.

VAST admin MCP server

The VAST admin MCP server (vast-admin-mcp) has nothing for Vector Search: it has no tools for tables, schemas or indexes, so an assistant can't create or inspect a vector index through it. It can list the database views that hold your vector tables, with list_views_vast:

call list_views_vast {"path": "/vectors*"}
isError: False
[
 {
  "cluster": "<cluster>",
  "tenant": "<tenant>",
  "name": "view-17",
  "path": "/vectors-dev",
  "view_policy": "my-db-policy",
  "bucket": "vectors-dev",
  "share": "",
  "protocols": "S3,DATABASE",
  "logical_used": "0.00 B",
  "physical_used": "0.00 B",
  "hard_quota": null,
  "qos_policy": null
 },
 {
  "cluster": "<cluster>",
  "tenant": "<tenant>",
  "name": "view-18",
  "path": "/vectors-ci",
  "view_policy": "vectors-ci-policy",
  "bucket": "vectors-ci",
  "share": "",
  "protocols": "S3,DATABASE",
  "logical_used": "0.00 B",
  "physical_used": "0.00 B",
  "hard_quota": null,
  "qos_policy": null
 }
]

For an assistant that works with the vectors themselves, give it the Python SDK or the Query Engine instead. Setup and the tools the server does have are in CI/CD and automation for VAST DataBase.

A CI job

This GitHub Actions workflow puts the Terraform steps above into one job: create the database, decrypt the key, and create and check the vector table. It was not run end to end on a CI runner. Each step it runs was run by hand on a 5.5.0-SP6 lab cluster, as shown above.

# Not run end to end on GitHub. The steps match the Terraform section of this page.
#
# Repository variables: VAST_HOST (VMS address), VAST_TENANT, DB_ENDPOINT (http://<database VIP>:9090)
# Repository secrets:   VASTDATA_API_TOKEN (tenant-admin API token),
#                       CI_PGP_PUBLIC_KEY, CI_PGP_PRIVATE_KEY (ASCII-armored, no passphrase)
# The runner must reach VMS on port 443 and the database VIP: use a self-hosted runner for a private network.
name: vector-db

on:
  push:
    branches: [main]

concurrency:
  group: vector-db
  cancel-in-progress: false

jobs:
  provision:
    runs-on: ubuntu-24.04
    env:
      VASTDATA_API_TOKEN: ${{ secrets.VASTDATA_API_TOKEN }}
      TF_VAR_vast_host: ${{ vars.VAST_HOST }}
      TF_VAR_tenant_name: ${{ vars.VAST_TENANT }}
      TF_VAR_pgp_public_key: ${{ secrets.CI_PGP_PUBLIC_KEY }}
    defaults:
      run:
        working-directory: terraform/vector-search
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_wrapper: false
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install vastdb==2.1.0
      - run: terraform init -input=false
      - run: terraform apply -input=false -auto-approve
      - name: Create and check the vector table
        run: |
          echo "${{ secrets.CI_PGP_PRIVATE_KEY }}" | gpg --batch --import
          export AWS_ACCESS_KEY_ID=$(terraform output -raw access_key)
          export AWS_SECRET_ACCESS_KEY=$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt)
          echo "::add-mask::$AWS_SECRET_ACCESS_KEY"
          python create_vector_table.py "${{ vars.DB_ENDPOINT }}" "$(terraform output -raw database)"

Terraform state holds the encrypted secret key and the access key: keep it in a remote backend with restricted access. The DataBase page covers state and secrets in more detail.