CI/CD and automation

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

A database your application depends on is more than tables. It needs a Database-enabled view, a view policy, an identity policy with the Tabular actions, an application user and that user's S3 key pair. This page shows how a CI job creates and checks those objects with the VMS REST API, vastpy, and the VAST Terraform provider, and how to create schemas and tables from a pipeline step. It ends with what the VAST admin MCP server can do for VAST DataBase in interactive, AI-assisted work.

For how the four tools compare across VAST DataBase, Vector Search, Event Broker and DataEngine, see automation tools.

The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6, with Terraform 1.16.5, provider vast-data/vastdata 4.0.2, vastpy 0.3.22, vast-admin-mcp 0.2.6 and the vastdb SDK 2.1.0. The tenant is my-tenant. Your names, IDs and addresses will differ.

What a CI job can automate

Object	REST API (VMS)	`vastpy` / `vastpy-cli`	Terraform provider 4.0.2	Admin MCP server 0.2.6
Database (a view with `S3` and `DATABASE`; the bucket is the database name)	`/views/`	Yes	`vastdata_view`	List; create in read-write mode
View policy	`/viewpolicies/`	Yes	`vastdata_view_policy`	List
Identity policy with the Tabular actions	`/s3policies/`	Yes	`vastdata_s3_policy`	List
Application user	`/users/`	Yes	`vastdata_user`	No
The user's S3 key pair	`/users/<id>/access_keys/`	Yes	`vastdata_user_key` (can't be imported)	List
Schemas, tables, columns	`/schemas/`, `/tables/`, `/tables/add_columns/`, `/columns/`	Yes	None	None
Database endpoints of the tenant			Data source `vastdata_vast_db_vips`

The VIP pool that serves VAST DataBase, and the tenant itself, are cluster-admin objects. Ask your platform team for them; they don't belong in an application's pipeline.

Before you start

You need:

A VMS API token for a tenant admin, held as a masked CI secret. REST, vastpy and Terraform all accept it. The admin MCP server doesn't for a tenant admin (see below).
Network access from the runner to the VMS management address on port 443, and, for the SDK smoke test, to a database endpoint (the setup prerequisites explain the endpoint).
The VMS CA certificate, if the job checks TLS (it should).

A tenant admin can create the token once with a password, through the REST API. The owner must be a manager in the same tenant, and expiry_date takes a duration such as 30d:

curl -sk -u "$VMS_USER:$VMS_PASSWORD" -H "X-Tenant-Name: my-tenant" -H "Content-Type: application/json" \
  -X POST "https://$VMS/api/v8/apitokens/" \
  -d "{\"name\": \"ci-database\", \"expiry_date\": \"30d\", \"owner\": \"$VMS_USER\"}" | jq .

{
  "id": "<token id>",
  "token": "<token>"
}

The token is shown only in this response: put it straight into your CI secret store. A token for a user outside the tenant is refused:

{"detail":"Manager admin not found, please provide a valid owner username","code":"bad_request"}

Admin task: send to your platform team if you can't create tokens yourself: Managing VMS API tokens.

The examples below use these variables:

export VMS="<VMS address>"        # curl
export VMS_ADDRESS="$VMS"         # vastpy and vastpy-cli
export VMS_TOKEN="<token>"        # vastpy and vastpy-cli
export VAST_TENANT="my-tenant"    # the ensure script

TLS

The curl examples use -k, as on the lab cluster, whose VMS certificate is self-signed. In a pipeline, check the certificate:

curl: --cacert <VMS CA file>.
vastpy doesn't check certificates unless you give it a CA file (VMS_CERT_FILE, or cert_file= in Python). If the certificate doesn't list the address you connect to, which is the case for the VMS's default certificate and an IP address, verification fails with SSLCertVerificationError. Set VMS_CERT_SERVER to a name the certificate does list:
```
VMS_CERT_FILE=vms-ca.pem VMS_CERT_SERVER=vms.vastdata.com vastpy-cli get views path=/analytics --json | jq -c ".[] | {id, path}"
```
```
{"id":12,"path":"/analytics"}
```
Terraform: the provider's skip_ssl_verify is set to true in the example below, for the lab cluster. Remove it when your VMS certificate is trusted.

API documentation

The VMS REST API reference for 5.5 in the KB: https://kb.vastdata.com/api-5-5/apidocs, for example create a view, create an identity policy, generate an S3 access key pair, create a database schema, create a database table and add columns to a database table.
The cluster's own copy: https://<VMS address>/docs/ ("VAST Management Service (VMS) REST API Documentation"). https://<VMS address>/api/ is a Swagger UI, but on 5.5.0-SP6 it lists only a handful of paths (tokens, login banner, LEDs), not the database endpoints.

Call /api/v8/... or /api/latest/... (the same on 5.5). An unversioned /api/... call gets the oldest API version's shape; see API versions.

Terraform

The Terraform provider manages everything an application needs to reach its database, except schemas and tables. This configuration creates a database analytics: the identity policy, an application user that owns the bucket, its S3 key pair, the view policy, and the view. It outputs the access key, the secret key encrypted to a CI PGP key, and the tenant's database endpoints.

versions.tf:

terraform {
  required_version = ">= 1.5"
  required_providers {
    vastdata = {
      source  = "vast-data/vastdata"
      version = "4.0.2"
    }
  }
}

# Provider 4.0.2 requires `host` in the configuration: the VASTDATA_HOST environment
# variable alone fails validation. Pass it as a variable (TF_VAR_vast_host in CI).
variable "vast_host" {
  type = string
}

provider "vastdata" {
  host            = var.vast_host
  skip_ssl_verify = true # lab cluster with a self-signed certificate
}

main.tf:

variable "tenant_name" {
  type    = string
  default = "my-tenant"
}

variable "database" {
  type    = string
  default = "analytics"
}

variable "app_user" {
  type    = string
  default = "analytics-app"
}

variable "app_uid" {
  type    = number
  default = 8201
}

variable "pgp_public_key" {
  description = "ASCII-armored public key of the CI key (gpg --armor --export KEYID)"
  type        = string
}

data "vastdata_tenant" "this" {
  name = var.tenant_name
}

resource "vastdata_s3_policy" "tabular" {
  name      = "${var.database}-tabular"
  tenant_id = data.vastdata_tenant.this.id
  enabled   = true
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:ListBucket", "s3:Tabular*"]
        Resource = [var.database, "${var.database}/*"]
      }
    ]
  })
}

resource "vastdata_user" "app" {
  name                = var.app_user
  uid                 = var.app_uid
  local_provider_id   = data.vastdata_tenant.this.local_provider_id
  allow_create_bucket = true
  s3_policies_ids     = [vastdata_s3_policy.tabular.id]
}

resource "vastdata_user_key" "app" {
  user_id        = vastdata_user.app.id
  tenant_id      = data.vastdata_tenant.this.id
  pgp_public_key = var.pgp_public_key # the secret key is stored only encrypted in state
}

resource "vastdata_view_policy" "db" {
  name      = "${var.database}-policy"
  tenant_id = data.vastdata_tenant.this.id
  flavor    = "S3_NATIVE"
}

resource "vastdata_view" "db" {
  path         = "/${var.database}"
  bucket       = var.database
  bucket_owner = vastdata_user.app.name
  protocols    = ["S3", "DATABASE"]
  policy_id    = vastdata_view_policy.db.id
  tenant_id    = data.vastdata_tenant.this.id
  create_dir   = true
}

data "vastdata_vast_db_vips" "this" {
  tenant_id = data.vastdata_tenant.this.id
}

output "database" {
  value = vastdata_view.db.bucket
}

output "access_key" {
  value = vastdata_user_key.app.access_key
}

output "encrypted_secret_key" {
  value = vastdata_user_key.app.encrypted_secret_key
}

output "db_endpoints" {
  value = data.vastdata_vast_db_vips.this.vips
}

The identity policy is the practitioner policy from the setup prerequisites. Narrow its actions if the application only reads.

Plan and apply

Set the credentials as environment variables, as a CI job would:

export TF_VAR_vast_host="<VMS address>"
export VASTDATA_API_TOKEN="<token>"
export TF_VAR_pgp_public_key="$(gpg --armor --export ci@example.com)"
terraform init -no-color | grep -E "Installing|Installed|initialized"

- Installing vast-data/vastdata v4.0.2...
- Installed vast-data/vastdata v4.0.2 (self-signed, key ID FADBA165459C78A3)
Terraform has been successfully initialized!

If host isn't in the configuration, plan and validate stop, even with VASTDATA_HOST set:

Error: Missing required argument

  on versions.tf line 13, in provider "vastdata":
  13: provider "vastdata" {

The argument "host" is required, but no definition was found.

terraform plan -no-color -out tfplan lists the five resources. It ends with:

Plan: 5 to add, 0 to change, 0 to destroy.
Changes to Outputs:
  + access_key           = (known after apply)
  + database             = "analytics"
  + db_endpoints         = [
      + {
          + port      = 9090
          + tenant_id = null
          + vip       = "<vip>"
        },
    ]
  + encrypted_secret_key = (known after apply)

pgp_public_key must be an ASCII-armored key. In our first run it held a base64 encoding of the binary key, and apply created the other four resources, then failed at the key:

Error: PrepareCreateResource["user_key"]

  with vastdata_user_key.app,
  on main.tf line 54, in resource "vastdata_user_key" "app":
  54: resource "vastdata_user_key" "app" {

gopenpgp: unable to parse public key: gopenpgp: error in reading key ring:
openpgp: invalid argument: no armored data found

With the armored key, the next terraform apply -no-color -auto-approve created only what was missing. The end of its output:

vastdata_user_key.app: Creating...
vastdata_user_key.app: Creation complete after 1s
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
access_key = "<access key>"
database = "analytics"
db_endpoints = tolist([
  {
    "port" = 9090
    "tenant_id" = tonumber(null)
    "vip" = "<vip>"
  },
])
encrypted_secret_key = <<EOT
-----BEGIN PGP MESSAGE-----
Comment: https://gopenpgp.org
Version: GopenPGP 2.9.0
<PGP message>
-----END PGP MESSAGE-----
EOT

With pgp_public_key set, the state holds only encrypted_secret_key. Without it, the state holds the secret key itself (as a sensitive value), so keep the state in a remote backend with encryption and access control either way: it also holds the user and policy details.

Check from the data plane

Decrypt the secret key with the CI key and run a smoke test with the SDK. db_smoke.py:

"""CI smoke test: connect with the app's key pair and list the database's schemas.
usage: python db_smoke.py <endpoint> <database>; env AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY"""
import os, sys
import vastdb

endpoint, database = sys.argv[1], sys.argv[2]
session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"],
                         secret=os.environ["AWS_SECRET_ACCESS_KEY"], timeout=10)
with session.transaction() as tx:
    bucket = tx.bucket(database)
    print("schemas:", [s.name for s in bucket.schemas()])

AWS_ACCESS_KEY_ID=$(terraform output -raw access_key) \
AWS_SECRET_ACCESS_KEY=$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt) \
python db_smoke.py http://<database endpoint> analytics

schemas: []

A new database has no schemas. If this fails, the setup prerequisites map each error to the item to check.

Bring an existing database under Terraform

To manage a view that a script or the Web UI created, write the resource and import the view by path and tenant. Here a script created /legacy-db first:

vastpy-cli post views path=/legacy-db bucket=legacy-db bucket_owner=analytics-app policy_id=10 protocols=S3 protocols=DATABASE create_dir=true --json | jq -c "{id, path, bucket, protocols, policy_id, tenant_id}"

{"id":13,"path":"/legacy-db","bucket":"legacy-db","protocols":["S3","DATABASE"],"policy_id":10,"tenant_id":3}

Add the resource (import.tf):

resource "vastdata_view" "legacy" {
  path         = "/legacy-db"
  bucket       = "legacy-db"
  bucket_owner = vastdata_user.app.name
  protocols    = ["S3", "DATABASE"]
  policy_id    = vastdata_view_policy.db.id
  tenant_id    = data.vastdata_tenant.this.id
}

Then import it:

terraform import -no-color vastdata_view.legacy "path=/legacy-db,tenant_name=my-tenant"

data.vastdata_tenant.this: Reading...
data.vastdata_tenant.this: Read complete after 3s [name=my-tenant]
data.vastdata_vast_db_vips.this: Reading...
vastdata_view.legacy: Importing from ID "path=/legacy-db,tenant_name=my-tenant"...
data.vastdata_vast_db_vips.this: Read complete after 0s
vastdata_view.legacy: Import prepared!
  Prepared vastdata_view for import
vastdata_view.legacy: Refreshing state... [name=view-13]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

The next terraform plan ends with No changes. Your infrastructure matches the configuration.

Import before you apply. If a configuration declares a view whose path already exists and you apply without importing, the provider doesn't fail: it reports Creation complete for the existing view, takes it over, and sets its attributes to the configuration's. In our lab, a second configuration that also declared /legacy-db changed that view's bucket_owner to other-app and its policy_id to 11, with no warning. Check that each path is declared in one configuration only.

S3 key pairs can't be imported. A key created outside Terraform stays outside it:

terraform import -no-color vastdata_user_key.existing "access_key=<access key>"

vastdata_user_key.existing: Importing from ID "access_key=<access key>"...
Error: ImportState["user_key"]: import not supported.
This resource has been marked as not importable.

Let Terraform create a new key for the user, move the application to it, then disable the old key.

Drift

The provider reverts changes made outside Terraform on the next apply. Here someone removes DATABASE from the view:

vastpy-cli patch views/12 protocols='["S3"]' --json | jq -c "{id, path, protocols}"

{"id":12,"path":"/analytics","protocols":["S3"]}

terraform plan -no-color -out tfplan shows the change it will undo:

Terraform will perform the following actions:
  # vastdata_view.db will be updated in-place
  ~ resource "vastdata_view" "db" {
        id                                 = 12
        name                               = "view-12"
      ~ protocols                          = [
          + "DATABASE",
            # (1 unchanged element hidden)
        ]
        # (64 unchanged attributes hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

terraform apply -no-color tfplan

vastdata_view.db: Modifying... [name=view-12]
vastdata_view.db: Modifications complete after 4s [name=view-12]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

To detect drift without fixing it, for example in a scheduled job, run terraform plan -detailed-exitcode: it exits with 2 when there are changes.

Some changes VMS refuses outright. A view policy that a Database-enabled view uses can't be switched to the SMB or Mixed Last Wins flavor:

vastpy-cli patch viewpolicies/10 flavor=MIXED_LAST_WINS --json

Error: response for PATCH api/latest/viewpolicies/10 with params=None failed with error 409 and message {"detail":"a View Policy with SMB or Mixed Last Wins cannot have attached View DATABASE protocol","code":"conflict_error"}

Destroy

terraform destroy deletes the key, the user, the identity policy, the view and the view policy. Two things stop it:

A view with schemas can't be deleted. Drop the tables and schemas first (see below).
A view policy that other views use can't be deleted. Destroy fails with 409 Cannot delete view policy that has referenced views until those views are gone or moved to another policy. Give each database's configuration its own view policy, as the example does.

vastpy

vastpy is the official Python client for the VMS REST API: pip install vastpy (it depends only on urllib3). It builds the URL from attribute and index chaining, so client.views[12].patch(...) sends a PATCH to /api/latest/views/12. It also installs vastpy-cli, which takes the same calls as get|post|patch|delete <endpoint> key=value ... and reads VMS_ADDRESS, VMS_TOKEN, VMS_CERT_FILE and VMS_CERT_SERVER from the environment. Source and documentation: github.com/vast-data/vastpy.

A re-runnable ensure script

Use a script instead of Terraform when you don't want to keep state. This one ensures a database exists for an application: it creates whatever is missing of the identity policy, the user, the user's key pair and the view, and leaves what exists alone. The view policy is an existing one, passed by name. ensure_database.py:

"""Ensure a VAST DataBase database exists for an application: identity policy, owner user,
S3 key pair, and a view with S3 + DATABASE. Safe to re-run (creates only what is missing).
Env: VMS_ADDRESS, VMS_TOKEN (tenant-admin API token), VAST_TENANT.
usage: python ensure_database.py <database> <app-user> <uid> <view-policy-name>"""
import json, os, sys
from vastpy import VASTClient

db, user_name, uid, vpolicy_name = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"])

def one(items):
    return items[0] if items else None

tenant = one(client.tenants.get(name=os.environ["VAST_TENANT"]))

# 1. Identity policy: s3:ListBucket + all Tabular actions on the database bucket.
pname = f"{db}-tabular"
policy = one(client.s3policies.get(name=pname))
if not policy:
    doc = {"Version": "2012-10-17", "Statement": [{"Effect": "Allow",
           "Action": ["s3:ListBucket", "s3:Tabular*"], "Resource": [db, f"{db}/*"]}]}
    policy = client.s3policies.post(name=pname, policy=json.dumps(doc), enabled=True,
                                    tenant_id=tenant["id"])
    print(f"created identity policy {pname} (id {policy['id']})")
else:
    print(f"ok identity policy {pname} (id {policy['id']})")

# 2. Application user in the tenant's local provider, with the policy attached.
user = one(client.users.get(name=user_name, local_provider_id=tenant["local_provider_id"]))
if not user:
    user = client.users.post(name=user_name, uid=uid, local_provider_id=tenant["local_provider_id"],
                             allow_create_bucket=True, s3_policies_ids=[policy["id"]])
    print(f"created user {user_name} (id {user['id']})")
else:
    print(f"ok user {user_name} (id {user['id']})")

# 3. One S3 key pair. The secret is returned only once: write it to a 0600 file for the CI
#    secret store, never to the log.
keys = [k for k in (user.get("access_keys") or []) if k.get("enabled")]
if not keys:
    k = client.users[user["id"]].access_keys.post(tenant_id=tenant["id"])
    fd = os.open(f"{user_name}.key.json", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
    with os.fdopen(fd, "w") as f:
        json.dump({"access_key": k["access_key"], "secret_key": k["secret_key"]}, f)
    print(f"created key {k['access_key']} (secret written to {user_name}.key.json)")
else:
    print(f"ok key {keys[0]['access_key']}")

# 4. The database itself: a view whose bucket is the database name.
view = one(client.views.get(path=f"/{db}", tenant_id=tenant["id"]))
if not view:
    vp = one(client.viewpolicies.get(name=vpolicy_name, tenant_id=tenant["id"]))
    view = client.views.post(path=f"/{db}", bucket=db, bucket_owner=user_name,
                             protocols=["S3", "DATABASE"], policy_id=vp["id"],
                             tenant_id=tenant["id"], create_dir=True)
    print(f"created view /{db} (id {view['id']}, protocols {view['protocols']})")
elif "DATABASE" not in view["protocols"]:
    view = client.views[view["id"]].patch(protocols=sorted(set(view["protocols"]) | {"DATABASE"}))
    print(f"added DATABASE to view /{db}: {view['protocols']}")
else:
    print(f"ok view /{db} (id {view['id']}, protocols {view['protocols']})")

The first run creates everything:

python ensure_database.py orders-db orders-app 8202 db-policy

created identity policy orders-db-tabular (id 9)
created user orders-app (id 10)
created key <access key> (secret written to orders-app.key.json)
created view /orders-db (id 16, protocols ['S3', 'DATABASE'])

A second run changes nothing:

python ensure_database.py orders-db orders-app 8202 db-policy

ok identity policy orders-db-tabular (id 9)
ok user orders-app (id 10)
ok key <access key>
ok view /orders-db (id 16, protocols ['S3', 'DATABASE'])

The cluster returns a secret key only when it creates it, so the script writes it to a file readable only by the job's user, for the next step to put into your secret store. Don't print it. The same REST calls with curl show what the script built:

curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/views/?path=/orders-db" | jq -c ".[] | {id, path, bucket, bucket_owner, protocols, policy_id}"

{"id":16,"path":"/orders-db","bucket":"orders-db","bucket_owner":"orders-app","protocols":["S3","DATABASE"],"policy_id":10}

curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/s3policies/?name=orders-db-tabular" | jq ".[] | {id, name, enabled, users, policy: (.policy|fromjson)}"

{
  "id": 9,
  "name": "orders-db-tabular",
  "enabled": true,
  "users": [
    "orders-app"
  ],
  "policy": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket",
          "s3:Tabular*"
        ],
        "Resource": [
          "orders-db",
          "orders-db/*"
        ]
      }
    ]
  }
}

Then check from the data plane with the new key, as for Terraform:

AWS_ACCESS_KEY_ID=$(jq -r .access_key orders-app.key.json) AWS_SECRET_ACCESS_KEY=$(jq -r .secret_key orders-app.key.json) python db_smoke.py http://<database endpoint> orders-db

schemas: []

Schemas and tables from a pipeline step

VMS also manages schemas, tables and columns, so a migration step can create them over HTTPS to the VMS with the tenant admin's token, without reaching a database endpoint. Use this for DDL that belongs with the rest of the infrastructure, such as creating a schema and its tables before the first deploy. Keep application-driven DDL (tables an application creates, evolves or drops at run time) in the SDK, which runs with the application's own key and permissions; see ingest and data modeling.

The SDK needs a Tabular identity policy to open a table created through VMS. In our lab, the bucket owner's key, without an identity policy that grants the Tabular actions, got 403 when it opened such a table (vastdb.errors.Forbidden: {'code': 'AccessDenied', 'message': '[req=...]: Permission check failed for @s3_object_name=<schema>/<table> ...). The application user above has the policy, so it can.

With vastpy-cli

Create a schema and a table. vastpy-cli post prints only an empty line when it succeeds:

vastpy-cli post schemas database_name=orders-db name=sales
vastpy-cli post tables database_name=orders-db schema_name=sales name=orders arrow_schema='[{"name": "order_id", "field": {"column_type": "int64"}}, {"name": "sku", "field": {"column_type": "string"}}, {"name": "amount", "field": {"column_type": "double"}}]'

Check the columns:

vastpy-cli get columns database_name=orders-db schema_name=sales table_name=orders

database_name |schema_name |table_name |name     |field               |raw_field 
--------------+------------+-----------+---------+--------------------+----------+
orders-db     |sales       |orders     |order_id |column_type: int64  |int64     
orders-db     |sales       |orders     |sku      |column_type: string |string    
orders-db     |sales       |orders     |amount   |column_type: double |double

vastpy-cli get schemas database_name=orders-db and vastpy-cli get tables database_name=orders-db schema_name=sales list the schemas and tables (with row counts, size and sorting state) the same way.

The application can use the table straight away. db_check_table.py opens it with the application's key, inserts a row and reads it back:

"""Open a table created through VMS with the SDK, insert a row, read it back.
usage: python db_check_table.py <endpoint> <database> <schema> <table>; env AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY"""
import os, sys
import pyarrow as pa
import vastdb

endpoint, database, schema, table = sys.argv[1:5]
session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"],
                         secret=os.environ["AWS_SECRET_ACCESS_KEY"])
with session.transaction() as tx:
    t = tx.bucket(database).schema(schema).table(table)
    print("columns:", [(f.name, str(f.type)) for f in t.columns()])
    t.insert(pa.record_batch([[1], ["A-1"], [2.5]], schema=t.columns()))
with session.transaction() as tx:
    t = tx.bucket(database).schema(schema).table(table)
    print(t.select().read_all().to_pandas().to_string(index=False))

AWS_ACCESS_KEY_ID=$(jq -r .access_key orders-app.key.json) AWS_SECRET_ACCESS_KEY=$(jq -r .secret_key orders-app.key.json) python db_check_table.py http://<database endpoint> orders-db sales orders 2>&1 | tail -4

columns: [('order_id', 'int64'), ('sku', 'string'), ('amount', 'double')]
 order_id sku  amount
        1 A-1     2.5

With curl

The same endpoints from curl. A sorted table takes sorted_column_names:

curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/schemas/" -d '{"database_name": "orders-db", "name": "staging"}'

HTTP 201

curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "ts", "field": {"column_type": "timestamp", "unit": "us"}}, {"name": "kind", "field": {"column_type": "string"}}], "sorted_column_names": ["ts"]}'

HTTP 201

curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/tables/?database_name=orders-db&schema_name=staging" | jq -c ".results[] | {name, num_rows, sorting_key_enabled}"

{"name":"events","num_rows":0,"sorting_key_enabled":true}

Adding columns is a PATCH; a POST is refused:

curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/add_columns/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "payload", "field": {"column_type": "string"}}]}'

{"detail":"Method \"POST\" not allowed."}HTTP 405

curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X PATCH "https://$VMS/api/v8/tables/add_columns/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "payload", "field": {"column_type": "string"}}]}'
curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/columns/?database_name=orders-db&schema_name=staging&table_name=events" | jq -c ".[] | {name, raw_field}"

HTTP 204
{"name":"ts","raw_field":"timestamp[us, tz=UTC]"}
{"name":"kind","raw_field":"string"}
{"name":"payload","raw_field":"string"}

Dropping a table or a schema is a DELETE with a JSON body that names it. Drop the tables before their schema:

curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X DELETE "https://$VMS/api/v8/tables/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events"}'
curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X DELETE "https://$VMS/api/v8/schemas/" -d '{"database_name": "orders-db", "name": "staging"}'

HTTP 204
HTTP 204

When you script against these endpoints, note the response shapes: /schemas/ and /columns/ return a JSON list, while /tables/ returns a paginated object (count, next, previous, results).

A CI job

These jobs apply the Terraform configuration above on the main branch and then run the SDK smoke test. They need a runner that can reach the VMS and a database endpoint, a remote Terraform backend (add a backend block for the one your team uses; state on the runner is lost after each job), and these secrets and variables:

Name	Kind	Value
`VMS_TOKEN`	Secret (masked)	The tenant admin's VMS API token
`CI_PGP_PRIVATE_KEY`	Secret (masked)	The CI PGP private key (armored), to decrypt the S3 secret key
`CI_PGP_PUBLIC_KEY`	Variable	The matching armored public key
`VMS_ADDRESS`	Variable	The VMS address
`DB_ENDPOINT`	Variable	A database endpoint, such as `http://<vip>:9090` (or use the `db_endpoints` output)

The job files were not run end to end: each step was run on its own against VAST Cluster 5.5.0-SP6, with the commands shown earlier on this page.

GitHub Actions (.github/workflows/database.yml):

# Not run end to end. Each step was exercised separately on VAST Cluster 5.5.0-SP6.
name: database
on:
  pull_request:
    paths: ["infra/database/**"]
  push:
    branches: [main]
    paths: ["infra/database/**"]

jobs:
  terraform:
    runs-on: self-hosted          # needs HTTPS to the VMS and access to a database endpoint
    defaults:
      run:
        working-directory: infra/database
    env:
      TF_VAR_vast_host: ${{ vars.VMS_ADDRESS }}
      TF_VAR_pgp_public_key: ${{ vars.CI_PGP_PUBLIC_KEY }}
      VASTDATA_API_TOKEN: ${{ secrets.VMS_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.16.5
          terraform_wrapper: false   # so $(terraform output -raw ...) returns the bare value
      - run: terraform init -input=false -no-color
      - run: terraform plan -input=false -no-color -out tfplan
      - if: github.event_name == 'push'
        run: terraform apply -input=false -no-color tfplan
      - name: SDK smoke test
        if: github.event_name == 'push'
        env:
          CI_PGP_PRIVATE_KEY: ${{ secrets.CI_PGP_PRIVATE_KEY }}
          DB_ENDPOINT: ${{ vars.DB_ENDPOINT }}
        run: |
          echo "$CI_PGP_PRIVATE_KEY" | gpg --batch --quiet --import
          python -m pip install --quiet vastdb==2.1.0
          export AWS_ACCESS_KEY_ID="$(terraform output -raw access_key)"
          export AWS_SECRET_ACCESS_KEY="$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt)"
          echo "::add-mask::$AWS_SECRET_ACCESS_KEY"
          python db_smoke.py "$DB_ENDPOINT" "$(terraform output -raw database)"

GitLab CI (.gitlab-ci.yml):

# Not run end to end. Each step was exercised separately on VAST Cluster 5.5.0-SP6.
# VMS_TOKEN and CI_PGP_PRIVATE_KEY are masked CI/CD variables; VMS_ADDRESS,
# CI_PGP_PUBLIC_KEY and DB_ENDPOINT are plain variables.
stages: [plan, apply]

variables:
  TF_VAR_vast_host: $VMS_ADDRESS
  TF_VAR_pgp_public_key: $CI_PGP_PUBLIC_KEY
  VASTDATA_API_TOKEN: $VMS_TOKEN

default:
  image:
    name: hashicorp/terraform:1.16.5
    entrypoint: [""]
  before_script:
    - cd infra/database
    - terraform init -input=false -no-color

plan:
  stage: plan
  script:
    - terraform plan -input=false -no-color -out tfplan
  artifacts:
    paths: [infra/database/tfplan]

apply:
  stage: apply
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - terraform apply -input=false -no-color tfplan
    - apk add --no-cache gnupg python3 py3-pip
    - echo "$CI_PGP_PRIVATE_KEY" | gpg --batch --quiet --import
    - python3 -m pip install --quiet --break-system-packages vastdb==2.1.0
    - export AWS_ACCESS_KEY_ID="$(terraform output -raw access_key)"
    - export AWS_SECRET_ACCESS_KEY="$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt)"
    - python3 db_smoke.py "$DB_ENDPOINT" "$(terraform output -raw database)"

For a script-based pipeline, replace the Terraform steps with python ensure_database.py ... and a step that moves the .key.json file into your secret store. The ensure script is safe to run on every push.

Interactive work with the admin MCP server

The VAST admin MCP server (vast-admin-mcp) lets an AI assistant such as Claude Desktop, Cursor or VS Code query and, if you allow it, change the cluster. It's for interactive work, not for pipelines. Install and setup: github.com/vast-data/vast-admin-mcp (pip install vast-admin-mcp, Python 3.10 or later; then vast-admin-mcp setup and vast-admin-mcp mcpsetup <client>).

For VAST DataBase, version 0.2.6 can:

Task	Tool	Notes
Find database views	`list_views_vast`	Filter by path, for example `/analytics*`. Each view shows its `protocols` (`S3,DATABASE`), bucket and view policy.
Show view policies and identity policies	`list_viewpolicies_vast`, `list_identity_policies_vast`	The identity policy's JSON is shown as a string.
Show a user's S3 keys	`list_s3_keys_vast`	Filter by the exact user name; a wildcard such as `analytics*` returned nothing.
Create a database view	`create_view_vast` with `protocols` `S3,DATABASE`, a `bucket`, `bucket_owner` and `policy`	Only in read-write mode. The view works for the SDK. The tool's response lists `https://<bucket>.s3.<VIP pool name>` URLs, which aren't usable addresses.

It has no tools for identity policies, users, keys, schemas or tables beyond listing them.

Things to know:

It starts read-only. Create tools fail with This operation is not available in readonly mode until the server runs with vast-admin-mcp mcp --read-write (mcpsetup --read-write writes that into the client's configuration). Leave it read-only unless you need to create views.
A tenant admin must set it up with a password. Setup with a tenant admin's API token fails, because the setup reads the clusters endpoint, which a tenant admin may not read (403 You do not have permission to perform this action.). With a password it works, and list_clusters_vast then returns an empty list for the tenant admin.
The password is stored by the OS keyring. The config file ~/.vast-admin-mcp/config.json keeps only a keyring: reference. On a headless Linux host without a desktop keyring, Python's fallback keyring is a plain file in your home directory (~/.local/share/python_keyring/), so treat that host as holding the password.
Database traffic doesn't show in the dataflow tools. list_dataflow_vast accepts protocol_filter DATABASE but returned an empty list on our lab cluster, also while an SDK client was connected. Without a filter, the same tool listed that client's connection to the database view.

Admin procedures

Admin task: send to your platform team. These are KB procedures, not CI steps:

Configuring the VAST Cluster for Database Access: VIP pools for VAST DataBase, the database view, the owner user and keys.
Managing Views, Managing View Policies and Managing Identity Policies.
Managing VMS API tokens.

Automation tools: REST, vastpy, Terraform and the admin MCP server compared, with authentication and API versions.
CI/CD and automation for the other sections: VAST Vector Search, VAST Event Broker, VAST DataEngine.
Setup prerequisites: what each identity-policy action allows, and the connection check.
Security: how identity policies affect your queries.