--- title: "CI/CD and automation" slug: "dev-database-operations-ci-cd" description: "Learn how to automate database management with CI jobs, VMS REST API, Terraform, and vastpy for efficient schema and table creation." updated: 2026-10-30T18:19:06Z published: 2026-10-30T18:19:06Z canonical: "docs.vastdata.com/dev-database-operations-ci-cd" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # CI/CD and automation *Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.* A database your application depends on is more than tables. It needs a Database-enabled view, a view policy, an identity policy with the Tabular actions, an application user and that user's S3 key pair. This page shows how a CI job creates and checks those objects with the VMS REST API, `vastpy`, and the VAST Terraform provider, and how to create schemas and tables from a pipeline step. It ends with what the VAST admin MCP server can do for VAST DataBase in interactive, AI-assisted work. For how the four tools compare across VAST DataBase, Vector Search, Event Broker and DataEngine, see [automation tools](/develop/docs/dev-database-reference-automation-tools). The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6, with Terraform 1.16.5, provider `vast-data/vastdata` 4.0.2, `vastpy` 0.3.22, `vast-admin-mcp` 0.2.6 and the `vastdb` SDK 2.1.0. The tenant is `my-tenant`. Your names, IDs and addresses will differ. ## What a CI job can automate | Object | REST API (VMS) | `vastpy` / `vastpy-cli` | Terraform provider 4.0.2 | Admin MCP server 0.2.6 | | --- | --- | --- | --- | --- | | Database (a view with `S3` and `DATABASE`; the bucket is the database name) | `/views/` | Yes | `vastdata_view` | List; create in read-write mode | | View policy | `/viewpolicies/` | Yes | `vastdata_view_policy` | List | | Identity policy with the Tabular actions | `/s3policies/` | Yes | `vastdata_s3_policy` | List | | Application user | `/users/` | Yes | `vastdata_user` | No | | The user's S3 key pair | `/users/<id>/access_keys/` | Yes | `vastdata_user_key` (can't be imported) | List | | Schemas, tables, columns | `/schemas/`, `/tables/`, `/tables/add_columns/`, `/columns/` | Yes | None | None | | Database endpoints of the tenant | | | Data source `vastdata_vast_db_vips` | | The VIP pool that serves VAST DataBase, and the tenant itself, are cluster-admin objects. Ask your platform team for them; they don't belong in an application's pipeline. ## Before you start You need: - **A VMS API token** for a tenant admin, held as a masked CI secret. REST, `vastpy` and Terraform all accept it. The admin MCP server doesn't for a tenant admin (see [below](/develop/docs/dev-database-operations-ci-cd#interactive-work-with-the-admin-mcp-server)). - **Network access** from the runner to the VMS management address on port 443, and, for the SDK smoke test, to a database endpoint (the [setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites) explain the endpoint). - **The VMS CA certificate**, if the job checks TLS (it should). A tenant admin can create the token once with a password, through the REST API. The owner must be a manager in the same tenant, and `expiry_date` takes a duration such as `30d`: ``` curl -sk -u "$VMS_USER:$VMS_PASSWORD" -H "X-Tenant-Name: my-tenant" -H "Content-Type: application/json" \ -X POST "https://$VMS/api/v8/apitokens/" \ -d "{\"name\": \"ci-database\", \"expiry_date\": \"30d\", \"owner\": \"$VMS_USER\"}" | jq . ``` ``` { "id": "", "token": "" } ``` The token is shown only in this response: put it straight into your CI secret store. A token for a user outside the tenant is refused: ``` {"detail":"Manager admin not found, please provide a valid owner username","code":"bad_request"} ``` **Admin task: send to your platform team** if you can't create tokens yourself: [Managing VMS API tokens](/docs-5-5/docs/managing-vms-api-tokens). The examples below use these variables: ``` export VMS="" # curl export VMS_ADDRESS="$VMS" # vastpy and vastpy-cli export VMS_TOKEN="" # vastpy and vastpy-cli export VAST_TENANT="my-tenant" # the ensure script ``` ### TLS The `curl` examples use `-k`, as on the lab cluster, whose VMS certificate is self-signed. In a pipeline, check the certificate: - `curl`: `--cacert <VMS CA file>`. - `vastpy` doesn't check certificates unless you give it a CA file (`VMS_CERT_FILE`, or `cert_file=` in Python). If the certificate doesn't list the address you connect to, which is the case for the VMS's default certificate and an IP address, verification fails with `SSLCertVerificationError`. Set `VMS_CERT_SERVER` to a name the certificate does list: ``` VMS_CERT_FILE=vms-ca.pem VMS_CERT_SERVER=vms.vastdata.com vastpy-cli get views path=/analytics --json | jq -c ".[] | {id, path}" ``` ``` {"id":12,"path":"/analytics"} ``` - Terraform: the provider's `skip_ssl_verify` is set to `true` in the example below, for the lab cluster. Remove it when your VMS certificate is trusted. ### API documentation - The VMS REST API reference for 5.5 in the KB: [https://kb.vastdata.com/api-5-5/apidocs](https://kb.vastdata.com/api-5-5/apidocs), for example [create a view](https://kb.vastdata.com/api-5-5/apidocs/create-view), [create an identity policy](https://kb.vastdata.com/api-5-5/apidocs/create-an-identity-policy), [generate an S3 access key pair](https://kb.vastdata.com/api-5-5/apidocs/generate-s3-access-key-pair-local-user), [create a database schema](https://kb.vastdata.com/api-5-5/apidocs/create-a-database-schema), [create a database table](https://kb.vastdata.com/api-5-5/apidocs/create-a-database-table) and [add columns to a database table](https://kb.vastdata.com/api-5-5/apidocs/add-columns-to-a-database-table). - The cluster's own copy: `https://<VMS address>/docs/` ("VAST Management Service (VMS) REST API Documentation"). `https://<VMS address>/api/` is a Swagger UI, but on 5.5.0-SP6 it lists only a handful of paths (tokens, login banner, LEDs), not the database endpoints. Call `/api/v8/...` or `/api/latest/...` (the same on 5.5). An unversioned `/api/...` call gets the oldest API version's shape; see [API versions](/develop/docs/dev-database-reference-automation-tools#api-versions). ## Terraform The Terraform provider manages everything an application needs to reach its database, except schemas and tables. This configuration creates a database `analytics`: the identity policy, an application user that owns the bucket, its S3 key pair, the view policy, and the view. It outputs the access key, the secret key encrypted to a CI PGP key, and the tenant's database endpoints. `versions.tf`: ``` terraform { required_version = ">= 1.5" required_providers { vastdata = { source = "vast-data/vastdata" version = "4.0.2" } } } # Provider 4.0.2 requires `host` in the configuration: the VASTDATA_HOST environment # variable alone fails validation. Pass it as a variable (TF_VAR_vast_host in CI). variable "vast_host" { type = string } provider "vastdata" { host = var.vast_host skip_ssl_verify = true # lab cluster with a self-signed certificate } ``` `main.tf`: ``` variable "tenant_name" { type = string default = "my-tenant" } variable "database" { type = string default = "analytics" } variable "app_user" { type = string default = "analytics-app" } variable "app_uid" { type = number default = 8201 } variable "pgp_public_key" { description = "ASCII-armored public key of the CI key (gpg --armor --export KEYID)" type = string } data "vastdata_tenant" "this" { name = var.tenant_name } resource "vastdata_s3_policy" "tabular" { name = "${var.database}-tabular" tenant_id = data.vastdata_tenant.this.id enabled = true policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = ["s3:ListBucket", "s3:Tabular*"] Resource = [var.database, "${var.database}/*"] } ] }) } resource "vastdata_user" "app" { name = var.app_user uid = var.app_uid local_provider_id = data.vastdata_tenant.this.local_provider_id allow_create_bucket = true s3_policies_ids = [vastdata_s3_policy.tabular.id] } resource "vastdata_user_key" "app" { user_id = vastdata_user.app.id tenant_id = data.vastdata_tenant.this.id pgp_public_key = var.pgp_public_key # the secret key is stored only encrypted in state } resource "vastdata_view_policy" "db" { name = "${var.database}-policy" tenant_id = data.vastdata_tenant.this.id flavor = "S3_NATIVE" } resource "vastdata_view" "db" { path = "/${var.database}" bucket = var.database bucket_owner = vastdata_user.app.name protocols = ["S3", "DATABASE"] policy_id = vastdata_view_policy.db.id tenant_id = data.vastdata_tenant.this.id create_dir = true } data "vastdata_vast_db_vips" "this" { tenant_id = data.vastdata_tenant.this.id } output "database" { value = vastdata_view.db.bucket } output "access_key" { value = vastdata_user_key.app.access_key } output "encrypted_secret_key" { value = vastdata_user_key.app.encrypted_secret_key } output "db_endpoints" { value = data.vastdata_vast_db_vips.this.vips } ``` The identity policy is the practitioner policy from the [setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites#example-policies). Narrow its actions if the application only reads. ### Plan and apply Set the credentials as environment variables, as a CI job would: ``` export TF_VAR_vast_host="" export VASTDATA_API_TOKEN="" export TF_VAR_pgp_public_key="$(gpg --armor --export ci@example.com)" terraform init -no-color | grep -E "Installing|Installed|initialized" ``` ``` - Installing vast-data/vastdata v4.0.2... - Installed vast-data/vastdata v4.0.2 (self-signed, key ID FADBA165459C78A3) Terraform has been successfully initialized! ``` If `host` isn't in the configuration, `plan` and `validate` stop, even with `VASTDATA_HOST` set: ``` Error: Missing required argument on versions.tf line 13, in provider "vastdata": 13: provider "vastdata" { The argument "host" is required, but no definition was found. ``` `terraform plan -no-color -out tfplan` lists the five resources. It ends with: ``` Plan: 5 to add, 0 to change, 0 to destroy. Changes to Outputs: + access_key = (known after apply) + database = "analytics" + db_endpoints = [ + { + port = 9090 + tenant_id = null + vip = "" }, ] + encrypted_secret_key = (known after apply) ``` `pgp_public_key` must be an ASCII-armored key. In our first run it held a base64 encoding of the binary key, and `apply` created the other four resources, then failed at the key: ``` Error: PrepareCreateResource["user_key"] with vastdata_user_key.app, on main.tf line 54, in resource "vastdata_user_key" "app": 54: resource "vastdata_user_key" "app" { gopenpgp: unable to parse public key: gopenpgp: error in reading key ring: openpgp: invalid argument: no armored data found ``` With the armored key, the next `terraform apply -no-color -auto-approve` created only what was missing. The end of its output: ``` vastdata_user_key.app: Creating... vastdata_user_key.app: Creation complete after 1s Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Outputs: access_key = "" database = "analytics" db_endpoints = tolist([ { "port" = 9090 "tenant_id" = tonumber(null) "vip" = "" }, ]) encrypted_secret_key = < -----END PGP MESSAGE----- EOT ``` With `pgp_public_key` set, the state holds only `encrypted_secret_key`. Without it, the state holds the secret key itself (as a sensitive value), so keep the state in a remote backend with encryption and access control either way: it also holds the user and policy details. ### Check from the data plane Decrypt the secret key with the CI key and run a smoke test with the SDK. `db_smoke.py`: ``` """CI smoke test: connect with the app's key pair and list the database's schemas. usage: python db_smoke.py ; env AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY""" import os, sys import vastdb endpoint, database = sys.argv[1], sys.argv[2] session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"], secret=os.environ["AWS_SECRET_ACCESS_KEY"], timeout=10) with session.transaction() as tx: bucket = tx.bucket(database) print("schemas:", [s.name for s in bucket.schemas()]) ``` ``` AWS_ACCESS_KEY_ID=$(terraform output -raw access_key) \ AWS_SECRET_ACCESS_KEY=$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt) \ python db_smoke.py http:// analytics ``` ``` schemas: [] ``` A new database has no schemas. If this fails, the [setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites#check-that-it-works) map each error to the item to check. ### Bring an existing database under Terraform To manage a view that a script or the Web UI created, write the resource and import the view by path and tenant. Here a script created `/legacy-db` first: ``` vastpy-cli post views path=/legacy-db bucket=legacy-db bucket_owner=analytics-app policy_id=10 protocols=S3 protocols=DATABASE create_dir=true --json | jq -c "{id, path, bucket, protocols, policy_id, tenant_id}" ``` ``` {"id":13,"path":"/legacy-db","bucket":"legacy-db","protocols":["S3","DATABASE"],"policy_id":10,"tenant_id":3} ``` Add the resource (`import.tf`): ``` resource "vastdata_view" "legacy" { path = "/legacy-db" bucket = "legacy-db" bucket_owner = vastdata_user.app.name protocols = ["S3", "DATABASE"] policy_id = vastdata_view_policy.db.id tenant_id = data.vastdata_tenant.this.id } ``` Then import it: ``` terraform import -no-color vastdata_view.legacy "path=/legacy-db,tenant_name=my-tenant" ``` ``` data.vastdata_tenant.this: Reading... data.vastdata_tenant.this: Read complete after 3s [name=my-tenant] data.vastdata_vast_db_vips.this: Reading... vastdata_view.legacy: Importing from ID "path=/legacy-db,tenant_name=my-tenant"... data.vastdata_vast_db_vips.this: Read complete after 0s vastdata_view.legacy: Import prepared! Prepared vastdata_view for import vastdata_view.legacy: Refreshing state... [name=view-13] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. ``` The next `terraform plan` ends with `No changes. Your infrastructure matches the configuration.` **Import before you apply.** If a configuration declares a view whose path already exists and you apply without importing, the provider doesn't fail: it reports `Creation complete` for the existing view, takes it over, and sets its attributes to the configuration's. In our lab, a second configuration that also declared `/legacy-db` changed that view's `bucket_owner` to `other-app` and its `policy_id` to `11`, with no warning. Check that each path is declared in one configuration only. **S3 key pairs can't be imported.** A key created outside Terraform stays outside it: ``` terraform import -no-color vastdata_user_key.existing "access_key=" ``` ``` vastdata_user_key.existing: Importing from ID "access_key="... Error: ImportState["user_key"]: import not supported. This resource has been marked as not importable. ``` Let Terraform create a new key for the user, move the application to it, then disable the old key. ### Drift The provider reverts changes made outside Terraform on the next apply. Here someone removes `DATABASE` from the view: ``` vastpy-cli patch views/12 protocols='["S3"]' --json | jq -c "{id, path, protocols}" ``` ``` {"id":12,"path":"/analytics","protocols":["S3"]} ``` `terraform plan -no-color -out tfplan` shows the change it will undo: ``` Terraform will perform the following actions: # vastdata_view.db will be updated in-place ~ resource "vastdata_view" "db" { id = 12 name = "view-12" ~ protocols = [ + "DATABASE", # (1 unchanged element hidden) ] # (64 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. ``` ``` terraform apply -no-color tfplan ``` ``` vastdata_view.db: Modifying... [name=view-12] vastdata_view.db: Modifications complete after 4s [name=view-12] Apply complete! Resources: 0 added, 1 changed, 0 destroyed. ``` To detect drift without fixing it, for example in a scheduled job, run `terraform plan -detailed-exitcode`: it exits with 2 when there are changes. Some changes VMS refuses outright. A view policy that a Database-enabled view uses can't be switched to the SMB or Mixed Last Wins flavor: ``` vastpy-cli patch viewpolicies/10 flavor=MIXED_LAST_WINS --json ``` ``` Error: response for PATCH api/latest/viewpolicies/10 with params=None failed with error 409 and message {"detail":"a View Policy with SMB or Mixed Last Wins cannot have attached View DATABASE protocol","code":"conflict_error"} ``` ### Destroy `terraform destroy` deletes the key, the user, the identity policy, the view and the view policy. Two things stop it: - **A view with schemas can't be deleted.** Drop the tables and schemas first (see [below](/develop/docs/dev-database-operations-ci-cd#schemas-and-tables-from-a-pipeline-step)). - **A view policy that other views use can't be deleted.** Destroy fails with `409` `Cannot delete view policy that has referenced views` until those views are gone or moved to another policy. Give each database's configuration its own view policy, as the example does. ## vastpy `vastpy` is the official Python client for the VMS REST API: `pip install vastpy` (it depends only on `urllib3`). It builds the URL from attribute and index chaining, so `client.views[12].patch(...)` sends a `PATCH` to `/api/latest/views/12`. It also installs `vastpy-cli`, which takes the same calls as `get|post|patch|delete <endpoint> key=value ...` and reads `VMS_ADDRESS`, `VMS_TOKEN`, `VMS_CERT_FILE` and `VMS_CERT_SERVER` from the environment. Source and documentation: [github.com/vast-data/vastpy](https://github.com/vast-data/vastpy). ### A re-runnable ensure script Use a script instead of Terraform when you don't want to keep state. This one ensures a database exists for an application: it creates whatever is missing of the identity policy, the user, the user's key pair and the view, and leaves what exists alone. The view policy is an existing one, passed by name. `ensure_database.py`: ``` """Ensure a VAST DataBase database exists for an application: identity policy, owner user, S3 key pair, and a view with S3 + DATABASE. Safe to re-run (creates only what is missing). Env: VMS_ADDRESS, VMS_TOKEN (tenant-admin API token), VAST_TENANT. usage: python ensure_database.py """ import json, os, sys from vastpy import VASTClient db, user_name, uid, vpolicy_name = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4] client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"]) def one(items): return items[0] if items else None tenant = one(client.tenants.get(name=os.environ["VAST_TENANT"])) # 1. Identity policy: s3:ListBucket + all Tabular actions on the database bucket. pname = f"{db}-tabular" policy = one(client.s3policies.get(name=pname)) if not policy: doc = {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["s3:ListBucket", "s3:Tabular*"], "Resource": [db, f"{db}/*"]}]} policy = client.s3policies.post(name=pname, policy=json.dumps(doc), enabled=True, tenant_id=tenant["id"]) print(f"created identity policy {pname} (id {policy['id']})") else: print(f"ok identity policy {pname} (id {policy['id']})") # 2. Application user in the tenant's local provider, with the policy attached. user = one(client.users.get(name=user_name, local_provider_id=tenant["local_provider_id"])) if not user: user = client.users.post(name=user_name, uid=uid, local_provider_id=tenant["local_provider_id"], allow_create_bucket=True, s3_policies_ids=[policy["id"]]) print(f"created user {user_name} (id {user['id']})") else: print(f"ok user {user_name} (id {user['id']})") # 3. One S3 key pair. The secret is returned only once: write it to a 0600 file for the CI # secret store, never to the log. keys = [k for k in (user.get("access_keys") or []) if k.get("enabled")] if not keys: k = client.users[user["id"]].access_keys.post(tenant_id=tenant["id"]) fd = os.open(f"{user_name}.key.json", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) with os.fdopen(fd, "w") as f: json.dump({"access_key": k["access_key"], "secret_key": k["secret_key"]}, f) print(f"created key {k['access_key']} (secret written to {user_name}.key.json)") else: print(f"ok key {keys[0]['access_key']}") # 4. The database itself: a view whose bucket is the database name. view = one(client.views.get(path=f"/{db}", tenant_id=tenant["id"])) if not view: vp = one(client.viewpolicies.get(name=vpolicy_name, tenant_id=tenant["id"])) view = client.views.post(path=f"/{db}", bucket=db, bucket_owner=user_name, protocols=["S3", "DATABASE"], policy_id=vp["id"], tenant_id=tenant["id"], create_dir=True) print(f"created view /{db} (id {view['id']}, protocols {view['protocols']})") elif "DATABASE" not in view["protocols"]: view = client.views[view["id"]].patch(protocols=sorted(set(view["protocols"]) | {"DATABASE"})) print(f"added DATABASE to view /{db}: {view['protocols']}") else: print(f"ok view /{db} (id {view['id']}, protocols {view['protocols']})") ``` The first run creates everything: ``` python ensure_database.py orders-db orders-app 8202 db-policy ``` ``` created identity policy orders-db-tabular (id 9) created user orders-app (id 10) created key (secret written to orders-app.key.json) created view /orders-db (id 16, protocols ['S3', 'DATABASE']) ``` A second run changes nothing: ``` python ensure_database.py orders-db orders-app 8202 db-policy ``` ``` ok identity policy orders-db-tabular (id 9) ok user orders-app (id 10) ok key ok view /orders-db (id 16, protocols ['S3', 'DATABASE']) ``` The cluster returns a secret key only when it creates it, so the script writes it to a file readable only by the job's user, for the next step to put into your secret store. Don't print it. The same REST calls with `curl` show what the script built: ``` curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/views/?path=/orders-db" | jq -c ".[] | {id, path, bucket, bucket_owner, protocols, policy_id}" ``` ``` {"id":16,"path":"/orders-db","bucket":"orders-db","bucket_owner":"orders-app","protocols":["S3","DATABASE"],"policy_id":10} ``` ``` curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/s3policies/?name=orders-db-tabular" | jq ".[] | {id, name, enabled, users, policy: (.policy|fromjson)}" ``` ``` { "id": 9, "name": "orders-db-tabular", "enabled": true, "users": [ "orders-app" ], "policy": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:Tabular*" ], "Resource": [ "orders-db", "orders-db/*" ] } ] } } ``` Then check from the data plane with the new key, as for Terraform: ``` AWS_ACCESS_KEY_ID=$(jq -r .access_key orders-app.key.json) AWS_SECRET_ACCESS_KEY=$(jq -r .secret_key orders-app.key.json) python db_smoke.py http:// orders-db ``` ``` schemas: [] ``` ## Schemas and tables from a pipeline step VMS also manages schemas, tables and columns, so a migration step can create them over HTTPS to the VMS with the tenant admin's token, without reaching a database endpoint. Use this for DDL that belongs with the rest of the infrastructure, such as creating a schema and its tables before the first deploy. Keep application-driven DDL (tables an application creates, evolves or drops at run time) in the SDK, which runs with the application's own key and permissions; see [ingest](/develop/docs/dev-database-guides-ingest) and [data modeling](/develop/docs/dev-database-guides-data-modeling). **The SDK needs a Tabular identity policy to open a table created through VMS.** In our lab, the bucket owner's key, without an identity policy that grants the Tabular actions, got `403` when it opened such a table (`vastdb.errors.Forbidden: {'code': 'AccessDenied', 'message': '[req=...]: Permission check failed for @s3_object_name=<schema>/<table> ...`). The application user above has the policy, so it can. ### With vastpy-cli Create a schema and a table. `vastpy-cli post` prints only an empty line when it succeeds: ``` vastpy-cli post schemas database_name=orders-db name=sales vastpy-cli post tables database_name=orders-db schema_name=sales name=orders arrow_schema='[{"name": "order_id", "field": {"column_type": "int64"}}, {"name": "sku", "field": {"column_type": "string"}}, {"name": "amount", "field": {"column_type": "double"}}]' ``` Check the columns: ``` vastpy-cli get columns database_name=orders-db schema_name=sales table_name=orders ``` ``` database_name |schema_name |table_name |name |field |raw_field --------------+------------+-----------+---------+--------------------+----------+ orders-db |sales |orders |order_id |column_type: int64 |int64 orders-db |sales |orders |sku |column_type: string |string orders-db |sales |orders |amount |column_type: double |double ``` `vastpy-cli get schemas database_name=orders-db` and `vastpy-cli get tables database_name=orders-db schema_name=sales` list the schemas and tables (with row counts, size and sorting state) the same way. The application can use the table straight away. `db_check_table.py` opens it with the application's key, inserts a row and reads it back: ``` """Open a table created through VMS with the SDK, insert a row, read it back. usage: python db_check_table.py ; env AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY""" import os, sys import pyarrow as pa import vastdb endpoint, database, schema, table = sys.argv[1:5] session = vastdb.connect(endpoint=endpoint, access=os.environ["AWS_ACCESS_KEY_ID"], secret=os.environ["AWS_SECRET_ACCESS_KEY"]) with session.transaction() as tx: t = tx.bucket(database).schema(schema).table(table) print("columns:", [(f.name, str(f.type)) for f in t.columns()]) t.insert(pa.record_batch([[1], ["A-1"], [2.5]], schema=t.columns())) with session.transaction() as tx: t = tx.bucket(database).schema(schema).table(table) print(t.select().read_all().to_pandas().to_string(index=False)) ``` ``` AWS_ACCESS_KEY_ID=$(jq -r .access_key orders-app.key.json) AWS_SECRET_ACCESS_KEY=$(jq -r .secret_key orders-app.key.json) python db_check_table.py http:// orders-db sales orders 2>&1 | tail -4 ``` ``` columns: [('order_id', 'int64'), ('sku', 'string'), ('amount', 'double')] order_id sku amount 1 A-1 2.5 ``` ### With curl The same endpoints from `curl`. A sorted table takes `sorted_column_names`: ``` curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/schemas/" -d '{"database_name": "orders-db", "name": "staging"}' ``` ``` HTTP 201 ``` ``` curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "ts", "field": {"column_type": "timestamp", "unit": "us"}}, {"name": "kind", "field": {"column_type": "string"}}], "sorted_column_names": ["ts"]}' ``` ``` HTTP 201 ``` ``` curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/tables/?database_name=orders-db&schema_name=staging" | jq -c ".results[] | {name, num_rows, sorting_key_enabled}" ``` ``` {"name":"events","num_rows":0,"sorting_key_enabled":true} ``` Adding columns is a `PATCH`; a `POST` is refused: ``` curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X POST "https://$VMS/api/v8/tables/add_columns/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "payload", "field": {"column_type": "string"}}]}' ``` ``` {"detail":"Method \"POST\" not allowed."}HTTP 405 ``` ``` curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X PATCH "https://$VMS/api/v8/tables/add_columns/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events", "arrow_schema": [{"name": "payload", "field": {"column_type": "string"}}]}' curl -sk -H "Authorization: Api-Token $VMS_TOKEN" "https://$VMS/api/v8/columns/?database_name=orders-db&schema_name=staging&table_name=events" | jq -c ".[] | {name, raw_field}" ``` ``` HTTP 204 {"name":"ts","raw_field":"timestamp[us, tz=UTC]"} {"name":"kind","raw_field":"string"} {"name":"payload","raw_field":"string"} ``` Dropping a table or a schema is a `DELETE` with a JSON body that names it. Drop the tables before their schema: ``` curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X DELETE "https://$VMS/api/v8/tables/" -d '{"database_name": "orders-db", "schema_name": "staging", "name": "events"}' curl -sk -w "HTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" -X DELETE "https://$VMS/api/v8/schemas/" -d '{"database_name": "orders-db", "name": "staging"}' ``` ``` HTTP 204 HTTP 204 ``` When you script against these endpoints, note the response shapes: `/schemas/` and `/columns/` return a JSON list, while `/tables/` returns a paginated object (`count`, `next`, `previous`, `results`). ## A CI job These jobs apply the Terraform configuration above on the main branch and then run the SDK smoke test. They need a runner that can reach the VMS and a database endpoint, a remote Terraform backend (add a `backend` block for the one your team uses; state on the runner is lost after each job), and these secrets and variables: | Name | Kind | Value | | --- | --- | --- | | `VMS_TOKEN` | Secret (masked) | The tenant admin's VMS API token | | `CI_PGP_PRIVATE_KEY` | Secret (masked) | The CI PGP private key (armored), to decrypt the S3 secret key | | `CI_PGP_PUBLIC_KEY` | Variable | The matching armored public key | | `VMS_ADDRESS` | Variable | The VMS address | | `DB_ENDPOINT` | Variable | A database endpoint, such as `http://<vip>:9090` (or use the `db_endpoints` output) | The job files were not run end to end: each step was run on its own against VAST Cluster 5.5.0-SP6, with the commands shown earlier on this page. GitHub Actions (`.github/workflows/database.yml`): ``` # Not run end to end. Each step was exercised separately on VAST Cluster 5.5.0-SP6. name: database on: pull_request: paths: ["infra/database/**"] push: branches: [main] paths: ["infra/database/**"] jobs: terraform: runs-on: self-hosted # needs HTTPS to the VMS and access to a database endpoint defaults: run: working-directory: infra/database env: TF_VAR_vast_host: ${{ vars.VMS_ADDRESS }} TF_VAR_pgp_public_key: ${{ vars.CI_PGP_PUBLIC_KEY }} VASTDATA_API_TOKEN: ${{ secrets.VMS_TOKEN }} steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.16.5 terraform_wrapper: false # so $(terraform output -raw ...) returns the bare value - run: terraform init -input=false -no-color - run: terraform plan -input=false -no-color -out tfplan - if: github.event_name == 'push' run: terraform apply -input=false -no-color tfplan - name: SDK smoke test if: github.event_name == 'push' env: CI_PGP_PRIVATE_KEY: ${{ secrets.CI_PGP_PRIVATE_KEY }} DB_ENDPOINT: ${{ vars.DB_ENDPOINT }} run: | echo "$CI_PGP_PRIVATE_KEY" | gpg --batch --quiet --import python -m pip install --quiet vastdb==2.1.0 export AWS_ACCESS_KEY_ID="$(terraform output -raw access_key)" export AWS_SECRET_ACCESS_KEY="$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt)" echo "::add-mask::$AWS_SECRET_ACCESS_KEY" python db_smoke.py "$DB_ENDPOINT" "$(terraform output -raw database)" ``` GitLab CI (`.gitlab-ci.yml`): ``` # Not run end to end. Each step was exercised separately on VAST Cluster 5.5.0-SP6. # VMS_TOKEN and CI_PGP_PRIVATE_KEY are masked CI/CD variables; VMS_ADDRESS, # CI_PGP_PUBLIC_KEY and DB_ENDPOINT are plain variables. stages: [plan, apply] variables: TF_VAR_vast_host: $VMS_ADDRESS TF_VAR_pgp_public_key: $CI_PGP_PUBLIC_KEY VASTDATA_API_TOKEN: $VMS_TOKEN default: image: name: hashicorp/terraform:1.16.5 entrypoint: [""] before_script: - cd infra/database - terraform init -input=false -no-color plan: stage: plan script: - terraform plan -input=false -no-color -out tfplan artifacts: paths: [infra/database/tfplan] apply: stage: apply rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH script: - terraform apply -input=false -no-color tfplan - apk add --no-cache gnupg python3 py3-pip - echo "$CI_PGP_PRIVATE_KEY" | gpg --batch --quiet --import - python3 -m pip install --quiet --break-system-packages vastdb==2.1.0 - export AWS_ACCESS_KEY_ID="$(terraform output -raw access_key)" - export AWS_SECRET_ACCESS_KEY="$(terraform output -raw encrypted_secret_key | gpg --batch --quiet --decrypt)" - python3 db_smoke.py "$DB_ENDPOINT" "$(terraform output -raw database)" ``` For a script-based pipeline, replace the Terraform steps with `python ensure_database.py ...` and a step that moves the `.key.json` file into your secret store. The ensure script is safe to run on every push. ## Interactive work with the admin MCP server The VAST admin MCP server (`vast-admin-mcp`) lets an AI assistant such as Claude Desktop, Cursor or VS Code query and, if you allow it, change the cluster. It's for interactive work, not for pipelines. Install and setup: [github.com/vast-data/vast-admin-mcp](https://github.com/vast-data/vast-admin-mcp) (`pip install vast-admin-mcp`, Python 3.10 or later; then `vast-admin-mcp setup` and `vast-admin-mcp mcpsetup <client>`). For VAST DataBase, version 0.2.6 can: | Task | Tool | Notes | | --- | --- | --- | | Find database views | `list_views_vast` | Filter by path, for example `/analytics*`. Each view shows its `protocols` (`S3,DATABASE`), bucket and view policy. | | Show view policies and identity policies | `list_viewpolicies_vast`, `list_identity_policies_vast` | The identity policy's JSON is shown as a string. | | Show a user's S3 keys | `list_s3_keys_vast` | Filter by the exact user name; a wildcard such as `analytics*` returned nothing. | | Create a database view | `create_view_vast` with `protocols` `S3,DATABASE`, a `bucket`, `bucket_owner` and `policy` | Only in read-write mode. The view works for the SDK. The tool's response lists `https://<bucket>.s3.<VIP pool name>` URLs, which aren't usable addresses. | It has no tools for identity policies, users, keys, schemas or tables beyond listing them. Things to know: - **It starts read-only.** Create tools fail with `This operation is not available in readonly mode` until the server runs with `vast-admin-mcp mcp --read-write` (`mcpsetup --read-write` writes that into the client's configuration). Leave it read-only unless you need to create views. - **A tenant admin must set it up with a password.** Setup with a tenant admin's API token fails, because the setup reads the clusters endpoint, which a tenant admin may not read (`403` `You do not have permission to perform this action.`). With a password it works, and `list_clusters_vast` then returns an empty list for the tenant admin. - **The password is stored by the OS keyring.** The config file `~/.vast-admin-mcp/config.json` keeps only a `keyring:` reference. On a headless Linux host without a desktop keyring, Python's fallback keyring is a plain file in your home directory (`~/.local/share/python_keyring/`), so treat that host as holding the password. - **Database traffic doesn't show in the dataflow tools.** `list_dataflow_vast` accepts `protocol_filter` `DATABASE` but returned an empty list on our lab cluster, also while an SDK client was connected. Without a filter, the same tool listed that client's connection to the database view. ## Admin procedures **Admin task: send to your platform team.** These are KB procedures, not CI steps: - [Configuring the VAST Cluster for Database Access](/docs-5-5/docs/configuring-the-vast-cluster-for-database-access): VIP pools for VAST DataBase, the database view, the owner user and keys. - [Managing Views](/docs-5-5/docs/managing-views), [Managing View Policies](/docs-5-5/docs/managing-view-policies) and [Managing Identity Policies](/docs-5-5/docs/managing-identity-policies). - [Managing VMS API tokens](/docs-5-5/docs/managing-vms-api-tokens). ## Related pages - [Automation tools](/develop/docs/dev-database-reference-automation-tools): REST, `vastpy`, Terraform and the admin MCP server compared, with authentication and API versions. - CI/CD and automation for the other sections: [VAST Vector Search](/develop/docs/dev-vector-search-operations-ci-cd), [VAST Event Broker](/develop/docs/dev-event-broker-operations-ci-cd), [VAST DataEngine](/develop/docs/dev-dataengine-operations-ci-cd). - [Setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites): what each identity-policy action allows, and the connection check. - [Security](/develop/docs/dev-database-operations-security): how identity policies affect your queries.