CI/CD and automation

Applies to VAST Cluster 5.5.

This page shows how to keep the VAST Catalog's settings in code, and how a pipeline can check the Catalog before it runs a report. The settings are the snapshot schedule and retention, and the user-defined columns. You change them through the VMS REST API or vastpy, the official Python client for it. Changing them is an admin task, so these examples need an admin's API token.

For the shared setup (API tokens, TLS, vastpy), see CI/CD and automation for VAST DataBase. To choose between the tools, see automation tools compared.

None of the admin examples on this page has been run on a 5.5 cluster. They're marked not-run.

What each tool can do

The REST paths are under /api/latest/ (or /api/v8/ on 5.5). The REST API reference for 5.5 is at https://kb.vastdata.com/api-5-5/apidocs, and your cluster has its own copy at https://<VMS address>/docs/.

Auditing settings (protocols, categories, retention, saving to VAST DataBase) are cluster and view-policy settings. Their automation isn't covered here; see Configuring Global Auditing Settings.

Before you start

Admin task: send to your platform team. The settings on this page are the platform team's. If they want the Catalog's settings in Git, these examples are for their pipeline, with their API token. If you're the developer, send them the settings you need (schedule, retention, columns) instead.

The examples use VMS_ADDRESS and VMS_TOKEN, as on CI/CD and automation for VAST DataBase, and check the VMS certificate with VMS_CERT_FILE (TLS).

Read the configuration

import os

from vastpy import VASTClient

client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"],
                    cert_file=os.environ.get("VMS_CERT_FILE"))
for config in client.bigcatalogconfig.get():
    print(config["id"], config["enable"], config["frames"])

With curl:

# not-run (to be exercised)
curl -s --cacert "$VMS_CERT_FILE" -H "Authorization: Api-Token $VMS_TOKEN" \
  "https://$VMS_ADDRESS/api/latest/bigcatalogconfig/" | jq ".[] | {id, enable, frames}"

The output of vastcatalogconfig show shows the shape of a configuration: an ID, a name, whether it's enabled, and its frames, the schedule. Each frame has every (the interval), start-at, keep-local and keep-remote.

Change the schedule

Set a snapshot every 15 minutes (the minimum), and keep copies for two hours. Read the current configuration first, and keep its start-at:

import os

from vastpy import VASTClient

client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"],
                    cert_file=os.environ.get("VMS_CERT_FILE"))
config = client.bigcatalogconfig.get()[0]
frame = {
    "every": "15m",
    "start-at": config["frames"][0]["start-at"],
    "keep-local": "2h",
    "keep-remote": "15m",
}
updated = client.bigcatalogconfig[config["id"]].patch(frames=[frame], enable=True)
print(updated["enable"], updated["frames"])

Check the response, then read the configuration again: the new frame should be there. Retention changes apply to snapshots taken after the change. Which of keep-local and keep-remote sets how long Catalog copies are kept, and which the file-system snapshots, isn't stated here.

User-defined columns

List the user-defined columns:

import os

from vastpy import VASTClient

client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"],
                    cert_file=os.environ.get("VMS_CERT_FILE"))
for column in client.bigcatalogindexedcolumns.get():
    print(column)

Adding and removing columns uses PATCH requests to /bigcatalogindexedcolumns/add/ and the matching remove endpoint. In 5.5 the column_type parameter can be nfs4_xattr, for NFSv4.2 extended attributes; the other parameters are in the REST API reference. New columns are filled in from the next snapshot on.

Check the Catalog is fresh

A report job can check how old the newest Catalog snapshot is before it runs, and fail instead of reporting on stale data. This uses the VAST DataBase SDK and the job's own S3 keys, not an admin token. It reads the time from the snapshot's name, bc_table_<date>_<time>:

from datetime import datetime, timezone

import vastdb

MAX_AGE_MINUTES = 90

session = vastdb.connect()
with session.transaction() as tx:
    names = sorted(
        s.name.split("/.snapshot/")[-1]
        for s in tx.catalog_snapshots()
        if s.name.split("/.snapshot/")[-1].startswith("bc_table_")
    )

if not names:
    raise RuntimeError("no Catalog snapshots: is the Catalog enabled?")
taken = datetime.strptime(names[-1].removeprefix("bc_table_"), "%Y-%m-%d_%H_%M_%S").replace(tzinfo=timezone.utc)
age = (datetime.now(timezone.utc) - taken).total_seconds() / 60
print(f"newest Catalog snapshot {names[-1]}, {age:.0f} minutes old")
if age > MAX_AGE_MINUTES:
    raise RuntimeError("Catalog is stale")

newest Catalog snapshot bc_table_2026-10-09_16_51_02, 2 minutes old

An exception ends the script with a non-zero exit code, which fails the job. Set MAX_AGE_MINUTES to a little more than the Catalog's snapshot interval.

Admin procedures

• KB: Configuring VAST Catalog: the Web UI and VAST CLI.
• KB: vastcatalogconfig create and vastcatalogconfig modify: the schedule format.
• KB: Configuring Global Auditing Settings.

See also

• Access: what to ask your platform team for.
• Concepts: snapshots and freshness.
• CI/CD and automation for VAST DataBase.