--- title: "CI/CD and automation" slug: "dev-datastore-operations-ci-cd" updated: 2026-10-30T18:18:46Z published: 2026-10-30T18:18:46Z canonical: "docs.vastdata.com/dev-datastore-operations-ci-cd" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # CI/CD and automation *Applies to VAST Cluster 5.5.* An application's storage is more than a bucket: it needs a view with the right protocols, a view policy, an application user, that user's S3 key pair and an identity policy. This page shows how a CI job creates those objects with `vastpy` (the official Python client for the VMS REST API), the REST API itself and the VAST Terraform provider, and how it then checks the bucket from the data side. These are admin objects, so the examples need a VMS API token for a tenant admin. For the shared setup (API tokens, TLS, `vastpy` installation), see [CI/CD and automation for VAST DataBase](/develop/docs/dev-database-operations-ci-cd). To choose between the tools, see [automation tools compared](/develop/docs/dev-database-reference-automation-tools). The `vastpy` script, the REST call, the Terraform configuration and the data-side check were run against VAST Cluster 5.5 with `vastpy` 0.3.22, Terraform 1.9.8 and provider 4.0.2. The outputs below are from those runs, with names replaced to match the examples. ## What each tool can do | Object | REST API (VMS) | `vastpy` | Terraform provider 4.0.2 | Admin MCP server 0.2.6 | | --- | --- | --- | --- | --- | | View with S3 Bucket, NFS, SMB | `POST /views/` | `client.views.post(...)` | `vastdata_view` | `create_view_vast` (not checked for S3 + NFS) | | View policy (flavor, host rules) | `POST /viewpolicies/` | `client.viewpolicies.post(...)` | `vastdata_view_policy` | Not checked | | Identity policy | `POST /s3policies/` | `client.s3policies.post(...)` | `vastdata_s3_policy` | Not checked | | User and S3 key pair | `POST /users/`, `POST /users/<id>/access_keys/` | `client.users...` | `vastdata_user`, `vastdata_user_key` | Lists keys only | | Bucket policy, tags, CORS rules, notifications | S3 API (your own key) | boto3 | CORS: `vastdata_view.s3_cors_config` | No | | Objects | S3 API | boto3 | No | No | Bucket-level settings that the S3 API covers (bucket policy, CORS, tagging, event notifications) can be applied by the application's own key in a deploy step, with boto3, rather than by an admin tool. ## Before you start - A VMS API token for a tenant admin, held as a masked CI secret, and the VMS address. See [Before you start](/develop/docs/dev-database-operations-ci-cd#before-you-start) on the VAST DataBase page. - For the NFS part, the client addresses that will mount the export, for the view policy's host rules. ## vastpy: an ensure script This script makes sure an application's bucket exists, with NFS on the same view: an identity policy for the bucket, an application user with a key pair, and a view with S3 and NFSv3. It creates only what's missing, so it's safe to re-run. The view policy (with the S3 Native flavor and host rules) is an existing one, passed by name. `ensure_bucket.py`: ``` """Ensure an application bucket with NFS access: identity policy, owner user, S3 key pair, and a view with S3 + NFS. Safe to re-run. Env: VMS_ADDRESS, VMS_TOKEN (tenant-admin API token), VAST_TENANT. usage: python ensure_bucket.py """ import json, os, sys from vastpy import VASTClient bucket, user_name, uid, vpolicy_name = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4] client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"]) def one(items): return items[0] if items else None tenant = one(client.tenants.get(name=os.environ["VAST_TENANT"])) # 1. Identity policy: read, write and list on the bucket. pname = f"{bucket}-rw" policy = one(client.s3policies.get(name=pname)) if not policy: doc = {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:GetObjectTagging", "s3:PutObjectTagging"], "Resource": [f"arn:aws:s3:::{bucket}", f"arn:aws:s3:::{bucket}/*"]}]} policy = client.s3policies.post(name=pname, policy=json.dumps(doc), enabled=True, tenant_id=tenant["id"]) print(f"created identity policy {pname} (id {policy['id']})") else: print(f"ok identity policy {pname} (id {policy['id']})") # 2. Application user, with the policy attached. Its UID is the owner NFS clients see. user = one(client.users.get(name=user_name, local_provider_id=tenant["local_provider_id"])) if not user: user = client.users.post(name=user_name, uid=uid, local_provider_id=tenant["local_provider_id"], s3_policies_ids=[policy["id"]]) print(f"created user {user_name} (id {user['id']})") else: print(f"ok user {user_name} (id {user['id']})") # 3. One S3 key pair. The secret is returned only once: write it to a 0600 file, never to the log. keys = [k for k in (user.get("access_keys") or []) if k.get("enabled")] if not keys: k = client.users[user["id"]].access_keys.post(tenant_id=tenant["id"]) fd = os.open(f"{user_name}.key.json", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) with os.fdopen(fd, "w") as f: json.dump({"access_key": k["access_key"], "secret_key": k["secret_key"]}, f) print(f"created key {k['access_key']} (secret written to {user_name}.key.json)") else: print(f"ok key {keys[0]['access_key']}") # 4. The view: S3 bucket + NFSv3 on one path. Use "NFS4" for NFSv4.1/4.2. view = one(client.views.get(path=f"/{bucket}", tenant_id=tenant["id"])) if not view: vp = one(client.viewpolicies.get(name=vpolicy_name, tenant_id=tenant["id"])) view = client.views.post(path=f"/{bucket}", bucket=bucket, bucket_owner=user_name, protocols=["S3", "NFS"], policy_id=vp["id"], tenant_id=tenant["id"], create_dir=True) print(f"created view /{bucket} (id {view['id']}, protocols {view['protocols']})") else: print(f"ok view /{bucket} (id {view['id']}, protocols {view['protocols']})") ``` The first run creates everything: ``` python ensure_bucket.py analytics-landing analytics-app 9202 s3-nfs-policy ``` ``` created identity policy analytics-landing-rw (id 22) created user analytics-app (id 19) created key (secret written to analytics-app.key.json) created view /analytics-landing (id 29, protocols ['S3', 'NFS']) ``` A second run changes nothing: ``` ok identity policy analytics-landing-rw (id 22) ok user analytics-app (id 19) ok key ok view /analytics-landing (id 29, protocols ['NFS', 'S3']) ``` Over NFS, the objects the application writes are owned by the user's UID (9202 here). Two settings on `views.post` can't be undone, so add them only on purpose: `s3_versioning=True` (versioning can't be disabled after the view is created) and `locking=True` (object locking can't be disabled). Both need a view with S3 as its only protocol, so neither works with this script's S3 + NFS view: VMS refuses versioning with `S3 versioning is only allowed for views in which the only protocol is S3.`. ## REST API The same view with `curl`, for a pipeline without Python: ``` # not captured: prints the new view as JSON curl -sS -X POST "https://$VMS_ADDRESS/api/latest/views/" \ -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" \ --cacert vms-ca.pem \ -d '{"path": "/analytics-landing", "create_dir": true, "protocols": ["S3", "NFS"], "bucket": "analytics-landing", "bucket_owner": "analytics-app", "policy_id": 7, "tenant_id": 1}' ``` The response is the new view as JSON: its `id`, and the `path`, `bucket`, `bucket_owner`, `protocols`, `policy_id` and `tenant_id` you sent. Connect with a name that the VMS certificate lists: with an address that isn't in it, `curl` refuses the connection even with `--cacert`. `protocols` takes `NFS` (NFSv3), `NFS4` (NFSv4.1 and 4.2), `SMB` (with `share`), `S3` (with `bucket` and `bucket_owner`) and `ENDPOINT` (a template for buckets created with the S3 API). `SMB` needs a tenant joined to Active Directory; otherwise the request fails with `Failed to enable SMB on view, Active Directory provider is not set for the tenant`. The API reference is on your cluster at `https://<VMS address>/docs/` and in the KB: [Create View](https://kb.vastdata.com/api-5-5/apidocs/create-view). ## Terraform Use the provider setup (`versions.tf`) from the [VAST DataBase page](/develop/docs/dev-database-operations-ci-cd#terraform). This `main.tf` creates a bucket with NFS for an application: ``` # not-run: apply with `terraform apply` variable "tenant_name" { default = "my-tenant" } variable "bucket" { default = "analytics-landing" } variable "app_user" { default = "analytics-app" } variable "app_uid" { default = 8301 } variable "nfs_clients" { default = ["10.0.0.0/24"] } variable "pgp_public_key" { type = string } data "vastdata_tenant" "this" { name = var.tenant_name } resource "vastdata_s3_policy" "rw" { name = "${var.bucket}-rw" tenant_id = data.vastdata_tenant.this.id enabled = true policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"] Resource = ["arn:aws:s3:::${var.bucket}", "arn:aws:s3:::${var.bucket}/*"] }] }) } resource "vastdata_user" "app" { name = var.app_user uid = var.app_uid local_provider_id = data.vastdata_tenant.this.local_provider_id s3_policies_ids = [vastdata_s3_policy.rw.id] } resource "vastdata_user_key" "app" { user_id = vastdata_user.app.id tenant_id = data.vastdata_tenant.this.id pgp_public_key = var.pgp_public_key # the secret key is stored only encrypted in state } resource "vastdata_view_policy" "app" { name = "${var.bucket}-policy" tenant_id = data.vastdata_tenant.this.id flavor = "S3_NATIVE" nfs_read_write = var.nfs_clients } resource "vastdata_view" "app" { path = "/${var.bucket}" bucket = var.bucket bucket_owner = vastdata_user.app.name protocols = ["S3", "NFS"] policy_id = vastdata_view_policy.app.id tenant_id = data.vastdata_tenant.this.id create_dir = true } output "access_key" { value = vastdata_user_key.app.access_key } output "encrypted_secret_key" { value = vastdata_user_key.app.encrypted_secret_key } ``` With `TF_VAR_pgp_public_key` set to an ASCII-armored public key, `terraform plan` ends with `Plan: 5 to add, 0 to change, 0 to destroy.`, and `terraform apply` prints: ``` vastdata_s3_policy.rw: Creating... vastdata_view_policy.app: Creating... vastdata_s3_policy.rw: Creation complete after 1s [name=analytics-landing-rw] vastdata_user.app: Creating... vastdata_view_policy.app: Creation complete after 1s [name=analytics-landing-policy] vastdata_user.app: Creation complete after 1s [name=analytics-app] vastdata_user_key.app: Creating... vastdata_view.app: Creating... vastdata_user_key.app: Creation complete after 0s vastdata_view.app: Creation complete after 3s [name=view-32] Apply complete! Resources: 5 added, 0 changed, 0 destroyed. ``` followed by the `access_key` output and the PGP-encrypted secret key. Decide `flavor` up front. Switching the security flavor of a view that has a bucket isn't supported, although VMS doesn't stop you: it accepted both a flavor change on the view policy and a switch to a policy with another flavor, and either changes whose permissions apply to existing data ([concepts](/develop/docs/dev-datastore-concepts-index#what-decides-whether-a-request-is-allowed)). Key pairs can't be imported into Terraform state; see the [VAST DataBase page](/develop/docs/dev-database-operations-ci-cd#terraform) for how the PGP-encrypted secret is handled. ## Check from the data side After the admin step, the pipeline checks the bucket with the application's own key, and applies bucket-level S3 settings it owns, such as tags: ``` import json import os import boto3 from botocore.config import Config key = json.load(open("analytics-app.key.json")) s3 = boto3.client( "s3", endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"], region_name="us-east-1", aws_access_key_id=key["access_key"], aws_secret_access_key=key["secret_key"], config=Config(signature_version="s3v4", s3={"addressing_style": "path"}), ) s3.head_bucket(Bucket="analytics-landing") s3.put_bucket_tagging(Bucket="analytics-landing", Tagging={"TagSet": [{"Key": "app", "Value": "analytics"}]}) s3.put_object(Bucket="analytics-landing", Key="_deploy/ok", Body=b"ok") print("bucket ready:", s3.get_object(Bucket="analytics-landing", Key="_deploy/ok")["Body"].read() == b"ok") ``` It prints `bucket ready: True`. With Terraform, decrypt `encrypted_secret_key` with your private PGP key first and write the pair to the key file. For the full permission probe, see [access](/develop/docs/dev-datastore-operations-access#check-what-your-key-can-do). ## See also - [Access](/develop/docs/dev-datastore-operations-access): what each object is for. - KB: [Managing Views](/docs-5-5/docs/managing-views), [Managing View Policies](/docs-5-5/docs/managing-view-policies) and [Managing S3 User Permissions](/docs-5-5/docs/managing-s3-user-permissions): the same tasks in the Web UI and CLI.