CI/CD and automation

Applies to VAST Cluster 5.5.

An application's storage is more than a bucket: it needs a view with the right protocols, a view policy, an application user, that user's S3 key pair and an identity policy. This page shows how a CI job creates those objects with vastpy (the official Python client for the VMS REST API), the REST API itself and the VAST Terraform provider, and how it then checks the bucket from the data side. These are admin objects, so the examples need a VMS API token for a tenant admin.

For the shared setup (API tokens, TLS, vastpy installation), see CI/CD and automation for VAST DataBase. To choose between the tools, see automation tools compared.

The vastpy script, the REST call, the Terraform configuration and the data-side check were run against VAST Cluster 5.5 with vastpy 0.3.22, Terraform 1.9.8 and provider 4.0.2. The outputs below are from those runs, with names replaced to match the examples.

What each tool can do

Bucket-level settings that the S3 API covers (bucket policy, CORS, tagging, event notifications) can be applied by the application's own key in a deploy step, with boto3, rather than by an admin tool.

Before you start

• A VMS API token for a tenant admin, held as a masked CI secret, and the VMS address. See Before you start on the VAST DataBase page.
• For the NFS part, the client addresses that will mount the export, for the view policy's host rules.

vastpy: an ensure script

This script makes sure an application's bucket exists, with NFS on the same view: an identity policy for the bucket, an application user with a key pair, and a view with S3 and NFSv3. It creates only what's missing, so it's safe to re-run. The view policy (with the S3 Native flavor and host rules) is an existing one, passed by name.

ensure_bucket.py:

"""Ensure an application bucket with NFS access: identity policy, owner user, S3 key pair,
and a view with S3 + NFS. Safe to re-run.
Env: VMS_ADDRESS, VMS_TOKEN (tenant-admin API token), VAST_TENANT.
usage: python ensure_bucket.py <bucket> <app-user> <uid> <view-policy-name>"""
import json, os, sys
from vastpy import VASTClient

bucket, user_name, uid, vpolicy_name = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
client = VASTClient(address=os.environ["VMS_ADDRESS"], token=os.environ["VMS_TOKEN"])

def one(items):
    return items[0] if items else None

tenant = one(client.tenants.get(name=os.environ["VAST_TENANT"]))

# 1. Identity policy: read, write and list on the bucket.
pname = f"{bucket}-rw"
policy = one(client.s3policies.get(name=pname))
if not policy:
    doc = {"Version": "2012-10-17", "Statement": [{"Effect": "Allow",
           "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject",
                      "s3:AbortMultipartUpload", "s3:GetObjectTagging", "s3:PutObjectTagging"],
           "Resource": [f"arn:aws:s3:::{bucket}", f"arn:aws:s3:::{bucket}/*"]}]}
    policy = client.s3policies.post(name=pname, policy=json.dumps(doc), enabled=True,
                                    tenant_id=tenant["id"])
    print(f"created identity policy {pname} (id {policy['id']})")
else:
    print(f"ok identity policy {pname} (id {policy['id']})")

# 2. Application user, with the policy attached. Its UID is the owner NFS clients see.
user = one(client.users.get(name=user_name, local_provider_id=tenant["local_provider_id"]))
if not user:
    user = client.users.post(name=user_name, uid=uid, local_provider_id=tenant["local_provider_id"],
                             s3_policies_ids=[policy["id"]])
    print(f"created user {user_name} (id {user['id']})")
else:
    print(f"ok user {user_name} (id {user['id']})")

# 3. One S3 key pair. The secret is returned only once: write it to a 0600 file, never to the log.
keys = [k for k in (user.get("access_keys") or []) if k.get("enabled")]
if not keys:
    k = client.users[user["id"]].access_keys.post(tenant_id=tenant["id"])
    fd = os.open(f"{user_name}.key.json", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
    with os.fdopen(fd, "w") as f:
        json.dump({"access_key": k["access_key"], "secret_key": k["secret_key"]}, f)
    print(f"created key {k['access_key']} (secret written to {user_name}.key.json)")
else:
    print(f"ok key {keys[0]['access_key']}")

# 4. The view: S3 bucket + NFSv3 on one path. Use "NFS4" for NFSv4.1/4.2.
view = one(client.views.get(path=f"/{bucket}", tenant_id=tenant["id"]))
if not view:
    vp = one(client.viewpolicies.get(name=vpolicy_name, tenant_id=tenant["id"]))
    view = client.views.post(path=f"/{bucket}", bucket=bucket, bucket_owner=user_name,
                             protocols=["S3", "NFS"], policy_id=vp["id"],
                             tenant_id=tenant["id"], create_dir=True)
    print(f"created view /{bucket} (id {view['id']}, protocols {view['protocols']})")
else:
    print(f"ok view /{bucket} (id {view['id']}, protocols {view['protocols']})")

The first run creates everything:

python ensure_bucket.py analytics-landing analytics-app 9202 s3-nfs-policy

created identity policy analytics-landing-rw (id 22)
created user analytics-app (id 19)
created key <access key> (secret written to analytics-app.key.json)
created view /analytics-landing (id 29, protocols ['S3', 'NFS'])

A second run changes nothing:

ok identity policy analytics-landing-rw (id 22)
ok user analytics-app (id 19)
ok key <access key>
ok view /analytics-landing (id 29, protocols ['NFS', 'S3'])

Over NFS, the objects the application writes are owned by the user's UID (9202 here).

Two settings on views.post can't be undone, so add them only on purpose: s3_versioning=True (versioning can't be disabled after the view is created) and locking=True (object locking can't be disabled). Both need a view with S3 as its only protocol, so neither works with this script's S3 + NFS view: VMS refuses versioning with S3 versioning is only allowed for views in which the only protocol is S3..

REST API

The same view with curl, for a pipeline without Python:

# not captured: prints the new view as JSON
curl -sS -X POST "https://$VMS_ADDRESS/api/latest/views/" \
  -H "Authorization: Api-Token $VMS_TOKEN" -H "Content-Type: application/json" \
  --cacert vms-ca.pem \
  -d '{"path": "/analytics-landing", "create_dir": true,
       "protocols": ["S3", "NFS"], "bucket": "analytics-landing",
       "bucket_owner": "analytics-app", "policy_id": 7, "tenant_id": 1}'

The response is the new view as JSON: its id, and the path, bucket, bucket_owner, protocols, policy_id and tenant_id you sent.

Connect with a name that the VMS certificate lists: with an address that isn't in it, curl refuses the connection even with --cacert.

protocols takes NFS (NFSv3), NFS4 (NFSv4.1 and 4.2), SMB (with share), S3 (with bucket and bucket_owner) and ENDPOINT (a template for buckets created with the S3 API). SMB needs a tenant joined to Active Directory; otherwise the request fails with Failed to enable SMB on view, Active Directory provider is not set for the tenant. The API reference is on your cluster at https://<VMS address>/docs/ and in the KB: Create View.

Terraform

Use the provider setup (versions.tf) from the VAST DataBase page. This main.tf creates a bucket with NFS for an application:

# not-run: apply with `terraform apply`
variable "tenant_name" { default = "my-tenant" }
variable "bucket"      { default = "analytics-landing" }
variable "app_user"    { default = "analytics-app" }
variable "app_uid"     { default = 8301 }
variable "nfs_clients" { default = ["10.0.0.0/24"] }
variable "pgp_public_key" { type = string }

data "vastdata_tenant" "this" {
  name = var.tenant_name
}

resource "vastdata_s3_policy" "rw" {
  name      = "${var.bucket}-rw"
  tenant_id = data.vastdata_tenant.this.id
  enabled   = true
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"]
      Resource = ["arn:aws:s3:::${var.bucket}", "arn:aws:s3:::${var.bucket}/*"]
    }]
  })
}

resource "vastdata_user" "app" {
  name              = var.app_user
  uid               = var.app_uid
  local_provider_id = data.vastdata_tenant.this.local_provider_id
  s3_policies_ids   = [vastdata_s3_policy.rw.id]
}

resource "vastdata_user_key" "app" {
  user_id        = vastdata_user.app.id
  tenant_id      = data.vastdata_tenant.this.id
  pgp_public_key = var.pgp_public_key # the secret key is stored only encrypted in state
}

resource "vastdata_view_policy" "app" {
  name      = "${var.bucket}-policy"
  tenant_id = data.vastdata_tenant.this.id
  flavor    = "S3_NATIVE"
  nfs_read_write = var.nfs_clients
}

resource "vastdata_view" "app" {
  path         = "/${var.bucket}"
  bucket       = var.bucket
  bucket_owner = vastdata_user.app.name
  protocols    = ["S3", "NFS"]
  policy_id    = vastdata_view_policy.app.id
  tenant_id    = data.vastdata_tenant.this.id
  create_dir   = true
}

output "access_key" {
  value = vastdata_user_key.app.access_key
}

output "encrypted_secret_key" {
  value = vastdata_user_key.app.encrypted_secret_key
}

With TF_VAR_pgp_public_key set to an ASCII-armored public key, terraform plan ends with Plan: 5 to add, 0 to change, 0 to destroy., and terraform apply prints:

vastdata_s3_policy.rw: Creating...
vastdata_view_policy.app: Creating...
vastdata_s3_policy.rw: Creation complete after 1s [name=analytics-landing-rw]
vastdata_user.app: Creating...
vastdata_view_policy.app: Creation complete after 1s [name=analytics-landing-policy]
vastdata_user.app: Creation complete after 1s [name=analytics-app]
vastdata_user_key.app: Creating...
vastdata_view.app: Creating...
vastdata_user_key.app: Creation complete after 0s
vastdata_view.app: Creation complete after 3s [name=view-32]
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

followed by the access_key output and the PGP-encrypted secret key.

Decide flavor up front. Switching the security flavor of a view that has a bucket isn't supported, although VMS doesn't stop you: it accepted both a flavor change on the view policy and a switch to a policy with another flavor, and either changes whose permissions apply to existing data (concepts). Key pairs can't be imported into Terraform state; see the VAST DataBase page for how the PGP-encrypted secret is handled.

Check from the data side

After the admin step, the pipeline checks the bucket with the application's own key, and applies bucket-level S3 settings it owns, such as tags:

import json
import os

import boto3
from botocore.config import Config

key = json.load(open("analytics-app.key.json"))
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    aws_access_key_id=key["access_key"],
    aws_secret_access_key=key["secret_key"],
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
s3.head_bucket(Bucket="analytics-landing")
s3.put_bucket_tagging(Bucket="analytics-landing", Tagging={"TagSet": [{"Key": "app", "Value": "analytics"}]})
s3.put_object(Bucket="analytics-landing", Key="_deploy/ok", Body=b"ok")
print("bucket ready:", s3.get_object(Bucket="analytics-landing", Key="_deploy/ok")["Body"].read() == b"ok")

It prints bucket ready: True. With Terraform, decrypt encrypted_secret_key with your private PGP key first and write the pair to the key file. For the full permission probe, see access.

See also

• Access: what each object is for.
• KB: Managing Views, Managing View Policies and Managing S3 User Permissions: the same tasks in the Web UI and CLI.