Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

CI/CD and automation

Prev Next

Applies to VAST Cluster 5.5.

This page shows how to release DataEngine functions and pipelines from a Git repository: a repository layout, a deploy script that builds, pushes, registers and deploys with vastde, the same script as a GitHub Actions workflow and a GitLab CI job, and how to promote a release from development to production. It ends with the DataEngine API, from curl and from vastpy, for automation that doesn't use the CLI, and with what the Terraform provider and the VAST admin MCP server cover (for DataEngine, almost nothing).

The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6 with vastde v5.5.0-sp2. The lab's compute cluster is test-cluster. Your names, GUIDs, addresses and times will differ. For a comparison of the automation tools across all four sections, see automation tools.

What to automate with what

Tool What it does for DataEngine
vastde CLI Everything a release needs: build and push images, create functions, triggers and pipelines, update, deploy, read logs and traces. The CI job on this page is built on it.
DataEngine API (REST) The same objects over HTTPS, under https://<VMS address>/api/dataengine/. Use it where the CLI falls short (updating a published function in v5.5.0-sp2), or from tools that aren't shell scripts, such as an orchestrator.
vastpy A thin Python client for the same API.
Terraform provider No DataEngine resources: no functions, triggers or pipelines.
VAST admin MCP server No DataEngine tools.

Lay out the repository

One repository per pipeline, with the function code, a template of the pipeline file and the deploy script:

ingest/
├── .github/workflows/deploy-dataengine.yml   # GitHub Actions (or .gitlab-ci.yml for GitLab)
├── .gitignore
├── deploy.sh                                  # build, push, register, deploy, wait
├── pipeline.yaml.tmpl                         # the pipeline file, with the environment left as variables
└── fn/                                        # the function project (vastde functions init)
    ├── main.py
    └── requirements.txt

The pipeline file is a template because the compute cluster, the namespace and the function revision differ per environment and per release. deploy.sh fills them in with envsubst, and the generated files stay out of Git:

.vast-build-cache/
fn.json
fn-update.json
pipeline.yaml

A pipeline with several functions has one directory per function, as in pipelines as code. Keep secret files out of Git, and generate them in CI from your secrets store; see security.

Configure vastde with environment variables

In CI, don't write a configuration file. Give vastde its connection settings as environment variables, from your CI system's variables and secrets:

Variable Value
VAST_VMS_URL https://<VMS address>
VAST_TENANT Your tenant
VAST_USERNAME A tenant admin or an application user on that tenant
VAST_PASSWORD That user's password (a masked secret)
VAST_BUILDER_IMAGE_URL The builder image, pinned to your cluster's release, for example vastdataorg/vast-builder:v5.5.0-sp2

With these set and no ~/.vast directory, the CLI works. The lab ran every command below with an empty home directory and only these variables (plus the script's own REGISTRY_PUSH, REGISTRY_NAME, CLUSTER and NAMESPACE):

vastde pipelines get thumbnails -o json | jq -c "{name, status}"

{"name":"thumbnails","status":"Ready"}

Set all four connection variables. With one missing (here VAST_TENANT), the CLI looks for a configuration file instead:

env -u VAST_TENANT vastde pipelines list

Cannot read config file: <home>/.vast/config.toml
Call 'vastde config init' to create a new config file

Don't run vastde config init in a CI job to create the file. With the variables set and no file, v5.5.0-sp2 refuses, says a file exists when it doesn't, and writes nothing:

vastde config init --vms-url "$VAST_VMS_URL" --tenant "$VAST_TENANT" --username "$VAST_USERNAME" --password "$VAST_PASSWORD"; echo "exit $?"

Config file found at: <home>/.vast/config.toml
use --force to overwrite
exit 1

See CLI credentials for keeping the password out of files and logs.

The deploy script

deploy.sh does a whole release, and is safe to run again: it creates what's missing and updates what exists. It takes all its settings from the environment. IMAGE_TAG is the commit, so each build has its own tag.

#!/usr/bin/env bash
# Build, push, create-or-update and deploy the ingest function and pipeline.
# Needs docker, vastde, jq, curl and envsubst. All configuration comes from the environment:
#   VAST_VMS_URL VAST_TENANT VAST_USERNAME VAST_PASSWORD VAST_BUILDER_IMAGE_URL
#   REGISTRY_PUSH  host[:port]/prefix you push to       REGISTRY_NAME  DataEngine container registry name
#   CLUSTER  NAMESPACE  IMAGE_TAG (immutable, for example the commit SHA)
set -euo pipefail
FN=ingest; PIPELINE=ingest; TRIGGER=ingest-uploads; REPO=team/ingest
TAG=${IMAGE_TAG:?set IMAGE_TAG}

echo "--- build and push $REGISTRY_PUSH/$REPO:$TAG"
vastde functions build "$REGISTRY_PUSH/$REPO" --target ./fn --image-tag "$TAG" --push

echo "--- trigger"
if vastde triggers get "$TRIGGER" > /dev/null 2>&1; then
  echo "trigger $TRIGGER exists"
else
  vastde triggers create element --name "$TRIGGER" --source-bucket my-bucket \
    --event "ObjectCreated:*" --name-prefix incoming/
fi

echo "--- function"
if vastde functions get "$FN" -o json > fn.json 2> /dev/null; then
  current=$(vastde functions get "$FN" -r -o json | jq -r '.revisions | max_by(.revision_number) | .image_tag')
  if [ "$current" = "$TAG" ]; then
    echo "function $FN already at image tag $TAG"
  else
    # vastde functions update fails with 422 in v5.5.0-sp2, so update over REST
    guid=$(jq -r .guid fn.json)
    jwt=$(curl -sfk -X POST "$VAST_VMS_URL/api/token/$VAST_TENANT" -H "Content-Type: application/json" \
      -d "$(jq -n --arg u "$VAST_USERNAME" --arg p "$VAST_PASSWORD" '{username: $u, password: $p}')" | jq -r .access)
    jq -n --arg tag "$TAG" --arg repo "$REPO" --arg reg "vast:dataengine:container-registries:$REGISTRY_NAME" \
      '{name: "ingest", artifact_type: "image", artifact_source: $repo, image_tag: $tag,
        container_registry_vrn: $reg, revision_description: ("image " + $tag), is_published: true}' > fn-update.json
    curl -sfk -X PUT "$VAST_VMS_URL/api/dataengine/functions/$guid" -H "Authorization: Bearer $jwt" \
      -H "Content-Type: application/json" -d @fn-update.json | jq -c '{name, last_revision_number}'
  fi
else
  vastde functions create --name "$FN" --artifact-type image --artifact-source "$REPO" \
    --image-tag "$TAG" --container-registry "$REGISTRY_NAME" --publish
fi
FN_REVISION=$(vastde functions get "$FN" -o json | jq .last_revision_number)
echo "function $FN revision $FN_REVISION"

echo "--- pipeline"
CLUSTER=$CLUSTER NAMESPACE=$NAMESPACE FN_REVISION=$FN_REVISION envsubst < pipeline.yaml.tmpl > pipeline.yaml
if vastde pipelines get "$PIPELINE" > /dev/null 2>&1; then
  vastde pipelines update "$PIPELINE" --config @pipeline.yaml
else
  vastde pipelines create --config @pipeline.yaml
fi
vastde pipelines deploy "$PIPELINE"

echo "--- wait for Ready"
for i in $(seq 1 60); do
  IFS=$'\t' read -r status reason < <(vastde pipelines get "$PIPELINE" -o json | jq -r '[.status // "-", .reason // "-"] | @tsv')
  echo "$(date -u +%T) $status $reason"
  case "$status" in
    Ready) exit 0 ;;
    Error) exit 1 ;;
  esac
  sleep 5
done
echo "timed out waiting for $PIPELINE"; exit 1

The pipeline template pins the function deployment to the revision the script just registered, so a deploy runs exactly what this commit built (see pinned revisions):

name: ingest
description: deployed by deploy.sh (CI-style)
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:${CLUSTER}
namespace: ${NAMESPACE}
manifest:
  triggers:
    - name: uploads
      vrn: vast:dataengine:triggers:ingest-uploads
  function_deployments:
    - name: ingest
      function_vrn: vast:dataengine:functions:ingest
      revision: ${FN_REVISION}
  links:
    - source: [uploads]
      destination: [ingest]

First run

On the first commit, the script creates everything:

IMAGE_TAG=$(git rev-parse --short HEAD) ./deploy.sh

--- build and push 127.0.0.1:5000/team/ingest:76ba6a6
Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/team/ingest:76ba6a6
App Path: ./fn
Handlers File: main.py
2026/10/08 13:53:17 [Started] Python Builder: 127.0.0.1:5000/team/ingest:76ba6a6
2026/10/08 13:54:02 [Completed] Python Builder: 127.0.0.1:5000/team/ingest:76ba6a6
Build completed: 127.0.0.1:5000/team/ingest:76ba6a6
--- trigger
Trigger created: ingest-uploads
Name: ingest-uploads
Description: N/A
Tags: []
GUID: 514d61d8-e964-4530-bf05-9f0aac2a9630
Owner: {4294967295 vid 0xc0002f4ba0 map[]}
CreatedAt: 2026-10-08T13:54:08Z
UpdatedAt: 2026-10-08T13:54:08Z
VRN: vast:dataengine:triggers:ingest-uploads
Type: Element
Events:
  -: ObjectCreated:*
Topic: vast:dataengine:topics:my-broker/default
SourceBucket: my-bucket
--- function
Function created: ingest
Name: ingest
Tags: []
GUID: cc4408e5-7092-43f7-83a0-e5318d27b7a9
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T13:54:19Z
Updated At: 2026-10-08T13:54:19Z
VRN: vast:dataengine:functions:ingest
Last Revision: 1

function ingest revision 1
--- pipeline
Pipeline created: ingest
Name: ingest
Description: deployed by deploy.sh (CI-style)
Tags: []
GUID: 00a4ccf0-4ef6-406f-83fd-1e25e0c28cae
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T13:54:30Z
Updated At: 2026-10-08T13:54:30Z
VRN: vast:dataengine:pipelines:ingest

Pipeline deployed successfully
--- wait for Ready
13:54:55 InProgress -
13:55:02 InProgress -
13:55:09 Ready Pipeline is ready and active.

The whole run took under two minutes, most of it the build.

A new commit

After a code change and a commit (ca86591), the same command builds a new image, registers it as revision 2 of the function, and deploys a pipeline pinned to it:

IMAGE_TAG=$(git rev-parse --short HEAD) ./deploy.sh

--- build and push 127.0.0.1:5000/team/ingest:ca86591
Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/team/ingest:ca86591
App Path: ./fn
Handlers File: main.py
2026/10/08 13:56:56 [Started] Python Builder: 127.0.0.1:5000/team/ingest:ca86591
2026/10/08 13:57:36 [Completed] Python Builder: 127.0.0.1:5000/team/ingest:ca86591
Build completed: 127.0.0.1:5000/team/ingest:ca86591
--- trigger
trigger ingest-uploads exists
--- function
{"name":"ingest","last_revision_number":2}
function ingest revision 2
--- pipeline
Pipeline updated successfully
Name: ingest
Description: deployed by deploy.sh (CI-style)
Tags: []
GUID: 00a4ccf0-4ef6-406f-83fd-1e25e0c28cae
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T13:54:30Z
Updated At: 2026-10-08T13:58:12Z
VRN: vast:dataengine:pipelines:ingest
Status: Ready

Pipeline deployed successfully
--- wait for Ready
13:58:39 InProgress Pipeline is ready and active.
13:58:48 Ready Pipeline is ready and active.

Each function revision records the commit it was built from:

vastde functions get ingest -r -o json | jq -c ".revisions[] | {revision_number, image_tag, is_published}"

{"revision_number":1,"image_tag":"76ba6a6","is_published":true}
{"revision_number":2,"image_tag":"ca86591","is_published":true}

An upload to incoming/ then ran the new code. The function's init logs its version, and the handler logs each object:

vastde logs get ingest --since 5m --scope user

2026-10-08 13:55:00.96 [ingest] [INFO]  [user] ingest version 1 ready
2026-10-08 13:56:29.55 [ingest] [INFO]  [user] ingest version 1 ready
2026-10-08 13:58:39.54 [ingest] [INFO]  [user] ingest version 2 ready
2026-10-08 13:58:57.95 [ingest] [INFO]  [user] ingest v2: incoming/ci-1.txt

The second version 1 ready line, at 13:56, is from a run with nothing to change; the next section explains it.

What to watch for in a CI job

Update published functions over REST

In vastde v5.5.0-sp2, vastde functions update fails with 422 Unprocessable Entity on any function that has a published revision, and changes nothing. The script therefore sends the update to the DataEngine API itself: PUT /api/dataengine/functions/<guid> with the function's full definition (name, artifact type and source, the new image_tag, the registry VRN, revision_description and is_published: true), authenticated with a token from POST /api/token/<tenant>. The PUT makes the next revision. The cause and a step-by-step version are in publish functions.

Poll on status, not reason

vastde pipelines deploy returns as soon as the cluster accepts the deploy; the pipeline then goes from InProgress to Ready or Error. Wait on the status field only. Right after a deploy, reason can still hold the previous deploy's text, as in the run above (InProgress Pipeline is ready and active.), or a progress message:

vastde pipelines deploy ingest -o json; echo "exit $?"; vastde pipelines get ingest -o json | jq -c "{status, reason}"

Pipeline deployed successfully
exit 0
{"status":"InProgress","reason":"Waiting for pipeline to become ready..."}

Give the loop a timeout, and on Error print reason and the pipeline's logs before failing the job (see observability).

An unchanged run still restarts the function

Running the script again on the same commit skipped the function (its tag hadn't changed), but pipelines update with an identical file still made a new pipeline revision, and the deploy restarted the function: that's the second version 1 ready line in the logs above. After the unchanged run:

vastde pipelines get ingest -o json | jq -c "{status, last_revision_number}"; vastde functions get ingest -o json | jq -c "{last_revision_number}"

{"status":"Ready","last_revision_number":2}
{"last_revision_number":1}

So don't deploy on every push. Run the job only when the function code, the pipeline template or the environment's settings changed, using your CI system's path filters, or by comparing the generated pipeline.yaml with the copy from the last successful deploy. Re-running a job by hand is safe, but it rolls the pods.

-o json output isn't pure JSON

pipelines update -o json prints a plain-text line before the JSON, and pipelines deploy -o json prints only the plain-text line, so don't pipe them straight into jq:

vastde pipelines update ingest --config @pipeline.yaml -o json | head -5; echo "exit ${PIPESTATUS[0]}"

Pipeline updated successfully
{
  "created_at": "2026-10-08T13:54:30.274Z",
  "description": "deployed by deploy.sh (CI-style)",
  "guid": "00a4ccf0-4ef6-406f-83fd-1e25e0c28cae",
exit 0

Rely on the exit status, and read state afterwards with get -o json, which is plain JSON. get exits 1 for an object that doesn't exist, which is what the script's create-or-update branches test:

vastde pipelines get no-such-pipeline -o json; echo "exit $?"

no pipeline found with name or guid: no-such-pipeline
exit 1

One deploy at a time

DataEngine doesn't check versions when it saves a pipeline: the last save wins. Two jobs deploying the same pipeline at once can leave it on either commit. Serialize deploys per pipeline (concurrency in GitHub Actions, resource_group in GitLab), and don't change a pipeline that CI owns from the Web UI; see one source of truth.

Build images in CI

vastde functions build needs a Docker daemon, and pushes with that machine's Docker credentials (docker login), not with the DataEngine registry settings. On a CI runner, that's the runner's own daemon or a Docker-in-Docker (DinD) service. With DinD:

  • Point the CLI at the daemon with DOCKER_HOST. The build uses DOCKER_HOST (and DOCKER_TLS_VERIFY and DOCKER_CERT_PATH for a TLS daemon) like the Docker CLI does. The lab ran a docker:27-dind daemon on a TCP port and built through it.

  • 127.0.0.1 is the DinD daemon, not your host. The build runs inside the DinD daemon's network. Pushing to a registry at 127.0.0.1:5000 that runs on the host failed:

    export DOCKER_HOST=tcp://127.0.0.1:23750
vastde functions build 127.0.0.1:5000/team/ingest --target ./fn --image-tag dind-push --push; echo "exit $?"

    Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/team/ingest:dind-push
App Path: ./fn
Handlers File: main.py
2026/10/08 14:34:29 [Started] Python Builder: 127.0.0.1:5000/team/ingest:dind-push
error building app: executing lifecycle: failed with status code: 1
exit 1

    Run it again with --save-build-log, and the log in fn/ shows why:

    ERROR: failed to initialize analyzer: validating registry read access: failed to ensure registry read access to 127.0.0.1:5000/team/ingest:dind-push: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused; Get "http://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

    Pushing the same build to the host's address on the DinD network, <host address>:5000, worked:

    vastde functions build <host address>:5000/team/ingest --target ./fn --image-tag dind-push --push --save-build-log | tail -2

    2026/10/08 14:45:00 [Started] Python Builder: <host address>:5000/team/ingest:dind-push
2026/10/08 14:45:44 [Completed] Python Builder: <host address>:5000/team/ingest:dind-push
Build log saved to: fn/build_ingest_dind-push_2026-10-08-14-44-59.log
Build completed: <host address>:5000/team/ingest:dind-push

    Use a registry address that the DinD service can reach, which is normally your registry's real host name.

  • Plain HTTP worked for the push, but use HTTPS. The lab registry speaks plain HTTP, and the push to it at a non-loopback address succeeded. DataEngine itself needs a registry served over HTTPS with a valid CA certificate to pull from; see setup prerequisites.

  • A fresh daemon pulls the builder every time. An empty DinD daemon pulled the builder image (2.14 GB) and the run image (740 MB) before its first build. Cache them, or use a runner with a persistent daemon, if build time matters.

  • Use an amd64 runner. Builds are linux/amd64 only.

  • The runner needs Docker Hub and PyPI, or mirrors of them: the build pulls its run image and installs your requirements.txt. See build and test locally.

GitHub Actions

This workflow runs deploy.sh on every push to main. It wasn't run on GitHub; every step it runs was run on the lab machine (above), except the CLI download and docker login. Set the repository variables and secrets named in the comments.

# Repository variables (Settings > Secrets and variables > Actions > Variables):
#   VAST_VMS_URL    https://<VMS address>
#   VAST_TENANT     tenant name
#   VAST_USERNAME   tenant admin or application user for that tenant
#   REGISTRY_PUSH   registry host[:port] you push images to; the DataEngine container registry
#                   REGISTRY_NAME must point at the same registry
#   REGISTRY_NAME   DataEngine container registry name (vastde container-registries list)
#   CLUSTER         DataEngine Kubernetes cluster name (vastde compute-clusters list)
#   NAMESPACE       namespace in that cluster (vastde compute-clusters get <cluster>)
#   REGISTRY_USER   user for docker login
# Repository secrets:
#   VAST_PASSWORD, REGISTRY_PASSWORD
#
# The job needs network access to VMS, to the registry, to Docker Hub (the builder's run image)
# and to PyPI (dependencies are installed during the build). For a VMS on a private network use a
# self-hosted runner: runs-on: [self-hosted, linux, x64]. The build is linux/amd64 only.
name: deploy-dataengine

on:
  push:
    branches: [main]

# One deploy at a time per pipeline: the API has no optimistic locking.
concurrency:
  group: dataengine-ingest
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-24.04
    env:
      VAST_VMS_URL: ${{ vars.VAST_VMS_URL }}
      VAST_TENANT: ${{ vars.VAST_TENANT }}
      VAST_USERNAME: ${{ vars.VAST_USERNAME }}
      VAST_PASSWORD: ${{ secrets.VAST_PASSWORD }}
      VAST_BUILDER_IMAGE_URL: vastdataorg/vast-builder:v5.5.0-sp2
      REGISTRY_PUSH: ${{ vars.REGISTRY_PUSH }}
      REGISTRY_NAME: ${{ vars.REGISTRY_NAME }}
      CLUSTER: ${{ vars.CLUSTER }}
      NAMESPACE: ${{ vars.NAMESPACE }}
    steps:
      - uses: actions/checkout@v4

      - name: Install the vastde CLI (pin the version that matches the cluster)
        run: |
          sudo curl -fsSL -o /usr/local/bin/vastde \
            https://github.com/vast-data/dataengine-cli/releases/download/v5.5.0-sp2/vastde_linux_amd64
          sudo chmod +x /usr/local/bin/vastde
          vastde version

      - name: Log in to the registry (vastde functions build --push uses ~/.docker/config.json)
        uses: docker/login-action@v3
        with:
          registry: ${{ vars.REGISTRY_PUSH }}
          username: ${{ vars.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Build, push, create or update, deploy, wait for Ready
        env:
          IMAGE_TAG: ${{ github.sha }}
        run: ./deploy.sh

To run only when something deployable changed, add paths: under push: (for example fn/**, pipeline.yaml.tmpl, deploy.sh).

GitLab CI

The same job for GitLab, using a docker:dind service over TLS. It wasn't run on GitLab. On the lab machine, the build ran through a DinD daemon over plain TCP, not TLS.

# CI/CD variables (Settings > CI/CD > Variables; mask and protect the passwords):
#   VAST_VMS_URL, VAST_TENANT, VAST_USERNAME, VAST_PASSWORD (masked)
#   REGISTRY_PUSH (registry host[:port]; the DataEngine registry REGISTRY_NAME points at it),
#   REGISTRY_NAME, REGISTRY_USER, REGISTRY_PASSWORD (masked), CLUSTER, NAMESPACE
#
# vastde functions build needs a Docker daemon. This job uses the docker:dind service over TLS;
# the CLI's builder picks up DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH from the
# environment. The build runs with host networking inside the dind daemon, so the registry must be
# reachable from the dind service. Use an amd64 runner: the build is linux/amd64 only.
stages: [deploy]

deploy-dataengine:
  stage: deploy
  image: docker:27
  services:
    - name: docker:27-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: /certs
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: /certs/client
    VAST_BUILDER_IMAGE_URL: vastdataorg/vast-builder:v5.5.0-sp2
    IMAGE_TAG: $CI_COMMIT_SHORT_SHA
  # One deploy at a time per pipeline: the API has no optimistic locking.
  resource_group: dataengine-ingest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  before_script:
    # deploy.sh needs bash, curl, jq and envsubst (gettext); the Python-version check needs unzip
    - apk add --no-cache bash curl jq gettext unzip
    - curl -fsSL -o /usr/local/bin/vastde https://github.com/vast-data/dataengine-cli/releases/download/v5.5.0-sp2/vastde_linux_amd64
    - chmod +x /usr/local/bin/vastde
    - vastde version
    - echo "$REGISTRY_PASSWORD" | docker login "$REGISTRY_PUSH" -u "$REGISTRY_USER" --password-stdin
  script:
    - ./deploy.sh

To run only on deployable changes, add changes: to the rule (for example fn/**/*, pipeline.yaml.tmpl, deploy.sh).

Image tags and promotion

Tag images with the commit, and never reuse a tag. Each function revision records an image tag, so a tag that moves means a revision no longer says what it runs. A commit SHA, as in the script, ties the image, the function revision and the Git history together. Never deploy latest. See name images and tags.

Promote the image, not the source. Build once, in the first environment, and deploy the same tag to the next ones. Rebuilding for production installs your dependencies again, and can produce a different image from the one you tested.

Give each environment its own namespace or cluster. A pipeline's compute cluster and namespace are fixed when you create it, and a namespace belongs to one tenant. Ask your platform team for a separate tenant, with its own namespace, for each environment, or a separate cluster; see setup prerequisites. Then:

  • Keep one CI variable set per environment (VAST_VMS_URL, VAST_TENANT, the credentials, REGISTRY_NAME, CLUSTER, NAMESPACE), for example as GitHub environments or GitLab environment-scoped variables. The template and script stay the same.
  • Run the script against development on every merge, and against test and production from a manual approval or a release tag, with IMAGE_TAG set to the tag that passed the previous stage. Skip the build step for those stages, or make the build a separate job, so that only the first stage builds.
  • If production pulls from a different registry, copy the image there with its tag unchanged before you run the script.
  • Pin function revisions in the pipeline file, as the template does, so that a deploy runs exactly the revision you registered from that tag.

The DataEngine API

DataEngine has its own API, served by VMS under https://<VMS address>/api/dataengine/ (also under /api/latest/dataengine/). Use it from tools that aren't shell scripts, and for anything the CLI can't do. Its references:

  • The KB's VAST API reference, with the DataEngine API under VAST DataEngine API.
  • Your cluster's own OpenAPI specs: https://<VMS address>/api/dataengine/openapi.json (functions, triggers, pipelines, deploy, registries) and https://<VMS address>/api/dataengine/telemetries/openapi.json (logs, traces, metrics). They need the same authentication as the API.
  • Your cluster's VMS REST API documentation at https://<VMS address>/docs/ and its Swagger UI at https://<VMS address>/api/, for the VMS objects around DataEngine.

The two specs, from the lab:

curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/openapi.json" | jq "{openapi, info, paths: (.paths | length)}"

{
  "openapi": "3.1.0",
  "info": {
    "title": "Provisioning Service",
    "description": "Provisioning Service",
    "version": "1.4.0"
  },
  "paths": 52
}

curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/telemetries/openapi.json" | jq "{openapi, info, paths: (.paths | keys)}"

{
  "openapi": "3.1.0",
  "info": {
    "title": "Telemetries Query Service",
    "description": "API for querying traces, logs and metrics from DB",
    "version": "1.4.0"
  },
  "paths": [
    "/",
    "/count-functions",
    "/logs",
    "/metrics",
    "/metrics/names",
    "/reports/{report_id}/execute",
    "/span-logs",
    "/trace-tree",
    "/traces"
  ]
}

The examples use curl -k, because the lab's VMS has a self-signed certificate. Leave -k out when your machine trusts the VMS certificate.

Authenticate

Use a tenant admin or an application user on your tenant, in one of two ways:

  • Basic authentication with X-Tenant-Name. Send the user and password, and name the tenant in an X-Tenant-Name header:

    curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" | jq ".data[] | {name, guid, status, reason, last_revision_number}"

    {
  "name": "thumbnails",
  "guid": "c7f63133-3438-4cae-af6a-91291041f1b6",
  "status": "Ready",
  "reason": "Pipeline is ready and active.",
  "last_revision_number": 5
}

    Without the header, the right password is refused with the same message as a wrong one:

    curl -sk -w "\nHTTP %{http_code}\n" -u "$VAST_USERNAME:$VAST_PASSWORD" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails"

    {"detail":"Invalid username/password."}
HTTP 403

  • A bearer token from POST /api/token/<tenant>. Exchange the user and password for an access token once, then send it as Authorization: Bearer. The tenant is in the path, so no X-Tenant-Name is needed:

    export TOKEN=$(curl -sk -X POST "$VAST_VMS_URL/api/token/$VAST_TENANT" -H "Content-Type: application/json" -d "{\"username\": \"$VAST_USERNAME\", \"password\": \"$VAST_PASSWORD\"}" | jq -r .access)
curl -sk -H "Authorization: Bearer $TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" | jq -c ".data[] | {name, status}"

    {"name":"thumbnails","status":"Ready"}

    The response also has a refresh token. Use /api/token/<tenant>, not /api/token/: for a tenant user, /api/token/ fails:

    curl -sk -X POST "$VAST_VMS_URL/api/token/" -H "Content-Type: application/json" -d "{\"username\": \"$VAST_USERNAME\", \"password\": \"$VAST_PASSWORD\"}"

    {"detail":"Authentication failure, user: <user>"}

VMS API tokens don't work for the provisioning API (functions, triggers, pipelines, deploy) in 5.5.0-SP6. A token that works for the VMS API is refused:

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_API_TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails"

{"detail":"{\"authorization\": \"Value error, Invalid Api-Token\"}"}
HTTP 422

The telemetries API (logs, traces, metrics) does accept one, which suits a read-only monitoring job:

curl -sk -o /dev/null -w "%{http_code} %{content_type}\n" -H "Authorization: Api-Token $VMS_API_TOKEN" "$VAST_VMS_URL/api/dataengine/telemetries/logs?pipeline_name=thumbnails&limit=2"

200 application/json

So a CI job that deploys needs a user and password, held as a masked secret. Use an application user with only the DataEngine permissions the job needs; see security.

Deploy and wait

The deploy endpoint is POST /api/dataengine/pipeline/<guid>/deploy, with pipeline in the singular. The plural path, which every other pipeline call uses, returns 404:

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Bearer $TOKEN" -X POST "$VAST_VMS_URL/api/dataengine/pipelines/bc7b6df0-4dbc-4c69-94b7-61153607b1c3/deploy"

{"detail":"Not Found"}
HTTP 404

curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Bearer $TOKEN" -X POST "$VAST_VMS_URL/api/dataengine/pipeline/bc7b6df0-4dbc-4c69-94b7-61153607b1c3/deploy"


HTTP 200

A successful deploy returns an empty body. Then poll the pipeline's status, as with the CLI:

for i in $(seq 1 30); do s=$(curl -sk -H "Authorization: Bearer $TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=orders-rest" | jq -c ".data[0] | {status, reason}"); echo "$(date -u +%T) $s"; case "$s" in *\"Ready\"*|*\"Error\"*) break;; esac; sleep 5; done

13:45:56 {"status":"InProgress","reason":null}
13:46:01 {"status":"InProgress","reason":null}
13:46:07 {"status":"InProgress","reason":null}
13:46:13 {"status":"Ready","reason":"Pipeline is ready and active."}

Creating triggers (POST /triggers/element, POST /triggers/schedule), functions (POST /functions) and pipelines (POST /pipelines, with the pipeline file as JSON) works the same way; for the request bodies, see create a new pipeline, create a new function and create a new element trigger. To look up a container registry by name, list them and filter on the client: GET /container-registries ignores a name parameter and returns them all.

For driving pipelines from an orchestrator, see orchestrators.

Cluster admins

A cluster admin isn't tied to a tenant, so names it on every call with a tenant_name query parameter, on reads and writes alike, for example POST /api/dataengine/triggers/schedule?tenant_name=<tenant>. The tenant isn't accepted in the request body (422, "tenant_name": "Extra inputs are not permitted"). With vastpy-cli (see below), logged in as a cluster admin:

vastpy-cli get dataengine/pipelines name=orders

Error: response for GET api/latest/dataengine/pipelines with params=[('name', 'orders')] failed with error 400 and message {"detail":"No tenant was provided in request"}

vastpy-cli get dataengine/pipelines name=orders tenant_name=<tenant> --json | jq -c ".data[]? | {name, status, tenant_guid} // ."

{"name":"orders","status":"Ready","tenant_guid":"<tenant GUID>"}

Most CI jobs should use a tenant user instead.

vastpy

vastpy (from PyPI) reaches the DataEngine API by path: client.dataengine.pipelines is /api/latest/dataengine/pipelines. Log in with a user, password and tenant; as with REST, a VMS API token is refused for anything but telemetries. This script creates or updates a pipeline from its file, deploys it and waits:

"""Create or update a DataEngine pipeline from pipeline.yaml, deploy it, and wait for Ready.
Env: VMS_ADDRESS, VMS_USER, VMS_PASSWORD, VMS_TENANT_NAME (tenant admin).
The DataEngine API doesn't accept VMS API tokens, so this uses username/password."""
import os, sys, time
import yaml
from vastpy import VASTClient

client = VASTClient(address=os.environ["VMS_ADDRESS"], user=os.environ["VMS_USER"],
                    password=os.environ["VMS_PASSWORD"], tenant=os.environ["VMS_TENANT_NAME"])
de = client.dataengine
spec = yaml.safe_load(open(sys.argv[1] if len(sys.argv) > 1 else "pipeline.yaml"))

found = de.pipelines.get(name=spec["name"])["data"]
if found:
    guid = found[0]["guid"]
    de.pipelines[guid].put(**spec)              # full update: new pipeline revision
    print(f"updated {spec['name']} ({guid})")
else:
    guid = de.pipelines.post(**spec)["guid"]
    print(f"created {spec['name']} ({guid})")

# deploy.post() with no arguments sends an empty chunked body that the server rejects
# (400 "Invalid HTTP request received."); request("POST") sends no body.
de.pipeline[guid].deploy.request("POST")        # POST .../pipeline/<guid>/deploy/
print("deploy requested")

deadline = time.time() + 300
while True:
    p = de.pipelines[guid].get()
    print(f"status={p['status']} reason={p['reason']} revision={p['last_revision_number']}")
    if p["status"] in ("Ready", "Error"):
        break
    if time.time() > deadline:
        sys.exit("timed out waiting for the pipeline")
    time.sleep(10)
sys.exit(0 if p["status"] == "Ready" else 1)

Run against an existing pipeline, orders (the script needs pip install vastpy pyyaml):

VMS_ADDRESS="<VMS address>" VMS_USER="$VAST_USERNAME" VMS_PASSWORD="$VAST_PASSWORD" VMS_TENANT_NAME="$VAST_TENANT" python deploy_pipeline.py pipeline.yaml

updated orders (a763a602-0c97-45ea-adfa-a84f338fef50)
deploy requested
status=InProgress reason=Pipeline is ready and active. revision=2
status=Ready reason=Pipeline is ready and active. revision=2

Again, status changed while reason still showed the last deploy's text.

Calls with no body need .request(). vastpy's .post() and .delete() with no arguments send an empty body that the server rejects. With de.pipeline[guid].deploy.post() the deploy fails:

vastpy.RESTFailure: response for POST api/latest/dataengine/pipeline/a763a602-0c97-45ea-adfa-a84f338fef50/deploy with params=None failed with error 400 and message "Invalid HTTP request received."

Use .request("POST") for a deploy and .request("DELETE") for a delete, for example de.pipelines[guid].request("DELETE"). vastpy-cli delete fails the same way, so delete from Python.

vastpy-cli reads VMS_USER, VMS_PASSWORD and VMS_TENANT_NAME, which is enough for quick reads in a job:

VMS_ADDRESS="<VMS address>" VMS_USER="$VAST_USERNAME" VMS_PASSWORD="$VAST_PASSWORD" VMS_TENANT_NAME="$VAST_TENANT" vastpy-cli get dataengine/pipelines name=orders --json | jq -c ".data[] | {name, status, reason}"

{"name":"orders","status":"Ready","reason":"Pipeline is ready and active."}

A cluster admin passes tenant_name=<tenant> as a parameter (above) instead of setting VMS_TENANT_NAME, which with a cluster-admin login is refused with 403 Invalid username/password.

Terraform and MCP

  • Terraform. The VAST Terraform provider (4.0.2) has no resources for DataEngine functions, triggers or pipelines. What it has around DataEngine is for your platform team: compute clusters, and a tenant's data_engine_enabled setting. Use the CLI or the API for everything on this page. The views and topics a pipeline reads from can be Terraform-managed; see the DataBase and VAST Event Broker CI/CD pages.
  • MCP. The VAST admin MCP server has no DataEngine tools: none of its tools lists or changes functions, triggers or pipelines.

See also

Related articles