--- title: "CI/CD and automation" slug: "dev-dataengine-operations-ci-cd" description: "Learn to deploy DataEngine functions and pipelines from Git, automate with vastde, and manage releases effectively across environments." tags: ["DataEngine"] updated: 2026-10-30T18:19:06Z published: 2026-10-30T18:19:06Z canonical: "docs.vastdata.com/dev-dataengine-operations-ci-cd" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # CI/CD and automation *Applies to VAST Cluster 5.5.* This page shows how to release DataEngine functions and pipelines from a Git repository: a repository layout, a deploy script that builds, pushes, registers and deploys with `vastde`, the same script as a GitHub Actions workflow and a GitLab CI job, and how to promote a release from development to production. It ends with the DataEngine API, from `curl` and from `vastpy`, for automation that doesn't use the CLI, and with what the Terraform provider and the VAST admin MCP server cover (for DataEngine, almost nothing). The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6 with `vastde` v5.5.0-sp2. The lab's compute cluster is `test-cluster`. Your names, GUIDs, addresses and times will differ. For a comparison of the automation tools across all four sections, see [automation tools](/develop/docs/dev-database-reference-automation-tools). ## What to automate with what | Tool | What it does for DataEngine | | --- | --- | | `vastde` CLI | Everything a release needs: build and push images, create functions, triggers and pipelines, update, deploy, read logs and traces. The CI job on this page is built on it. | | DataEngine API (REST) | The same objects over HTTPS, under `https://<VMS address>/api/dataengine/`. Use it where the CLI falls short (updating a published function in v5.5.0-sp2), or from tools that aren't shell scripts, such as an orchestrator. | | `vastpy` | A thin Python client for the same API. | | Terraform provider | No DataEngine resources: no functions, triggers or pipelines. | | VAST admin MCP server | No DataEngine tools. | ## Lay out the repository One repository per pipeline, with the function code, a template of the pipeline file and the deploy script: ``` ingest/ ├── .github/workflows/deploy-dataengine.yml # GitHub Actions (or .gitlab-ci.yml for GitLab) ├── .gitignore ├── deploy.sh # build, push, register, deploy, wait ├── pipeline.yaml.tmpl # the pipeline file, with the environment left as variables └── fn/ # the function project (vastde functions init) ├── main.py └── requirements.txt ``` The pipeline file is a template because the compute cluster, the namespace and the function revision differ per environment and per release. `deploy.sh` fills them in with `envsubst`, and the generated files stay out of Git: ``` .vast-build-cache/ fn.json fn-update.json pipeline.yaml ``` A pipeline with several functions has one directory per function, as in [pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code#lay-out-a-repository). Keep secret files out of Git, and generate them in CI from your secrets store; see [security](/develop/docs/dev-dataengine-operations-security#secrets). ## Configure `vastde` with environment variables In CI, don't write a configuration file. Give `vastde` its connection settings as environment variables, from your CI system's variables and secrets: | Variable | Value | | --- | --- | | `VAST_VMS_URL` | `https://<VMS address>` | | `VAST_TENANT` | Your tenant | | `VAST_USERNAME` | A tenant admin or an application user on that tenant | | `VAST_PASSWORD` | That user's password (a masked secret) | | `VAST_BUILDER_IMAGE_URL` | The builder image, pinned to your cluster's release, for example `vastdataorg/vast-builder:v5.5.0-sp2` | With these set and no `~/.vast` directory, the CLI works. The lab ran every command below with an empty home directory and only these variables (plus the script's own `REGISTRY_PUSH`, `REGISTRY_NAME`, `CLUSTER` and `NAMESPACE`): ``` vastde pipelines get thumbnails -o json | jq -c "{name, status}" ``` ``` {"name":"thumbnails","status":"Ready"} ``` Set all four connection variables. With one missing (here `VAST_TENANT`), the CLI looks for a configuration file instead: ``` env -u VAST_TENANT vastde pipelines list ``` ``` Cannot read config file: /.vast/config.toml Call 'vastde config init' to create a new config file ``` Don't run `vastde config init` in a CI job to create the file. With the variables set and no file, v5.5.0-sp2 refuses, says a file exists when it doesn't, and writes nothing: ``` vastde config init --vms-url "$VAST_VMS_URL" --tenant "$VAST_TENANT" --username "$VAST_USERNAME" --password "$VAST_PASSWORD"; echo "exit $?" ``` ``` Config file found at: /.vast/config.toml use --force to overwrite exit 1 ``` See [CLI credentials](/develop/docs/dev-dataengine-operations-security#cli-credentials) for keeping the password out of files and logs. ## The deploy script `deploy.sh` does a whole release, and is safe to run again: it creates what's missing and updates what exists. It takes all its settings from the environment. `IMAGE_TAG` is the commit, so each build has its own tag. ``` #!/usr/bin/env bash # Build, push, create-or-update and deploy the ingest function and pipeline. # Needs docker, vastde, jq, curl and envsubst. All configuration comes from the environment: # VAST_VMS_URL VAST_TENANT VAST_USERNAME VAST_PASSWORD VAST_BUILDER_IMAGE_URL # REGISTRY_PUSH host[:port]/prefix you push to REGISTRY_NAME DataEngine container registry name # CLUSTER NAMESPACE IMAGE_TAG (immutable, for example the commit SHA) set -euo pipefail FN=ingest; PIPELINE=ingest; TRIGGER=ingest-uploads; REPO=team/ingest TAG=${IMAGE_TAG:?set IMAGE_TAG} echo "--- build and push $REGISTRY_PUSH/$REPO:$TAG" vastde functions build "$REGISTRY_PUSH/$REPO" --target ./fn --image-tag "$TAG" --push echo "--- trigger" if vastde triggers get "$TRIGGER" > /dev/null 2>&1; then echo "trigger $TRIGGER exists" else vastde triggers create element --name "$TRIGGER" --source-bucket my-bucket \ --event "ObjectCreated:*" --name-prefix incoming/ fi echo "--- function" if vastde functions get "$FN" -o json > fn.json 2> /dev/null; then current=$(vastde functions get "$FN" -r -o json | jq -r '.revisions | max_by(.revision_number) | .image_tag') if [ "$current" = "$TAG" ]; then echo "function $FN already at image tag $TAG" else # vastde functions update fails with 422 in v5.5.0-sp2, so update over REST guid=$(jq -r .guid fn.json) jwt=$(curl -sfk -X POST "$VAST_VMS_URL/api/token/$VAST_TENANT" -H "Content-Type: application/json" \ -d "$(jq -n --arg u "$VAST_USERNAME" --arg p "$VAST_PASSWORD" '{username: $u, password: $p}')" | jq -r .access) jq -n --arg tag "$TAG" --arg repo "$REPO" --arg reg "vast:dataengine:container-registries:$REGISTRY_NAME" \ '{name: "ingest", artifact_type: "image", artifact_source: $repo, image_tag: $tag, container_registry_vrn: $reg, revision_description: ("image " + $tag), is_published: true}' > fn-update.json curl -sfk -X PUT "$VAST_VMS_URL/api/dataengine/functions/$guid" -H "Authorization: Bearer $jwt" \ -H "Content-Type: application/json" -d @fn-update.json | jq -c '{name, last_revision_number}' fi else vastde functions create --name "$FN" --artifact-type image --artifact-source "$REPO" \ --image-tag "$TAG" --container-registry "$REGISTRY_NAME" --publish fi FN_REVISION=$(vastde functions get "$FN" -o json | jq .last_revision_number) echo "function $FN revision $FN_REVISION" echo "--- pipeline" CLUSTER=$CLUSTER NAMESPACE=$NAMESPACE FN_REVISION=$FN_REVISION envsubst < pipeline.yaml.tmpl > pipeline.yaml if vastde pipelines get "$PIPELINE" > /dev/null 2>&1; then vastde pipelines update "$PIPELINE" --config @pipeline.yaml else vastde pipelines create --config @pipeline.yaml fi vastde pipelines deploy "$PIPELINE" echo "--- wait for Ready" for i in $(seq 1 60); do IFS=$'\t' read -r status reason < <(vastde pipelines get "$PIPELINE" -o json | jq -r '[.status // "-", .reason // "-"] | @tsv') echo "$(date -u +%T) $status $reason" case "$status" in Ready) exit 0 ;; Error) exit 1 ;; esac sleep 5 done echo "timed out waiting for $PIPELINE"; exit 1 ``` The pipeline template pins the function deployment to the revision the script just registered, so a deploy runs exactly what this commit built (see [pinned revisions](/develop/docs/dev-dataengine-guides-publish-functions#pinned-run-exactly-the-revision-you-tested)): ``` name: ingest description: deployed by deploy.sh (CI-style) kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:${CLUSTER} namespace: ${NAMESPACE} manifest: triggers: - name: uploads vrn: vast:dataengine:triggers:ingest-uploads function_deployments: - name: ingest function_vrn: vast:dataengine:functions:ingest revision: ${FN_REVISION} links: - source: [uploads] destination: [ingest] ``` ### First run On the first commit, the script creates everything: ``` IMAGE_TAG=$(git rev-parse --short HEAD) ./deploy.sh ``` ``` --- build and push 127.0.0.1:5000/team/ingest:76ba6a6 Detected language: python Validating Python version 3.12.*... Python version 3.12.* resolved to 3.12.13 Building 127.0.0.1:5000/team/ingest:76ba6a6 App Path: ./fn Handlers File: main.py 2026/10/08 13:53:17 [Started] Python Builder: 127.0.0.1:5000/team/ingest:76ba6a6 2026/10/08 13:54:02 [Completed] Python Builder: 127.0.0.1:5000/team/ingest:76ba6a6 Build completed: 127.0.0.1:5000/team/ingest:76ba6a6 --- trigger Trigger created: ingest-uploads Name: ingest-uploads Description: N/A Tags: [] GUID: 514d61d8-e964-4530-bf05-9f0aac2a9630 Owner: {4294967295 vid 0xc0002f4ba0 map[]} CreatedAt: 2026-10-08T13:54:08Z UpdatedAt: 2026-10-08T13:54:08Z VRN: vast:dataengine:triggers:ingest-uploads Type: Element Events: -: ObjectCreated:* Topic: vast:dataengine:topics:my-broker/default SourceBucket: my-bucket --- function Function created: ingest Name: ingest Tags: [] GUID: cc4408e5-7092-43f7-83a0-e5318d27b7a9 Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T13:54:19Z Updated At: 2026-10-08T13:54:19Z VRN: vast:dataengine:functions:ingest Last Revision: 1 function ingest revision 1 --- pipeline Pipeline created: ingest Name: ingest Description: deployed by deploy.sh (CI-style) Tags: [] GUID: 00a4ccf0-4ef6-406f-83fd-1e25e0c28cae Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T13:54:30Z Updated At: 2026-10-08T13:54:30Z VRN: vast:dataengine:pipelines:ingest Pipeline deployed successfully --- wait for Ready 13:54:55 InProgress - 13:55:02 InProgress - 13:55:09 Ready Pipeline is ready and active. ``` The whole run took under two minutes, most of it the build. ### A new commit After a code change and a commit (`ca86591`), the same command builds a new image, registers it as revision 2 of the function, and deploys a pipeline pinned to it: ``` IMAGE_TAG=$(git rev-parse --short HEAD) ./deploy.sh ``` ``` --- build and push 127.0.0.1:5000/team/ingest:ca86591 Detected language: python Validating Python version 3.12.*... Python version 3.12.* resolved to 3.12.13 Building 127.0.0.1:5000/team/ingest:ca86591 App Path: ./fn Handlers File: main.py 2026/10/08 13:56:56 [Started] Python Builder: 127.0.0.1:5000/team/ingest:ca86591 2026/10/08 13:57:36 [Completed] Python Builder: 127.0.0.1:5000/team/ingest:ca86591 Build completed: 127.0.0.1:5000/team/ingest:ca86591 --- trigger trigger ingest-uploads exists --- function {"name":"ingest","last_revision_number":2} function ingest revision 2 --- pipeline Pipeline updated successfully Name: ingest Description: deployed by deploy.sh (CI-style) Tags: [] GUID: 00a4ccf0-4ef6-406f-83fd-1e25e0c28cae Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T13:54:30Z Updated At: 2026-10-08T13:58:12Z VRN: vast:dataengine:pipelines:ingest Status: Ready Pipeline deployed successfully --- wait for Ready 13:58:39 InProgress Pipeline is ready and active. 13:58:48 Ready Pipeline is ready and active. ``` Each function revision records the commit it was built from: ``` vastde functions get ingest -r -o json | jq -c ".revisions[] | {revision_number, image_tag, is_published}" ``` ``` {"revision_number":1,"image_tag":"76ba6a6","is_published":true} {"revision_number":2,"image_tag":"ca86591","is_published":true} ``` An upload to `incoming/` then ran the new code. The function's `init` logs its version, and the handler logs each object: ``` vastde logs get ingest --since 5m --scope user ``` ``` 2026-10-08 13:55:00.96 [ingest] [INFO] [user] ingest version 1 ready 2026-10-08 13:56:29.55 [ingest] [INFO] [user] ingest version 1 ready 2026-10-08 13:58:39.54 [ingest] [INFO] [user] ingest version 2 ready 2026-10-08 13:58:57.95 [ingest] [INFO] [user] ingest v2: incoming/ci-1.txt ``` The second `version 1 ready` line, at 13:56, is from a run with nothing to change; the next section explains it. ## What to watch for in a CI job ### Update published functions over REST In `vastde` v5.5.0-sp2, `vastde functions update` fails with `422 Unprocessable Entity` on any function that has a published revision, and changes nothing. The script therefore sends the update to the DataEngine API itself: `PUT /api/dataengine/functions/<guid>` with the function's full definition (name, artifact type and source, the new `image_tag`, the registry VRN, `revision_description` and `is_published: true`), authenticated with a token from `POST /api/token/<tenant>`. The PUT makes the next revision. The cause and a step-by-step version are in [publish functions](/develop/docs/dev-dataengine-guides-publish-functions#known-issue-functions-update-fails-on-published-functions). ### Poll on `status`, not `reason` `vastde pipelines deploy` returns as soon as the cluster accepts the deploy; the pipeline then goes from `InProgress` to `Ready` or `Error`. Wait on the `status` field only. Right after a deploy, `reason` can still hold the previous deploy's text, as in the run above (`InProgress Pipeline is ready and active.`), or a progress message: ``` vastde pipelines deploy ingest -o json; echo "exit $?"; vastde pipelines get ingest -o json | jq -c "{status, reason}" ``` ``` Pipeline deployed successfully exit 0 {"status":"InProgress","reason":"Waiting for pipeline to become ready..."} ``` Give the loop a timeout, and on `Error` print `reason` and the pipeline's logs before failing the job (see [observability](/develop/docs/dev-dataengine-guides-observability#logs)). ### An unchanged run still restarts the function Running the script again on the same commit skipped the function (its tag hadn't changed), but `pipelines update` with an identical file still made a new pipeline revision, and the deploy restarted the function: that's the second `version 1 ready` line in the logs above. After the unchanged run: ``` vastde pipelines get ingest -o json | jq -c "{status, last_revision_number}"; vastde functions get ingest -o json | jq -c "{last_revision_number}" ``` ``` {"status":"Ready","last_revision_number":2} {"last_revision_number":1} ``` So don't deploy on every push. Run the job only when the function code, the pipeline template or the environment's settings changed, using your CI system's path filters, or by comparing the generated `pipeline.yaml` with the copy from the last successful deploy. Re-running a job by hand is safe, but it rolls the pods. ### `-o json` output isn't pure JSON `pipelines update -o json` prints a plain-text line before the JSON, and `pipelines deploy -o json` prints only the plain-text line, so don't pipe them straight into `jq`: ``` vastde pipelines update ingest --config @pipeline.yaml -o json | head -5; echo "exit ${PIPESTATUS[0]}" ``` ``` Pipeline updated successfully { "created_at": "2026-10-08T13:54:30.274Z", "description": "deployed by deploy.sh (CI-style)", "guid": "00a4ccf0-4ef6-406f-83fd-1e25e0c28cae", exit 0 ``` Rely on the exit status, and read state afterwards with `get -o json`, which is plain JSON. `get` exits 1 for an object that doesn't exist, which is what the script's create-or-update branches test: ``` vastde pipelines get no-such-pipeline -o json; echo "exit $?" ``` ``` no pipeline found with name or guid: no-such-pipeline exit 1 ``` ### One deploy at a time DataEngine doesn't check versions when it saves a pipeline: the last save wins. Two jobs deploying the same pipeline at once can leave it on either commit. Serialize deploys per pipeline (`concurrency` in GitHub Actions, `resource_group` in GitLab), and don't change a pipeline that CI owns from the Web UI; see [one source of truth](/develop/docs/dev-dataengine-guides-pipelines-as-code#one-source-of-truth). ## Build images in CI `vastde functions build` needs a Docker daemon, and pushes with that machine's Docker credentials (`docker login`), not with the DataEngine registry settings. On a CI runner, that's the runner's own daemon or a Docker-in-Docker (DinD) service. With DinD: - **Point the CLI at the daemon with `DOCKER_HOST`.** The build uses `DOCKER_HOST` (and `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` for a TLS daemon) like the Docker CLI does. The lab ran a `docker:27-dind` daemon on a TCP port and built through it. - **`127.0.0.1` is the DinD daemon, not your host.** The build runs inside the DinD daemon's network. Pushing to a registry at `127.0.0.1:5000` that runs on the host failed: ``` export DOCKER_HOST=tcp://127.0.0.1:23750 vastde functions build 127.0.0.1:5000/team/ingest --target ./fn --image-tag dind-push --push; echo "exit $?" ``` ``` Detected language: python Validating Python version 3.12.*... Python version 3.12.* resolved to 3.12.13 Building 127.0.0.1:5000/team/ingest:dind-push App Path: ./fn Handlers File: main.py 2026/10/08 14:34:29 [Started] Python Builder: 127.0.0.1:5000/team/ingest:dind-push error building app: executing lifecycle: failed with status code: 1 exit 1 ``` Run it again with `--save-build-log`, and the log in `fn/` shows why: ``` ERROR: failed to initialize analyzer: validating registry read access: failed to ensure registry read access to 127.0.0.1:5000/team/ingest:dind-push: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused; Get "http://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused ``` Pushing the same build to the host's address on the DinD network, `<host address>:5000`, worked: ``` vastde functions build :5000/team/ingest --target ./fn --image-tag dind-push --push --save-build-log | tail -2 ``` ``` 2026/10/08 14:45:00 [Started] Python Builder: :5000/team/ingest:dind-push 2026/10/08 14:45:44 [Completed] Python Builder: :5000/team/ingest:dind-push Build log saved to: fn/build_ingest_dind-push_2026-10-08-14-44-59.log Build completed: :5000/team/ingest:dind-push ``` Use a registry address that the DinD service can reach, which is normally your registry's real host name. - **Plain HTTP worked for the push, but use HTTPS.** The lab registry speaks plain HTTP, and the push to it at a non-loopback address succeeded. DataEngine itself needs a registry served over HTTPS with a valid CA certificate to pull from; see [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites). - **A fresh daemon pulls the builder every time.** An empty DinD daemon pulled the builder image (2.14 GB) and the run image (740 MB) before its first build. Cache them, or use a runner with a persistent daemon, if build time matters. - **Use an amd64 runner.** Builds are `linux/amd64` only. - **The runner needs Docker Hub and PyPI**, or mirrors of them: the build pulls its run image and installs your `requirements.txt`. See [build and test locally](/develop/docs/dev-dataengine-guides-build-and-test-locally#the-builder-image). ## GitHub Actions This workflow runs `deploy.sh` on every push to `main`. It wasn't run on GitHub; every step it runs was run on the lab machine (above), except the CLI download and `docker login`. Set the repository variables and secrets named in the comments. ``` # Repository variables (Settings > Secrets and variables > Actions > Variables): # VAST_VMS_URL https:// # VAST_TENANT tenant name # VAST_USERNAME tenant admin or application user for that tenant # REGISTRY_PUSH registry host[:port] you push images to; the DataEngine container registry # REGISTRY_NAME must point at the same registry # REGISTRY_NAME DataEngine container registry name (vastde container-registries list) # CLUSTER DataEngine Kubernetes cluster name (vastde compute-clusters list) # NAMESPACE namespace in that cluster (vastde compute-clusters get ) # REGISTRY_USER user for docker login # Repository secrets: # VAST_PASSWORD, REGISTRY_PASSWORD # # The job needs network access to VMS, to the registry, to Docker Hub (the builder's run image) # and to PyPI (dependencies are installed during the build). For a VMS on a private network use a # self-hosted runner: runs-on: [self-hosted, linux, x64]. The build is linux/amd64 only. name: deploy-dataengine on: push: branches: [main] # One deploy at a time per pipeline: the API has no optimistic locking. concurrency: group: dataengine-ingest cancel-in-progress: false jobs: deploy: runs-on: ubuntu-24.04 env: VAST_VMS_URL: ${{ vars.VAST_VMS_URL }} VAST_TENANT: ${{ vars.VAST_TENANT }} VAST_USERNAME: ${{ vars.VAST_USERNAME }} VAST_PASSWORD: ${{ secrets.VAST_PASSWORD }} VAST_BUILDER_IMAGE_URL: vastdataorg/vast-builder:v5.5.0-sp2 REGISTRY_PUSH: ${{ vars.REGISTRY_PUSH }} REGISTRY_NAME: ${{ vars.REGISTRY_NAME }} CLUSTER: ${{ vars.CLUSTER }} NAMESPACE: ${{ vars.NAMESPACE }} steps: - uses: actions/checkout@v4 - name: Install the vastde CLI (pin the version that matches the cluster) run: | sudo curl -fsSL -o /usr/local/bin/vastde \ https://github.com/vast-data/dataengine-cli/releases/download/v5.5.0-sp2/vastde_linux_amd64 sudo chmod +x /usr/local/bin/vastde vastde version - name: Log in to the registry (vastde functions build --push uses ~/.docker/config.json) uses: docker/login-action@v3 with: registry: ${{ vars.REGISTRY_PUSH }} username: ${{ vars.REGISTRY_USER }} password: ${{ secrets.REGISTRY_PASSWORD }} - name: Build, push, create or update, deploy, wait for Ready env: IMAGE_TAG: ${{ github.sha }} run: ./deploy.sh ``` To run only when something deployable changed, add `paths:` under `push:` (for example `fn/**`, `pipeline.yaml.tmpl`, `deploy.sh`). ## GitLab CI The same job for GitLab, using a `docker:dind` service over TLS. It wasn't run on GitLab. On the lab machine, the build ran through a DinD daemon over plain TCP, not TLS. ``` # CI/CD variables (Settings > CI/CD > Variables; mask and protect the passwords): # VAST_VMS_URL, VAST_TENANT, VAST_USERNAME, VAST_PASSWORD (masked) # REGISTRY_PUSH (registry host[:port]; the DataEngine registry REGISTRY_NAME points at it), # REGISTRY_NAME, REGISTRY_USER, REGISTRY_PASSWORD (masked), CLUSTER, NAMESPACE # # vastde functions build needs a Docker daemon. This job uses the docker:dind service over TLS; # the CLI's builder picks up DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH from the # environment. The build runs with host networking inside the dind daemon, so the registry must be # reachable from the dind service. Use an amd64 runner: the build is linux/amd64 only. stages: [deploy] deploy-dataengine: stage: deploy image: docker:27 services: - name: docker:27-dind alias: docker variables: DOCKER_HOST: tcp://docker:2376 DOCKER_TLS_CERTDIR: /certs DOCKER_TLS_VERIFY: "1" DOCKER_CERT_PATH: /certs/client VAST_BUILDER_IMAGE_URL: vastdataorg/vast-builder:v5.5.0-sp2 IMAGE_TAG: $CI_COMMIT_SHORT_SHA # One deploy at a time per pipeline: the API has no optimistic locking. resource_group: dataengine-ingest rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH before_script: # deploy.sh needs bash, curl, jq and envsubst (gettext); the Python-version check needs unzip - apk add --no-cache bash curl jq gettext unzip - curl -fsSL -o /usr/local/bin/vastde https://github.com/vast-data/dataengine-cli/releases/download/v5.5.0-sp2/vastde_linux_amd64 - chmod +x /usr/local/bin/vastde - vastde version - echo "$REGISTRY_PASSWORD" | docker login "$REGISTRY_PUSH" -u "$REGISTRY_USER" --password-stdin script: - ./deploy.sh ``` To run only on deployable changes, add `changes:` to the rule (for example `fn/**/*`, `pipeline.yaml.tmpl`, `deploy.sh`). ## Image tags and promotion **Tag images with the commit, and never reuse a tag.** Each function revision records an image tag, so a tag that moves means a revision no longer says what it runs. A commit SHA, as in the script, ties the image, the function revision and the Git history together. Never deploy `latest`. See [name images and tags](/develop/docs/dev-dataengine-guides-publish-functions#name-images-and-tags). **Promote the image, not the source.** Build once, in the first environment, and deploy the same tag to the next ones. Rebuilding for production installs your dependencies again, and can produce a different image from the one you tested. **Give each environment its own namespace or cluster.** A pipeline's compute cluster and namespace are fixed when you create it, and a namespace belongs to one tenant. Ask your platform team for a separate tenant, with its own namespace, for each environment, or a separate cluster; see [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites#decide-before-you-start). Then: - Keep one CI variable set per environment (`VAST_VMS_URL`, `VAST_TENANT`, the credentials, `REGISTRY_NAME`, `CLUSTER`, `NAMESPACE`), for example as GitHub environments or GitLab environment-scoped variables. The template and script stay the same. - Run the script against development on every merge, and against test and production from a manual approval or a release tag, with `IMAGE_TAG` set to the tag that passed the previous stage. Skip the build step for those stages, or make the build a separate job, so that only the first stage builds. - If production pulls from a different registry, copy the image there with its tag unchanged before you run the script. - Pin function revisions in the pipeline file, as the template does, so that a deploy runs exactly the revision you registered from that tag. ## The DataEngine API DataEngine has its own API, served by VMS under `https://<VMS address>/api/dataengine/` (also under `/api/latest/dataengine/`). Use it from tools that aren't shell scripts, and for anything the CLI can't do. Its references: - The KB's [VAST API reference](https://kb.vastdata.com/api-5-5/apidocs), with the DataEngine API under [VAST DataEngine API](https://kb.vastdata.com/api-5-5/apidocs/engine-api-introduction). - Your cluster's own OpenAPI specs: `https://<VMS address>/api/dataengine/openapi.json` (functions, triggers, pipelines, deploy, registries) and `https://<VMS address>/api/dataengine/telemetries/openapi.json` (logs, traces, metrics). They need the same authentication as the API. - Your cluster's VMS REST API documentation at `https://<VMS address>/docs/` and its Swagger UI at `https://<VMS address>/api/`, for the VMS objects around DataEngine. The two specs, from the lab: ``` curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/openapi.json" | jq "{openapi, info, paths: (.paths | length)}" ``` ``` { "openapi": "3.1.0", "info": { "title": "Provisioning Service", "description": "Provisioning Service", "version": "1.4.0" }, "paths": 52 } ``` ``` curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/telemetries/openapi.json" | jq "{openapi, info, paths: (.paths | keys)}" ``` ``` { "openapi": "3.1.0", "info": { "title": "Telemetries Query Service", "description": "API for querying traces, logs and metrics from DB", "version": "1.4.0" }, "paths": [ "/", "/count-functions", "/logs", "/metrics", "/metrics/names", "/reports/{report_id}/execute", "/span-logs", "/trace-tree", "/traces" ] } ``` The examples use `curl -k`, because the lab's VMS has a self-signed certificate. Leave `-k` out when your machine trusts the VMS certificate. ### Authenticate Use a tenant admin or an application user on your tenant, in one of two ways: - **Basic authentication with `X-Tenant-Name`.** Send the user and password, and name the tenant in an `X-Tenant-Name` header: ``` curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" | jq ".data[] | {name, guid, status, reason, last_revision_number}" ``` ``` { "name": "thumbnails", "guid": "c7f63133-3438-4cae-af6a-91291041f1b6", "status": "Ready", "reason": "Pipeline is ready and active.", "last_revision_number": 5 } ``` Without the header, the right password is refused with the same message as a wrong one: ``` curl -sk -w "\nHTTP %{http_code}\n" -u "$VAST_USERNAME:$VAST_PASSWORD" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" ``` ``` {"detail":"Invalid username/password."} HTTP 403 ``` - **A bearer token from `POST /api/token/<tenant>`.** Exchange the user and password for an access token once, then send it as `Authorization: Bearer`. The tenant is in the path, so no `X-Tenant-Name` is needed: ``` export TOKEN=$(curl -sk -X POST "$VAST_VMS_URL/api/token/$VAST_TENANT" -H "Content-Type: application/json" -d "{\"username\": \"$VAST_USERNAME\", \"password\": \"$VAST_PASSWORD\"}" | jq -r .access) curl -sk -H "Authorization: Bearer $TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" | jq -c ".data[] | {name, status}" ``` ``` {"name":"thumbnails","status":"Ready"} ``` The response also has a `refresh` token. Use `/api/token/<tenant>`, not `/api/token/`: for a tenant user, `/api/token/` fails: ``` curl -sk -X POST "$VAST_VMS_URL/api/token/" -H "Content-Type: application/json" -d "{\"username\": \"$VAST_USERNAME\", \"password\": \"$VAST_PASSWORD\"}" ``` ``` {"detail":"Authentication failure, user: "} ``` **VMS API tokens don't work for the provisioning API** (functions, triggers, pipelines, deploy) in 5.5.0-SP6. A token that works for the VMS API is refused: ``` curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Api-Token $VMS_API_TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=thumbnails" ``` ``` {"detail":"{\"authorization\": \"Value error, Invalid Api-Token\"}"} HTTP 422 ``` The telemetries API (logs, traces, metrics) does accept one, which suits a read-only monitoring job: ``` curl -sk -o /dev/null -w "%{http_code} %{content_type}\n" -H "Authorization: Api-Token $VMS_API_TOKEN" "$VAST_VMS_URL/api/dataengine/telemetries/logs?pipeline_name=thumbnails&limit=2" ``` ``` 200 application/json ``` So a CI job that deploys needs a user and password, held as a masked secret. Use an application user with only the DataEngine permissions the job needs; see [security](/develop/docs/dev-dataengine-operations-security#identity-policy-permissions). ### Deploy and wait The deploy endpoint is `POST /api/dataengine/pipeline/<guid>/deploy`, with `pipeline` in the singular. The plural path, which every other pipeline call uses, returns 404: ``` curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Bearer $TOKEN" -X POST "$VAST_VMS_URL/api/dataengine/pipelines/bc7b6df0-4dbc-4c69-94b7-61153607b1c3/deploy" ``` ``` {"detail":"Not Found"} HTTP 404 ``` ``` curl -sk -w "\nHTTP %{http_code}\n" -H "Authorization: Bearer $TOKEN" -X POST "$VAST_VMS_URL/api/dataengine/pipeline/bc7b6df0-4dbc-4c69-94b7-61153607b1c3/deploy" ``` ``` HTTP 200 ``` A successful deploy returns an empty body. Then poll the pipeline's `status`, as with the CLI: ``` for i in $(seq 1 30); do s=$(curl -sk -H "Authorization: Bearer $TOKEN" "$VAST_VMS_URL/api/dataengine/pipelines?name=orders-rest" | jq -c ".data[0] | {status, reason}"); echo "$(date -u +%T) $s"; case "$s" in *\"Ready\"*|*\"Error\"*) break;; esac; sleep 5; done ``` ``` 13:45:56 {"status":"InProgress","reason":null} 13:46:01 {"status":"InProgress","reason":null} 13:46:07 {"status":"InProgress","reason":null} 13:46:13 {"status":"Ready","reason":"Pipeline is ready and active."} ``` Creating triggers (`POST /triggers/element`, `POST /triggers/schedule`), functions (`POST /functions`) and pipelines (`POST /pipelines`, with the pipeline file as JSON) works the same way; for the request bodies, see [create a new pipeline](https://kb.vastdata.com/api-5-5/apidocs/create-a-new-pipeline), [create a new function](https://kb.vastdata.com/api-5-5/apidocs/create-a-new-function) and [create a new element trigger](https://kb.vastdata.com/api-5-5/apidocs/create-a-new-element-trigger). To look up a container registry by name, list them and filter on the client: `GET /container-registries` ignores a `name` parameter and returns them all. For driving pipelines from an orchestrator, see [orchestrators](/develop/docs/dev-dataengine-integrations-orchestrators). ### Cluster admins A cluster admin isn't tied to a tenant, so names it on every call with a `tenant_name` **query parameter**, on reads and writes alike, for example `POST /api/dataengine/triggers/schedule?tenant_name=<tenant>`. The tenant isn't accepted in the request body (`422`, `"tenant_name": "Extra inputs are not permitted"`). With `vastpy-cli` (see below), logged in as a cluster admin: ``` vastpy-cli get dataengine/pipelines name=orders ``` ``` Error: response for GET api/latest/dataengine/pipelines with params=[('name', 'orders')] failed with error 400 and message {"detail":"No tenant was provided in request"} ``` ``` vastpy-cli get dataengine/pipelines name=orders tenant_name= --json | jq -c ".data[]? | {name, status, tenant_guid} // ." ``` ``` {"name":"orders","status":"Ready","tenant_guid":""} ``` Most CI jobs should use a tenant user instead. ## vastpy `vastpy` (from PyPI) reaches the DataEngine API by path: `client.dataengine.pipelines` is `/api/latest/dataengine/pipelines`. Log in with a user, password and tenant; as with REST, a VMS API token is refused for anything but telemetries. This script creates or updates a pipeline from its file, deploys it and waits: ``` """Create or update a DataEngine pipeline from pipeline.yaml, deploy it, and wait for Ready. Env: VMS_ADDRESS, VMS_USER, VMS_PASSWORD, VMS_TENANT_NAME (tenant admin). The DataEngine API doesn't accept VMS API tokens, so this uses username/password.""" import os, sys, time import yaml from vastpy import VASTClient client = VASTClient(address=os.environ["VMS_ADDRESS"], user=os.environ["VMS_USER"], password=os.environ["VMS_PASSWORD"], tenant=os.environ["VMS_TENANT_NAME"]) de = client.dataengine spec = yaml.safe_load(open(sys.argv[1] if len(sys.argv) > 1 else "pipeline.yaml")) found = de.pipelines.get(name=spec["name"])["data"] if found: guid = found[0]["guid"] de.pipelines[guid].put(**spec) # full update: new pipeline revision print(f"updated {spec['name']} ({guid})") else: guid = de.pipelines.post(**spec)["guid"] print(f"created {spec['name']} ({guid})") # deploy.post() with no arguments sends an empty chunked body that the server rejects # (400 "Invalid HTTP request received."); request("POST") sends no body. de.pipeline[guid].deploy.request("POST") # POST .../pipeline//deploy/ print("deploy requested") deadline = time.time() + 300 while True: p = de.pipelines[guid].get() print(f"status={p['status']} reason={p['reason']} revision={p['last_revision_number']}") if p["status"] in ("Ready", "Error"): break if time.time() > deadline: sys.exit("timed out waiting for the pipeline") time.sleep(10) sys.exit(0 if p["status"] == "Ready" else 1) ``` Run against an existing pipeline, `orders` (the script needs `pip install vastpy pyyaml`): ``` VMS_ADDRESS="" VMS_USER="$VAST_USERNAME" VMS_PASSWORD="$VAST_PASSWORD" VMS_TENANT_NAME="$VAST_TENANT" python deploy_pipeline.py pipeline.yaml ``` ``` updated orders (a763a602-0c97-45ea-adfa-a84f338fef50) deploy requested status=InProgress reason=Pipeline is ready and active. revision=2 status=Ready reason=Pipeline is ready and active. revision=2 ``` Again, `status` changed while `reason` still showed the last deploy's text. **Calls with no body need `.request()`.** `vastpy`'s `.post()` and `.delete()` with no arguments send an empty body that the server rejects. With `de.pipeline[guid].deploy.post()` the deploy fails: ``` vastpy.RESTFailure: response for POST api/latest/dataengine/pipeline/a763a602-0c97-45ea-adfa-a84f338fef50/deploy with params=None failed with error 400 and message "Invalid HTTP request received." ``` Use `.request("POST")` for a deploy and `.request("DELETE")` for a delete, for example `de.pipelines[guid].request("DELETE")`. `vastpy-cli delete` fails the same way, so delete from Python. `vastpy-cli` reads `VMS_USER`, `VMS_PASSWORD` and `VMS_TENANT_NAME`, which is enough for quick reads in a job: ``` VMS_ADDRESS="" VMS_USER="$VAST_USERNAME" VMS_PASSWORD="$VAST_PASSWORD" VMS_TENANT_NAME="$VAST_TENANT" vastpy-cli get dataengine/pipelines name=orders --json | jq -c ".data[] | {name, status, reason}" ``` ``` {"name":"orders","status":"Ready","reason":"Pipeline is ready and active."} ``` A cluster admin passes `tenant_name=<tenant>` as a parameter (above) instead of setting `VMS_TENANT_NAME`, which with a cluster-admin login is refused with `403 Invalid username/password.` ## Terraform and MCP - **Terraform.** The VAST Terraform provider (4.0.2) has no resources for DataEngine functions, triggers or pipelines. What it has around DataEngine is for your platform team: compute clusters, and a tenant's `data_engine_enabled` setting. Use the CLI or the API for everything on this page. The views and topics a pipeline reads from can be Terraform-managed; see the [DataBase](/develop/docs/dev-database-operations-ci-cd) and [VAST Event Broker](/develop/docs/dev-event-broker-operations-ci-cd) CI/CD pages. - **MCP.** The VAST admin MCP server has no DataEngine tools: none of its tools lists or changes functions, triggers or pipelines. ## See also - [Pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code): the pipeline file, update and deploy, revisions, exporting a pipeline into Git. - [Publish functions](/develop/docs/dev-dataengine-guides-publish-functions): image tags, revisions, pinning, and the REST update in detail. - [Build and test locally](/develop/docs/dev-dataengine-guides-build-and-test-locally): the build and the builder image. - [Security](/develop/docs/dev-dataengine-operations-security): user types, permissions, secrets, CLI credentials. - [Observability](/develop/docs/dev-dataengine-guides-observability): logs and traces to print when a deploy fails. - [Automation tools](/develop/docs/dev-database-reference-automation-tools): the CLI, REST API, `vastpy`, Terraform and MCP compared across sections. - CI/CD pages for [VAST DataBase](/develop/docs/dev-database-operations-ci-cd), [VAST Vector Search](/develop/docs/dev-vector-search-operations-ci-cd) and [VAST Event Broker](/develop/docs/dev-event-broker-operations-ci-cd). - KB: [VAST DataEngine API](https://kb.vastdata.com/api-5-5/apidocs/engine-api-introduction), [Getting started with DataEngine](/docs-5-5/docs/getting-started-dataengine).