Building a VAST DataEngine Lab/Demo Environment from scratch - Omnibus Guide

Deploy a non-production VAST DataEngine lab, from VAST tenant setup through a running scheduled function.

⚠️ Warning

Lab use only. This path omits production security controls and is not supported as a production architecture.

Supported environment

• VAST AI OS 5.4.3-SP5 or 5.4.4-SP6, or a later supported 5.4.x release

• Ubuntu 24.04 virtual machines, four to seven in total, each with access to both the VAST management (VMS) network and the VAST storage network

Size the virtual machines as follows.

📝 Note

VAST AI OS 5.5.x and the single-node microk8s/k3s path are not covered by this guide.

Procedure

1. Collect VAST DataEngine Lab Values

   Complete the worksheet of names, VIPs, credentials, and generated values.

2. Set Up VAST Tenants and VIP Pools

   Create the DataEngine tenant, optional CSI tenant, VIP pools, policies, and users.

   Alternate path: Set Up VAST Tenants and VIP Pools for the CSI Driver Only — for environments that need only the VAST CSI driver.

   Alternate path: Enable DataEngine with vastpy-cli — command-line setup of the tenant, event broker, and DataEngine registration with vastpy-cli.

3. Install an RKE2 Cluster

   Build an RKE2 cluster and verify that every node is Ready.

4. Create a Podman Registry for the RKE2 Cluster

   Create a TLS registry that every RKE2 node can use.

5. Install the VAST CSI Driver

   Optional: install CSI and verify dynamic PVC provisioning.

   Alternate path: Install the VAST CSI Driver on OpenShift from the Software Catalog — for Red Hat OpenShift clusters.

6. Install DataEngine Prerequisites on the Kubernetes Cluster

   Install namespaces, Zarf prerequisites, and the DataEngine operator.

   Alternate path: Install DataEngine Prerequisites on an OpenShift Cluster — for Red Hat OpenShift clusters running VAST AI OS 5.4.3 or 5.4.4.

7. Connect DataEngine to the Kubernetes Cluster and Registry

   Link DataEngine to the broker, cluster, namespaces, and registry.

   Alternate path: Enable DataEngine with vastpy-cli — command-line setup of the tenant, event broker, and DataEngine registration with vastpy-cli.

8. Install the VAST DataEngine CLI Tools

   Install and configure the vastde CLI.

9. Run a “Hello World” Pipeline

   Build, deploy, and run a scheduled “Hello World” pipeline.

Alternate paths

These procedures are also available for environments that differ from the RKE2 lab above. Each one lists the outcome it produces, when it applies, and where it rejoins the main series.

1. Set Up VAST Tenants and VIP Pools for the CSI Driver Only

   Creates a CSI tenant, VIP pool, view policy, VMS roles, and manager accounts for environments that need only the VAST CSI driver.

2. Install the VAST CSI Driver on OpenShift from the Software Catalog

   Installs the certified VAST CSI Operator from the OpenShift Software Catalog and creates a VAST-backed StorageClass.

3. Install DataEngine Prerequisites on an OpenShift Cluster

   Installs the namespaces, Zot registry, and VAST DataEngine operator on a Red Hat OpenShift cluster running VAST AI OS 5.4.3 or 5.4.4.

4. Enable DataEngine with vastpy-cli

   Creates the tenant, group, user, VIP pool, view policy, broker view, and topics with vastpy-cli, then enables DataEngine and registers the cluster and registry through the API.

Completion criteria

The lab is complete when the RKE2 nodes are Ready, the registry accepts images, DataEngine appears under VAST Admin, and the scheduled Hello World pipeline is Running.