Setup prerequisites

Applies to VAST Cluster 5.5.

Before you can deploy a DataEngine pipeline, your platform team has to enable DataEngine on your tenant and connect it to an event broker, a compute cluster and a container registry. This page lists what to ask them for, gives you a request you can copy, and covers the tools you install on your own machine.

What you need

You need	What it is	Who provides it
DataEngine enabled on your tenant	DataEngine is enabled per tenant, and you use it on exactly one tenant. Enabling it sets the tenant's default topic (where triggers send events unless you choose another topic) and its dead-letter topic (where events go after their retries run out).	Platform team
An event broker	A Kafka-compatible broker for the topics: a VAST Event Broker view, or a third-party Kafka broker.	Platform team
A compute cluster and a namespace	Where your pipelines run: a VAST compute cluster (Kubernetes on some of the cluster's CNodes) or an external Kubernetes cluster, linked to DataEngine with at least one namespace for your tenant. When the platform team attaches your tenant to a VAST compute cluster, it gets your tenant's DataEngine namespace, `dataengine-ns-<tenant GUID>`, and is linked to your tenant. A namespace can't be shared between tenants.	Platform team
A container registry	Where your function images live. It must be served over HTTPS with a certificate from a valid CA, and linked to the compute cluster and the namespace your pipelines use.	Platform team
A DataEngine user	Your tenant's URL, and a user name and password: a tenant admin, or an application user with DataEngine permissions (the `data-engine-<tenant>` identity policy or your own). See security.	Platform team
An S3 access key pair	For your user, on the tenant. You can create your own after you log in.	You, or the platform team
Source views	For element triggers: the views (buckets) whose objects or files you want to react to, configured for triggers. Application users can't create them.	Platform team
S3 keys for your functions	If your functions read or write objects or tables: a key pair, ideally for a separate user, with access to those buckets. See security.	Platform team
Registry push access	Credentials that let you push images to the registry from your machine or CI.	Registry owner
Tools	Docker, the `vastde` CLI and a configured builder image. See your workstation.	You

Admin task: send to your platform team. The setup steps are in the KB's VAST DataEngine admin guide:

Enabling DataEngine on a VAST Cluster tenant: the prerequisites, the event broker with its default and dead-letter topics, the compute cluster and the container registry. Tenant admins use the tenant guide version.
Managing VAST compute clusters (cluster admins only) or managing external Kubernetes clusters.
Managing container registries.
Provisioning user access and permissions for DataEngine: application users, the DataEngine role and the identity policy.
Provisioning source views for element type triggers.

For a VAST Event Broker view and its topics, see also the Event Broker setup page.

What your platform team sets up for DataEngine, in order, and what you install, before vastde connects.

Decide before you start

Some choices are hard to change later, so agree on them with your platform team first:

Which cluster and namespace your pipelines use. A pipeline's compute cluster and namespace are set when you create it and can't be changed afterwards. Moving to another namespace means creating the pipeline again. If you want separate development and production environments, ask for a namespace (or a cluster) for each.
That the registry is linked to that cluster and namespace. A pipeline deploys only if every function's registry is linked to the pipeline's cluster and namespace (or to all its namespaces). Otherwise the deploy fails with an "Access to Container registry ... is denied" error.
The default topic's partitions. A topic's partition count can't be changed after it's created. With ordered delivery, a function processes at most one event per partition at a time, so the partition count caps its concurrency. 50 to 100 partitions are suggested for the default topic.
How your images are named. The registry's base URL in DataEngine, the image name you push, and the artifact source you give vastde functions create must line up. See publish functions.

What to send your platform team

Copy this and fill it in:

I'd like to build and deploy VAST DataEngine pipelines on tenant <tenant>.

- DataEngine: enabled on the tenant. Please tell me the event broker, the
  default topic and the dead-letter topic.
- Compute: a compute cluster linked to DataEngine, with a namespace for
  <project/environment> (on a VAST compute cluster, the tenant's
  dataengine-ns-<tenant GUID> namespace). Please tell me the cluster name
  and the namespace.
- Registry: a container registry, served over HTTPS with a valid CA
  certificate, linked to that cluster and namespace. Please tell me its
  name in DataEngine, its base URL, and how I get push credentials.
- User: an application user <user name> in the DataEngine group, with the
  data-engine-<tenant> identity policy (or: these actions <actions>).
- S3 keys: an access key pair for my user on the tenant.
- Source views: <view/bucket names>, configured for <S3 | NFS> element
  triggers, with my group allowed to list the bucket.
- Function access: a key pair for a separate user that my functions use,
  with <read | read and write> access to <buckets>.
- Versions: which vastde CLI release and builder image tag match the cluster.

Your workstation

You need:

Docker, running. vastde functions build builds images with Docker and the builder image.
The vastde CLI. Download the binary for your operating system and architecture from the VAST DataEngine CLI releases page and put it on your PATH. Use the release your platform team names for your cluster version; see versions.
A CLI configuration. Point the CLI at your tenant:
```
vastde config init --vms-url https://<vms-address> --tenant <tenant> \
                   --username <user> --password <password>
vastde config view
```
If you have no configuration file yet, the 5.5.0-sp2 CLI exits with an error and writes nothing when you pass these flags to config init. Run vastde config init once with no flags to create the file, then run the command above again with --force.

The configuration is saved in ~/.vast/config.toml. To keep the password out of your shell history and out of files, see security.
A builder image. By default the CLI uses the builder image whose tag matches the CLI's version. To use another one, for example a newer service pack:
```
vastde builders list
vastde builders set <tag>
```
builders list and builders inspect look the tags up on Docker Hub, so they need internet access.
Internet access for builds, or mirrors. vastde functions build pulls the base image for your function image from Docker Hub, and installs Python packages from PyPI, so the machine that builds needs access to both, or to mirrors of them.
Push access to the registry. vastde functions build --push pushes with your local Docker credentials, so run docker login <registry host> first. The DataEngine registry settings aren't used for the push.

Check your setup

Each command checks one item on the list. They only read, so an application user can run them.

vastde config view                  # the VMS URL, tenant and user you configured
vastde compute-clusters list        # the compute cluster linked to DataEngine
vastde compute-clusters get <cluster> -o yaml   # its VRN, and your namespace under namespaces
vastde container-registries list    # your registry, and the cluster it's linked to
vastde buckets list                 # the S3-enabled views (buckets) you can see

If a command fails to log in, check the VMS URL, tenant, user name and password. If vastde compute-clusters list fails with 403 Forbidden: Tenant is not provisioned for Data Engine. Complete data engine setup first., DataEngine isn't enabled on your tenant yet. Enabling it finishes in the background, so if your platform team has only just done it, try again shortly. If a list is empty, the item hasn't been set up for your tenant, or your user can't see it: send the request above. Then try the quickstart.

Next steps

Quickstart: deploy a pipeline that runs on an S3 upload.
Security: permissions, secrets and credentials.
Write a function.