Overview

Applies to VAST Cluster 5.5.

VAST DataEngine runs your Python functions on Kubernetes when something happens to your data on the cluster: an S3 object or an NFSv4 file is created or deleted in a view, an S3 object's tags change, or a schedule fires. You connect triggers and functions into pipelines. Events travel between them as CloudEvents on Kafka topics, usually topics on VAST Event Broker, and the compute cluster scales each function up and down with the load.

What it is

Functions are container images. You write a Python handler with the DataEngine Runtime SDK, and vastde builds it into an image that you push to a container registry. A DataEngine function points at that image, and every change to it is a numbered revision.
Triggers turn data events into CloudEvents. An element trigger watches a view for objects or files being created or deleted (S3 and NFSv4), or for S3 tags being added or removed. A schedule trigger fires on a cron schedule. Each trigger publishes CloudEvents to a Kafka topic.
Pipelines wire triggers to functions. A pipeline deploys functions and links them: trigger to function, and function to function. A function's return value becomes the next function's event, and labels set in code can route an event down one link and not another.
It runs on Kubernetes, next to your data. A pipeline deploys into a namespace on a compute cluster: an external Kubernetes cluster, or a VAST compute cluster that runs Kubernetes on some of the cluster's CNodes (new in 5.5). Each function runs as an autoscaling Knative service.
Telemetry is built in. Logs, traces and metrics from every function go to DataEngine's telemetry service, which stores them on the cluster. You query them with vastde or the Web UI.

Triggers turn data changes into events, topics carry them, and your functions process them on a compute cluster; you drive it all with vastde and a YAML file in Git.

See concepts for the objects, and architecture for what runs where.

What you get

You'd otherwise build	DataEngine gives you
A notification consumer that polls a bucket or listens to S3 events and calls your code	Element triggers on S3 buckets and NFSv4 exports, with name-prefix and name-suffix filters, that deliver CloudEvents straight to your function
A cron host for scheduled jobs	Schedule triggers
A queue between each pair of processing steps	Links between functions, carried on Kafka topics, with per-link retries and a dead-letter topic for events that still fail
Container build scripts and a base image with an HTTP server	`vastde functions build`, which packages your handler with the DataEngine runtime
Deployments, autoscalers and their YAML	One function deployment per step, with minimum and maximum instances, CPU and memory, and an autoscaling metric, set in the pipeline file
Glue code to batch events for bulk writes or batch inference	Event batching per function deployment, by count or time window (new in 5.5)
A logging, tracing and metrics stack	Logs, traces that follow each event across functions, and custom metrics, all captured by the runtime

How you work with it

These docs teach one path: the vastde CLI, with your function code and pipeline files kept in Git.

Interface	Use it to	In these docs
`vastde` CLI	Scaffold, build and test functions locally; register functions and triggers; create, update and deploy pipelines from YAML; read logs, traces and metrics	Every quickstart and guide
Pipeline YAML, in Git	Describe a pipeline's triggers, function deployments, links and settings. `vastde pipelines create --config` and `vastde pipelines update --config` send it to the cluster.	Pipelines as code
VAST Web UI	Watch runs: the dashboard, the trace tree and span logs. It can also build pipelines in a visual builder, but these docs don't teach that.	Coming from the UI maps every UI field to its CLI flag or YAML key
REST API	Automate from your own tools. The base URL is `https://<VMS address>/api/dataengine/`, with an API token, Basic authentication or a JWT.	VAST DataEngine API. Your cluster also serves the VMS REST API documentation at `https://<VMS address>/docs/`.

The UI and the CLI change the same objects, and the cluster doesn't check versions when either one saves. If one person edits a pipeline in the UI and another updates it from a file, the last save wins. So pick Git as the source of truth, and treat the UI as read-only for pipelines you manage from YAML. See concepts.

Use cases and considerations

Example use cases:

Processing data as it lands on the cluster: thumbnails, format conversion, metadata extraction, embedding, indexing, validation, one object or file at a time.
Multi-step processing in Python, where each step scales on its own and failed events are kept for inspection.
Scheduled jobs that run next to your data, with the same logging and tracing as your event-driven ones.
Writing results to VAST DataBase tables or S3, with the processing on the same platform.

Complete examples (more in reference applications):

VSS blueprint: video search and analysis. A DataEngine pipeline segments, analyses and embeds video with NVIDIA NIM models, and writes the results to VAST DataBase (reference application on GitHub, not verified against these docs).
Research assistant agent: a RAG chat assistant. A DataEngine pipeline ingests uploaded documents, and VAST DataBase holds the vectors and the conversation history (reference application on GitHub, not verified against these docs).

Important to note:

Python only. Functions are Python 3 only in 5.5.
Not for SQL over large tables. Use the VAST Query Engine, Trino or Spark; see querying. A function is for per-event or per-batch work, not for big scans.
No exactly-once guarantee. Failed calls are retried, so the same event can arrive more than once. Write handlers that are safe to run again on the same event.
S3 and NFSv4 only for element triggers. Element triggers fire for S3 objects and for regular files over NFSv4 only. NFSv3 and SMB changes, and changes in nested views, don't fire.
A compute cluster is required. Your platform team must provide an external Kubernetes cluster or a VAST compute cluster. VAST compute clusters aren't available in VAST on Cloud, and don't support L3 networking or IPv6 in 5.5.
Schedules can be missed. In 5.5, schedule triggers don't fire while the VMS is down (for example during an upgrade or an HA failover). Element triggers are unaffected.

How it fits with VAST DataBase, Event Broker and Vector Search

Product	How DataEngine uses it
VAST DataBase	Your functions read and write tables with the VAST DataBase SDK, with the endpoint and keys passed as secrets and environment variables. DataEngine also stores its telemetry in tables on the cluster.
VAST Event Broker	Triggers publish events to topics, and links between functions carry events on topics. The broker is usually a VAST Event Broker view (it can also be an external Kafka broker). The tenant's default topic and dead-letter topic live there too. Topics are explained in the Event Broker section; DataEngine adds only the topic roles described in concepts.
VAST Vector Search	A common pipeline step computes embeddings for new objects and writes them to a table with a vector column. Vector search itself is covered there.

If you only want S3 bucket notifications in a Kafka topic, for your own consumers, you don't need DataEngine: see S3 bucket events.

What's new in 5.5

NFS-based triggers. Element triggers on NFSv4 files, as well as S3 objects.
Conditional triggers. A function sets labels on the events it returns, and a link fires only for events with the labels it names.
Event batching. A function deployment can receive events in batches, by count or time window.
User-defined metrics. Counters, histograms, gauges and up-down counters from your code, read with vastde metrics get or on the dashboard.
VAST compute clusters. Kubernetes on a subset of the cluster's CNodes, as an alternative to an external Kubernetes cluster.

What's in this section

Start: this overview and the quickstart: from an empty directory to a deployed pipeline that runs on an S3 upload, CLI only.
Concepts: objects and revisions and architecture.
Guides: write a function, build and test locally, publish functions, triggers, pipelines as code, observability and working with data.
Reference: CLI, pipeline YAML, events, Runtime SDK, limits and coming from the UI.
Integrations: patterns for VAST DataBase and Vector Search, the dead-letter topic, OpenTelemetry, orchestrators and OpenLineage and OpenMetadata.
Examples: reference applications: complete applications built on DataEngine.
Operations: setup prerequisites, security and CI/CD and automation.
Troubleshooting: symptom, cause and fix.

Where to go next

Get DataEngine from your platform team: setup prerequisites lists what to ask for (DataEngine enabled on your tenant, a compute cluster and namespace, a linked container registry, permissions and S3 keys).
Run the quickstart.
Read concepts and architecture before you design a pipeline.
Put your pipeline under Git with pipelines as code. If you've built pipelines in the UI, start with coming from the UI.