Concepts

Applies to VAST Cluster 5.5.

This page names the objects you work with in VAST DataEngine, says how they relate, and explains the two things that most often surprise people: which function revision a pipeline runs, and what happens when the UI and the CLI edit the same pipeline. Read it before you design a pipeline.

Objects

Object	What it is	Defined by
Function	A named pointer to a container image in a registry: the registry, the image's repository path (the artifact source) and its tag. Every change that affects the image creates a new revision.	`vastde functions create` / `update`
Function revision	One numbered version of a function. Numbers start at 1 and go up by one. A revision can have an alias and a description.	Created by `functions create` and `functions update`
Function deployment	One use of a function inside a pipeline, with its own revision, resources, scaling, batching, log level, environment variables and secrets. The same function can be deployed several times, in one pipeline or many.	`function_deployments` in the pipeline YAML
Trigger	A source of events: an Element trigger watches a view, a Schedule trigger fires on a cron expression. Each publishes CloudEvents to one topic. Triggers are reusable across pipelines.	`vastde triggers create element` / `schedule`
Pipeline	A named set of triggers, function deployments and links, deployed to one namespace of one compute cluster. Its name, cluster and namespace can't change after you create it.	`vastde pipelines create --config`
Pipeline revision	One saved version of the pipeline's contents (its manifest). At most one revision is deployed at a time.	Created by `pipelines create` and `pipelines update`
Link	A connection from one or more sources (triggers or function deployments) to one or more destination function deployments, with optional retries, ordering and labels. The UI calls it a connection.	`links` in the pipeline YAML
Event	A CloudEvent (v1.0) delivered to a function. The runtime wraps it in a `VastEvent`.	Triggers and functions
Topic	A Kafka topic that carries events. See tenants and topics.	Your platform team, or created for a trigger
Compute cluster	The Kubernetes cluster that runs pipelines: an external Kubernetes cluster or a VAST compute cluster. Pipelines deploy into one of its namespaces.	Your platform team
Container registry	An external registry that holds function images, linked to DataEngine by your platform team.	Your platform team

Triggers and functions are reusable; a pipeline revision wires them together.

Functions and revisions

A function doesn't contain your code; it points at an image you've already pushed. When you rebuild and push a new image tag, you run vastde functions update to point the function at it, and that creates a new revision. Changing only the description doesn't.

Two settings decide which revisions are used:

Default revision. A function can name a default revision (default_revision_number). If it doesn't, the latest revision is the default.
Published. A revision can be marked published (--publish on functions create or update). The API describes an unpublished revision as one other users can't use.

The image must already be in the registry when you create the revision: the cluster checks that it can reach it.

Which revision runs

Each function deployment in a pipeline either pins a revision or leaves it unpinned:

In the pipeline YAML	Revision that runs
`revision: 3` (pinned)	Revision 3, until you change the YAML and deploy again
`revision` left out or null (unpinned)	The function's default revision, which is the latest revision if no default is set

The cluster resolves the revision when you deploy the pipeline, not when the function changes. So a new function revision never changes a running pipeline by itself. A pinned deployment keeps its revision until you edit the YAML. An unpinned one picks up the new default the next time anyone deploys the pipeline, for any reason.

Pin revisions in pipelines you promote between environments, so that a deploy runs exactly what you tested. See publish functions.

Triggers

An element trigger watches elements: the files and S3 objects under a view's path in the cluster's Element Store (background: the cluster underneath). Its source types (below) decide which protocol's changes it reports.

	Element trigger	Schedule trigger
Fires when	An element is created or deleted in a view, or an S3 object's tags are added or removed	A cron expression comes due
Event types (CLI names)	`ObjectCreated:`, `ObjectRemoved:`, `ObjectTagging:Put`, `ObjectTagging:Delete`	—
Source	A view's bucket. Elements are S3 objects or NFSv4 files (`source_types` `S3`, `NFS4`). Tag events are S3 only.	—
Filters	Name prefixes and suffixes: the object key for S3, the file path for NFS. Tag prefixes and suffixes for tag events.	—
Event type delivered	`vastdata.com:Element.<subtype>`, for example `vastdata.com:Element.ElementCreated` for `ObjectCreated:*`	`vastdata.com:Schedule.TimerElapsed`

You configure element triggers with the S3-style names above, but the events they deliver use the UI's names: a trigger set to ObjectCreated:* delivers events whose subtype is ElementCreated. Coming from the UI maps the names.

Both kinds can add custom extensions: fixed key/value pairs that the trigger adds to every event as CloudEvent attributes. Keys are lowercase letters and digits, start with a letter, and are 1 to 20 characters long.

The source view must be set up for triggers by your platform team (for NFS, it needs NFSv4 and the S3 bucket protocol, with delegations off). See setup prerequisites and triggers.

Pipelines and pipeline revisions

A pipeline holds one or more flows. Every flow starts at a trigger: links go from a trigger to function deployments, and from function deployments to other function deployments.

A pipeline's contents are kept as revisions:

vastde pipelines create saves revision 1. The pipeline's status is Draft: nothing runs yet.
vastde pipelines deploy deploys the latest revision and marks it as the deployed one. See what deploy does.
vastde pipelines update saves your changes:
- If the latest revision is deployed, it creates a new revision (N+1). The deployed revision keeps running unchanged.
- If the latest revision isn't deployed yet, it overwrites it in place.
So a run of updates between two deploys makes one new revision, not one per update. The changes don't run until you deploy.
vastde pipelines get <name> --manifest prints the latest revision's manifest (only the manifest: not the pipeline's name, cluster or namespace); add --deployed for the one that's running.

You can't deploy an older revision directly: deploy always takes the latest. To roll back, update the pipeline with the old file from Git and deploy it.

Status (CLI and API)	UI label	Meaning
`Draft`	Draft	Created, never deployed
`InProgress`	In progress	Deploying
`Ready`	Running	Deployed and running
`Error`	Failure	The deploy failed or the deployed pipeline isn't healthy. The pipeline's `reason` says why.
`Deleting`	—	Being removed from the compute cluster

Links and routing

A link sends every event from its sources to all of its destinations. In the YAML, sources and destinations are the local names you give triggers and function deployments in the same file.

Trigger to function. The function receives the trigger's events, from the trigger's topic.
Function to function. The function's return value becomes a new event, of type vastdata.com:Function, for the destinations. A link can name the topic that carries it; otherwise the tenant's default topic is used.
Conditional routing. A link can list labels (conditional_trigger_labels). A function sets labels on an event with event.set_trigger_labels(); the event goes down a link only if it has all the labels the link lists. Routing is hop by hop: labels decide only the next link. Label names and values have strict patterns; see pipeline YAML.
Per-link settings. retries (how many times a failed delivery is retried after the first attempt: retries: 2 means up to three attempts) and events_order: ordered processes one event per topic partition at a time, so concurrency is capped at the topic's partition count; unordered processes events concurrently.

Events

Every event is a CloudEvent v1.0. The attributes you use most:

Attribute	Holds
`type`	`vastdata.com:<Type>[.<SubType>]`. The type is `Element`, `Schedule` or `Function`.
`source`	Where the event came from: `vastdata.com:<trigger name>.<trigger ID>`, or the upstream function
`subject`	The broker and topic the event travelled on, as `<broker>.<topic>`. With VAST Event Broker, the broker part is `vast-broker-<view name>`.
`id`, `time`	The event's ID and time
`partitionkey`	Schedule and function events: the Kafka partition key. Element events don't carry it; `event.partition_key` returns their `elementpath` instead.
`elementpath`	Element events: `<bucket>/<object key>`
`cronschedule`, `timerelapsedtimestamp`	Schedule events
`data`	The payload. For S3 element events, an S3-style notification with a `Records` list. For function events, what the upstream function returned.

In code you rarely read attributes directly: the SDK's VastEvent exposes them as properties (event.type, event.subtype, event.partition_key), and event.as_element_event() gives you bucket and object_key. See events and write a function.

The SDK also has a Manual event type. No 5.5 interface is documented for sending manual events, so pipelines receive only the three types above.

VRNs

Every DataEngine object has a VRN (VAST resource name) of the form vast:dataengine:<type>:<name>. The types are functions, triggers, pipelines, topics, kubernetes-clusters, container-registries and mtls-authentication-credentials.

The pipeline YAML refers to other objects by VRN:

You refer to	As
A compute cluster	`kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:<cluster>`
A trigger	`vrn: vast:dataengine:triggers:<trigger>`
A function	`function_vrn: vast:dataengine:functions:<function>`
A topic	`topic: vast:dataengine:topics:<broker>/<topic>`

Identity policies use the same form, for example vast:dataengine:pipelines:*. See security.

Tenants and topics

DataEngine is enabled per tenant, and you use it on exactly one tenant. Everything on this page belongs to that tenant.

When your platform team enables DataEngine, they choose an event broker (a VAST Event Broker view or an external Kafka broker) and two topics on it:

The default topic. A trigger that doesn't name a broker and topic publishes here. If a trigger names a topic that doesn't exist yet, it's created. Most pipelines can use the default topic for everything.
The dead-letter topic. Events that fail end up here, once their link's retries run out: function failures, routing failures and dispatching failures. Each record is the original event with the error added in a header, so you can read it with any Kafka client. It can't be a trigger's topic, or a link's. See retries and the dead-letter topic.

Topics, partitions and retention are VAST Event Broker concepts; see Event Broker concepts. The partition count matters to DataEngine because ordered links process one event per partition at a time.

Compute clusters and namespaces

A pipeline runs on one compute cluster, in one namespace:

External Kubernetes cluster: a cluster your platform team runs, with DataEngine's services installed on it.
VAST compute cluster (new in 5.5): Kubernetes on a subset of the VAST cluster's CNodes, managed from VMS.

A namespace belongs to one tenant; tenants can't share namespaces. When your platform team attaches your tenant to a compute cluster, the cluster gets a DataEngine namespace for the tenant, named dataengine-ns-<tenant GUID>, and the cluster is linked to your tenant. Your pipelines deploy into that namespace. To find it, run vastde compute-clusters get <cluster> -o yaml and read its namespaces list, or ask your platform team.

You choose the cluster and namespace when you create the pipeline, and can't change them later. To run the same pipeline in another namespace (for example a test namespace), create a second pipeline from the same file, with that cluster and namespace.

Container registries

Function images live in an external container registry, which your platform team links to DataEngine and to one or more compute clusters and namespaces. The registry must use HTTPS with a certificate from a trusted CA. When you deploy, the function's registry must be linked to the pipeline's cluster and namespace, or the deploy fails.

Source of truth

The UI, the CLI and the REST API write the same objects, and none of them checks a version before saving. Two edits to the same pipeline don't conflict: the last save wins.

It's worse than a simple overwrite, because an update merges by section. If your file leaves out a section (for example links), the cluster keeps that section from the latest revision. If your file includes it, it replaces the whole section. So a teammate's UI change to one section survives your next CLI update, and their change to another section is overwritten by yours.

Pick one source of truth per pipeline:

Keep the pipeline YAML in Git, and change the pipeline only by updating from that file (by hand or from CI).
Treat the UI as read-only for those pipelines: use it to watch runs, not to edit.
To bring a pipeline built in the UI under Git, export it once and commit the file. See coming from the UI.

Functions and triggers have the same last-writer-wins behaviour.