---
title: "Concepts"
slug: "dev-dataengine-concepts-index"
description: "Learn about VAST DataEngine objects, pipeline revisions, and how triggers and functions interact to optimize your data workflows effectively."
tags: ["DataEngine"]
updated: 2026-10-30T18:19:01Z
published: 2026-10-30T18:19:01Z
canonical: "docs.vastdata.com/dev-dataengine-concepts-index"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Concepts

*Applies to VAST Cluster 5.5.*

This page names the objects you work with in VAST DataEngine, says how they relate, and explains the two things that most often surprise people: which function revision a pipeline runs, and what happens when the UI and the CLI edit the same pipeline. Read it before you design a pipeline.

## Objects

| Object | What it is | Defined by |
| --- | --- | --- |
| Function | A named pointer to a container image in a registry: the registry, the image's repository path (the *artifact source*) and its tag. Every change that affects the image creates a new revision. | `vastde functions create` / `update` |
| Function revision | One numbered version of a function. Numbers start at 1 and go up by one. A revision can have an alias and a description. | Created by `functions create` and `functions update` |
| Function deployment | One use of a function inside a pipeline, with its own revision, resources, scaling, batching, log level, environment variables and secrets. The same function can be deployed several times, in one pipeline or many. | `function_deployments` in the pipeline YAML |
| Trigger | A source of events: an **Element** trigger watches a view, a **Schedule** trigger fires on a cron expression. Each publishes CloudEvents to one topic. Triggers are reusable across pipelines. | `vastde triggers create element` / `schedule` |
| Pipeline | A named set of triggers, function deployments and links, deployed to one namespace of one compute cluster. Its name, cluster and namespace can't change after you create it. | `vastde pipelines create --config` |
| Pipeline revision | One saved version of the pipeline's contents (its *manifest*). At most one revision is deployed at a time. | Created by `pipelines create` and `pipelines update` |
| Link | A connection from one or more sources (triggers or function deployments) to one or more destination function deployments, with optional retries, ordering and labels. The UI calls it a connection. | `links` in the pipeline YAML |
| Event | A CloudEvent (v1.0) delivered to a function. The runtime wraps it in a `VastEvent`. | Triggers and functions |
| Topic | A Kafka topic that carries events. See [tenants and topics](/develop/docs/dev-dataengine-concepts-index#tenants-and-topics). | Your platform team, or created for a trigger |
| Compute cluster | The Kubernetes cluster that runs pipelines: an external Kubernetes cluster or a VAST compute cluster. Pipelines deploy into one of its namespaces. | Your platform team |
| Container registry | An external registry that holds function images, linked to DataEngine by your platform team. | Your platform team |

![Left: your tenant, with DataEngine enabled. It holds compute clusters and namespaces, where pipelines run; container registries, which hold the images functions point at; triggers, element or schedule, which are reusable; functions, with numbered revisions, also reusable; the default topic, used when a trigger or link names none; and the dead-letter topic, where failed events end up. Right: a pipeline, on one compute cluster and namespace, has revisions; at most one is deployed, and deploy always takes the latest. The latest revision's manifest holds trigger references, function deployments (a function and its pinned or default revision, with resources, scaling, batching, environment variables and secrets) and links (trigger to function or function to function, with topic, retries, ordering and labels). An update overwrites an undeployed latest revision, or adds a new one.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/de-object-model.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A00Z&se=2026-11-08T09%3A55%3A00Z&sr=c&sp=r&sig=Y3JrXclYk1c01hrEqse4t1fYh7wmVann1k1JDyC5u50%3D)

*Triggers and functions are reusable; a pipeline revision wires them together.*

## Functions and revisions

A function doesn't contain your code; it points at an image you've already pushed. When you rebuild and push a new image tag, you run `vastde functions update` to point the function at it, and that creates a new revision. Changing only the description doesn't.

Two settings decide which revisions are used:

- **Default revision.** A function can name a default revision (`default_revision_number`). If it doesn't, the latest revision is the default.
- **Published.** A revision can be marked published (`--publish` on `functions create` or `update`). The API describes an unpublished revision as one other users can't use.

The image must already be in the registry when you create the revision: the cluster checks that it can reach it.

### Which revision runs

Each function deployment in a pipeline either **pins** a revision or leaves it **unpinned**:

| In the pipeline YAML | Revision that runs |
| --- | --- |
| `revision: 3` (pinned) | Revision 3, until you change the YAML and deploy again |
| `revision` left out or null (unpinned) | The function's default revision, which is the latest revision if no default is set |

The cluster resolves the revision **when you deploy the pipeline**, not when the function changes. So a new function revision never changes a running pipeline by itself. A pinned deployment keeps its revision until you edit the YAML. An unpinned one picks up the new default the next time anyone deploys the pipeline, for any reason.

Pin revisions in pipelines you promote between environments, so that a deploy runs exactly what you tested. See [publish functions](/develop/docs/dev-dataengine-guides-publish-functions).

## Triggers

An element trigger watches **elements**: the files and S3 objects under a view's path in the cluster's Element Store (background: [the cluster underneath](/develop/docs/dev-database-concepts-architecture-cluster-architecture#one-element-store)). Its source types (below) decide which protocol's changes it reports.

|  | Element trigger | Schedule trigger |
| --- | --- | --- |
| Fires when | An element is created or deleted in a view, or an S3 object's tags are added or removed | A cron expression comes due |
| Event types (CLI names) | `ObjectCreated:*`, `ObjectRemoved:*`, `ObjectTagging:Put`, `ObjectTagging:Delete` | — |
| Source | A view's bucket. Elements are S3 objects or NFSv4 files (`source_types` `S3`, `NFS4`). Tag events are S3 only. | — |
| Filters | Name prefixes and suffixes: the object key for S3, the file path for NFS. Tag prefixes and suffixes for tag events. | — |
| Event type delivered | `vastdata.com:Element.<subtype>`, for example `vastdata.com:Element.ElementCreated` for `ObjectCreated:*` | `vastdata.com:Schedule.TimerElapsed` |

You configure element triggers with the S3-style names above, but the events they deliver use the UI's names: a trigger set to `ObjectCreated:*` delivers events whose subtype is `ElementCreated`. [Coming from the UI](/develop/docs/dev-dataengine-reference-coming-from-the-ui) maps the names.

Both kinds can add **custom extensions**: fixed key/value pairs that the trigger adds to every event as CloudEvent attributes. Keys are lowercase letters and digits, start with a letter, and are 1 to 20 characters long.

The source view must be set up for triggers by your platform team (for NFS, it needs NFSv4 and the S3 bucket protocol, with delegations off). See [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites) and [triggers](/develop/docs/dev-dataengine-guides-triggers).

## Pipelines and pipeline revisions

A pipeline holds one or more flows. Every flow starts at a trigger: links go from a trigger to function deployments, and from function deployments to other function deployments.

A pipeline's contents are kept as revisions:

1. `vastde pipelines create` saves revision 1. The pipeline's status is **Draft**: nothing runs yet.
2. `vastde pipelines deploy` deploys the **latest** revision and marks it as the deployed one. See [what deploy does](/develop/docs/dev-dataengine-concepts-architecture#what-deploy-does).
3. `vastde pipelines update` saves your changes:

So a run of updates between two deploys makes one new revision, not one per update. The changes don't run until you deploy.
  - If the latest revision is deployed, it creates a new revision (N+1). The deployed revision keeps running unchanged.
  - If the latest revision isn't deployed yet, it overwrites it in place.
4. `vastde pipelines get <name> --manifest` prints the latest revision's manifest (only the manifest: not the pipeline's name, cluster or namespace); add `--deployed` for the one that's running.

You can't deploy an older revision directly: deploy always takes the latest. To roll back, update the pipeline with the old file from Git and deploy it.

| Status (CLI and API) | UI label | Meaning |
| --- | --- | --- |
| `Draft` | Draft | Created, never deployed |
| `InProgress` | In progress | Deploying |
| `Ready` | Running | Deployed and running |
| `Error` | Failure | The deploy failed or the deployed pipeline isn't healthy. The pipeline's `reason` says why. |
| `Deleting` | — | Being removed from the compute cluster |

## Links and routing

A link sends every event from its sources to all of its destinations. In the YAML, sources and destinations are the local names you give triggers and function deployments in the same file.

- **Trigger to function.** The function receives the trigger's events, from the trigger's topic.
- **Function to function.** The function's return value becomes a new event, of type `vastdata.com:Function`, for the destinations. A link can name the topic that carries it; otherwise the tenant's default topic is used.
- **Conditional routing.** A link can list labels (`conditional_trigger_labels`). A function sets labels on an event with `event.set_trigger_labels()`; the event goes down a link only if it has **all** the labels the link lists. Routing is hop by hop: labels decide only the next link. Label names and values have strict patterns; see [pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml).
- **Per-link settings.** `retries` (how many times a failed delivery is retried after the first attempt: `retries: 2` means up to three attempts) and `events_order`: `ordered` processes one event per topic partition at a time, so concurrency is capped at the topic's partition count; `unordered` processes events concurrently.

## Events

Every event is a CloudEvent v1.0. The attributes you use most:

| Attribute | Holds |
| --- | --- |
| `type` | `vastdata.com:<Type>[.<SubType>]`. The type is `Element`, `Schedule` or `Function`. |
| `source` | Where the event came from: `vastdata.com:<trigger name>.<trigger ID>`, or the upstream function |
| `subject` | The broker and topic the event travelled on, as `<broker>.<topic>`. With VAST Event Broker, the broker part is `vast-broker-<view name>`. |
| `id`, `time` | The event's ID and time |
| `partitionkey` | Schedule and function events: the Kafka partition key. Element events don't carry it; `event.partition_key` returns their `elementpath` instead. |
| `elementpath` | Element events: `<bucket>/<object key>` |
| `cronschedule`, `timerelapsedtimestamp` | Schedule events |
| `data` | The payload. For S3 element events, an S3-style notification with a `Records` list. For function events, what the upstream function returned. |

In code you rarely read attributes directly: the SDK's `VastEvent` exposes them as properties (`event.type`, `event.subtype`, `event.partition_key`), and `event.as_element_event()` gives you `bucket` and `object_key`. See [events](/develop/docs/dev-dataengine-reference-events) and [write a function](/develop/docs/dev-dataengine-guides-write-a-function).

The SDK also has a `Manual` event type. No 5.5 interface is documented for sending manual events, so pipelines receive only the three types above.

## VRNs

Every DataEngine object has a VRN (VAST resource name) of the form `vast:dataengine:<type>:<name>`. The types are `functions`, `triggers`, `pipelines`, `topics`, `kubernetes-clusters`, `container-registries` and `mtls-authentication-credentials`.

The pipeline YAML refers to other objects by VRN:

| You refer to | As |
| --- | --- |
| A compute cluster | `kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:<cluster>` |
| A trigger | `vrn: vast:dataengine:triggers:<trigger>` |
| A function | `function_vrn: vast:dataengine:functions:<function>` |
| A topic | `topic: vast:dataengine:topics:<broker>/<topic>` |

Identity policies use the same form, for example `vast:dataengine:pipelines:*`. See [security](/develop/docs/dev-dataengine-operations-security).

## Tenants and topics

DataEngine is enabled per tenant, and you use it on exactly one tenant. Everything on this page belongs to that tenant.

When your platform team enables DataEngine, they choose an event broker (a VAST Event Broker view or an external Kafka broker) and two topics on it:

- **The default topic.** A trigger that doesn't name a broker and topic publishes here. If a trigger names a topic that doesn't exist yet, it's created.  Most pipelines can use the default topic for everything.
- **The dead-letter topic.** Events that fail end up here, once their link's retries run out: function failures, routing failures and dispatching failures. Each record is the original event with the error added in a header, so you can read it with any Kafka client. It can't be a trigger's topic, or a link's.  See [retries and the dead-letter topic](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic).

Topics, partitions and retention are VAST Event Broker concepts; see [Event Broker concepts](/develop/docs/dev-event-broker-concepts-index). The partition count matters to DataEngine because ordered links process one event per partition at a time.

## Compute clusters and namespaces

A pipeline runs on one compute cluster, in one namespace:

- **External Kubernetes cluster**: a cluster your platform team runs, with DataEngine's services installed on it.
- **VAST compute cluster** (new in 5.5): Kubernetes on a subset of the VAST cluster's CNodes, managed from VMS.

A namespace belongs to one tenant; tenants can't share namespaces. When your platform team attaches your tenant to a compute cluster, the cluster gets a DataEngine namespace for the tenant, named `dataengine-ns-<tenant GUID>`, and the cluster is linked to your tenant. Your pipelines deploy into that namespace. To find it, run `vastde compute-clusters get <cluster> -o yaml` and read its `namespaces` list, or ask your platform team.

You choose the cluster and namespace when you create the pipeline, and can't change them later. To run the same pipeline in another namespace (for example a test namespace), create a second pipeline from the same file, with that cluster and namespace.

## Container registries

Function images live in an external container registry, which your platform team links to DataEngine and to one or more compute clusters and namespaces. The registry must use HTTPS with a certificate from a trusted CA. When you deploy, the function's registry must be linked to the pipeline's cluster and namespace, or the deploy fails.

## Source of truth

The UI, the CLI and the REST API write the same objects, and none of them checks a version before saving. Two edits to the same pipeline don't conflict: the last save wins.

It's worse than a simple overwrite, because an update merges by section. If your file leaves out a section (for example `links`), the cluster keeps that section from the latest revision. If your file includes it, it replaces the whole section. So a teammate's UI change to one section survives your next CLI update, and their change to another section is overwritten by yours.

Pick one source of truth per pipeline:

- Keep the pipeline YAML in Git, and change the pipeline only by updating from that file (by hand or from CI).
- Treat the UI as read-only for those pipelines: use it to watch runs, not to edit.
- To bring a pipeline built in the UI under Git, export it once and commit the file. See [coming from the UI](/develop/docs/dev-dataengine-reference-coming-from-the-ui).

Functions and triggers have the same last-writer-wins behaviour.

## See also

- [Architecture](/develop/docs/dev-dataengine-concepts-architecture): what runs where, and the path of one event.
- [Pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code): the pipeline YAML end to end.
- [Pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml): every field.
- [VAST Event Broker concepts](/develop/docs/dev-event-broker-concepts-index): topics, partitions and retention.