---
title: "Architecture"
slug: "dev-dataengine-concepts-architecture"
description: "Explore how VAST DataEngine processes events from S3 uploads, manages retries, scaling, and telemetry for efficient pipeline deployment and debugging."
tags: ["DataEngine"]
updated: 2026-10-28T02:14:59Z
published: 2026-10-28T02:14:59Z
canonical: "docs.vastdata.com/dev-dataengine-concepts-architecture"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture

*Applies to VAST Cluster 5.5.*

This page explains what VAST DataEngine runs where, the path one event takes from an S3 upload through your functions, what `vastde pipelines deploy` does, and how retries, scaling and telemetry work. You don't need it to run the quickstart, but read it before you size a pipeline or debug one that's stuck or dropping events.

## What runs where

| Where | What runs there |
| --- | --- |
| **VAST cluster: VMS** | DataEngine's control plane. It stores your functions, triggers and pipelines, checks and deploys pipelines, runs schedule triggers , and answers telemetry queries. `vastde`, the Web UI and the REST API all talk to it. |
| **VAST cluster: your views** | Element events. When an object or file changes in a view that an element trigger watches, the cluster publishes a CloudEvent to the trigger's topic. |
| **Event broker** | The topics: each trigger's topic, the tenant's default topic and its dead-letter topic. Usually a VAST Event Broker view, served by the cluster's CNodes; it can be an external Kafka broker. |
| **Compute cluster** | Your functions. DataEngine's services, Knative Serving and Knative Eventing are installed there by your platform team. Each pipeline gets Kubernetes resources in its namespace: one autoscaling Knative service per function deployment, and the consumers and delivery rules that move events from topics to functions. The namespace is your tenant's DataEngine namespace, `dataengine-ns-<tenant GUID>`. |
| **Compute cluster: telemetry collector** | Receives logs, traces and metrics from your functions and writes them to tables on the VAST cluster. |
| **Container registry** | Your function images. The compute cluster pulls them when a pipeline deploys or scales up. |

![You use vastde, the Web UI or the REST API, which all talk to VMS on the VAST cluster. VMS is DataEngine's control plane: it stores functions, triggers and pipelines, deploys pipelines to the compute cluster, runs schedule triggers and answers telemetry queries. When an object or file changes in a view, the cluster publishes an element event to a topic on the event broker, which also holds the default and dead-letter topics. On the compute cluster, each pipeline's namespace has one autoscaling Knative service per function deployment, and consumers and delivery rules that move events from topics to functions over HTTP. The compute cluster pulls function images from the container registry. A telemetry collector on the compute cluster receives logs, traces and metrics and writes them to VAST DataBase tables, which VMS queries. Schedule triggers don't fire while the VMS is down.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/de-runs-where.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A02Z&se=2026-11-08T09%3A55%3A02Z&sr=c&sp=r&sig=5pbHqXv2CHkSZBuZeSwMP0p7SofA7O%2F%2FOyEIUopHxzU%3D)

*VMS on the VAST cluster controls everything; your functions run on the compute cluster, fed from topics on the event broker.*

Schedule triggers depend on VMS: in 5.5 they don't fire while the VMS is down, for example during an upgrade or an HA failover. Element triggers are unaffected.

## The path of one event

Take a pipeline with an element trigger on bucket `images`, a function deployment `thumbnail` linked from it, and a function deployment `index` linked from `thumbnail`:

1. **An object lands.** A client uploads `images/cat.jpg` over S3.
2. **The trigger publishes.** The object's name passes the trigger's filters, so the cluster publishes a CloudEvent of type `vastdata.com:Element.ElementCreated` to the trigger's topic, with `elementpath` set to `images/cat.jpg`. Its `data` is an S3-style notification record for the upload. See [events](/develop/docs/dev-dataengine-reference-events#element-events).
3. **The compute cluster consumes.** In the pipeline's namespace, a consumer reads the trigger's topic, and the link's delivery rule sends the event over HTTP to one instance of `thumbnail`. If no instance is free, the service scales up.
4. **The handler runs.** The DataEngine runtime in the container turns the request into a `VastEvent` and calls your `handler(ctx, event)`. If the handler returns normally, the delivery succeeds. If it raises, the delivery fails and is retried (see [retries](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic)).
5. **The return value becomes the next event.** The runtime sends what the handler returned as a new CloudEvent of type `vastdata.com:Function`, whose source is `thumbnail`. It keeps the incoming event's subject, ID and partition key, and carries any labels the handler set. A handler that returns `None` still sends an event, with no data. If no link starts at the function, the return value is discarded.
6. **The next link routes it.** The event travels on the link's topic (the default topic if the link names none). The delivery rule for the `thumbnail` → `index` link matches events from `thumbnail` that carry all of the link's labels, and sends them to `index`.
7. **The trace follows it.** Each step adds spans to the event's trace, so `vastde traces get <trace-id>` and the UI's trace view show the whole path.

Nothing goes back to the original writer: the S3 upload completed at step 1. Element events are asynchronous.

![Step 1: a client uploads images/cat.jpg over S3. Step 2: the trigger publishes an Element.ElementCreated CloudEvent to its topic, with elementpath images/cat.jpg. Steps 3 and 4: a consumer on the compute cluster delivers it over HTTP to an instance of thumbnail, which runs handler(ctx, event). Step 5: the return value is sent as a Function event on the link's topic, the default topic if the link names none. Step 6: index receives the events from thumbnail that carry all of the link's labels. If a handler raises or the call times out, the delivery is retried up to the link's retries with increasing waits, and the handler runs again; when the retries run out, the event goes to the dead-letter topic. Each link retries the same way. Step 7: every step adds spans to the event's trace. The upload completed at step 1: element events are asynchronous.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/de-event-path.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A02Z&se=2026-11-08T09%3A55%3A02Z&sr=c&sp=r&sig=5pbHqXv2CHkSZBuZeSwMP0p7SofA7O%2F%2FOyEIUopHxzU%3D)

*Each hop is a topic, a delivery with retries, and an HTTP call to a function instance.*

If `thumbnail` is configured for [batching](/develop/docs/dev-dataengine-concepts-architecture#batching), step 4 waits until enough events arrive or the batch timeout passes, then calls the handler once with the whole batch. Each item in the list it returns is sent on as its own event.

## What deploy does

`vastde pipelines deploy <pipeline>` (or `pipelines create --deploy`) deploys the pipeline's **latest** revision:

1. **Checks.** The revision must have at least one trigger, at least one link, and at least one link from a trigger. Every secret it names must exist in the pipeline's namespace, and every function's container registry must be linked to the pipeline's cluster and namespace.  If a check fails, the command fails and the pipeline stays as it was; after `pipelines create --deploy`, that means the pipeline exists, in `Draft`. Other mistakes, such as a link that names an unknown function deployment, are caught earlier, when you create or update the pipeline. See the [error reference](/develop/docs/dev-dataengine-troubleshooting-errors#pipelines).
2. **Resolves revisions.** Each function deployment gets its pinned revision, or the function's default revision if it's unpinned. See [which revision runs](/develop/docs/dev-dataengine-concepts-index#which-revision-runs).
3. **Applies it to the compute cluster.** VMS creates or updates the pipeline's Kubernetes resources in its namespace, and marks this revision as the deployed one.
4. **Status `InProgress`.** The command returns once the request is accepted; Kubernetes is still pulling images and starting services, so check the status before you send events.
5. **Status `Ready` or `Error`.** In the background, the cluster checks the pipeline's resources on the compute cluster and updates its status.

```
Draft ──deploy──▶ InProgress ──▶ Ready
                       │
                       └──────▶ Error (reason)
```

`Ready` means every part of the pipeline reports ready: the function services, the consumers on the trigger topics and the delivery rules for the links. `Error` means at least one doesn't, and the pipeline's `reason` carries that part's own message. For a function whose container exits at start-up, for example because `main.py` imports a package that isn't installed, the reason includes the container's output with the Python traceback. The pipeline stays `InProgress` for several minutes before it goes to `Error`, and `vastde logs` shows nothing for that function, because it never starts. See [troubleshooting](/develop/docs/dev-dataengine-troubleshooting-index#the-pipeline-stays-inprogress-then-goes-to-error). If the pipeline's resources have disappeared from the namespace, the reason says the pipeline wasn't found there.

Read the status with `vastde pipelines get <pipeline>`, and the reason with `vastde pipelines get <pipeline> -o yaml` (the default output doesn't show it). A check that fails in step 1 fails the command itself, and nothing changes on the compute cluster.

Updating a pipeline doesn't touch what's running. The deployed revision keeps running until you deploy again. Deleting a pipeline sets it to `Deleting`, removes its resources from the compute cluster, and then removes the pipeline.

## Retries and the dead-letter topic

Each link delivers events to its destination functions over HTTP. A delivery **fails** when the handler raises an exception, when the call times out, or when the runtime can't send the handler's result on to the next link.

1. A failed delivery is retried, up to the link's `retries` setting, with a wait between attempts that grows each time. `retries` counts the deliveries after the first: with `retries: 2`, the handler ran three times. If you leave `retries` out, a server default applies, which can keep retrying a failing event for a long time before it's dead-lettered, so set `retries` explicitly on every link.
2. When the retries run out, the event goes to the tenant's **dead-letter topic**, with the other function, routing and dispatching failures.

Every attempt runs in the same trace, so `vastde traces list` shows one trace with status `ERROR` that lasts from the first attempt to the last. Each attempt also logs an `Error in handler execution: <message>` record at `ERROR`.

Consequences for your code:

- **A handler can see the same event more than once.** A retry calls the handler again, even if the first attempt did its work and only failed afterwards. Make handlers idempotent: write output under names derived from the input, use upserts, or check before you write.
- **Exceptions are your failure signal.** Raise when you want the event retried and, eventually, dead-lettered. Catch and log when you don't.
- **Watch the dead-letter topic.** It's a topic on the tenant's broker, so you can consume it with a Kafka client. Every pipeline in the tenant shares it. See [the dead-lettered record](/develop/docs/dev-dataengine-concepts-architecture#the-dead-lettered-record).

On a 5.5 cluster, an event whose handler failed was delivered again, with the same `id`, on every retry, and then reached the dead-letter topic; no events were lost. Design for at-least-once delivery, and don't rely on events never being lost or never being repeated.

### The dead-lettered record

A dead-lettered record is the original event, unchanged, written to the dead-letter topic as a Kafka record whose headers hold the CloudEvent attributes (`ce_id`, `ce_type`, `ce_source`, `ce_subject`, `ce_time`, the extensions such as `ce_elementpath`) and whose value is the original `data`. Three headers are added:

| Header | Contents |
| --- | --- |
| `ce_knativeerrorcode` | The HTTP status of the last failed attempt, for example `500` when the handler raised |
| `ce_knativeerrordata` | The failed attempt's response body, base64-encoded. When the handler raised, it's the exception's message. |
| `ce_knativeerrordest` | The address of the function deployment that failed |

For the handler above, which raised `ValueError(f"cannot process {event.object_key}")`, the record for `fail/broken.txt` had these headers, among others:

```
ce_id: 88051124535306
ce_type: vastdata.com:Element.ElementCreated
ce_elementpath: docs-src/fail/broken.txt
ce_knativeerrorcode: 500
ce_knativeerrordata: Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0
```

`ce_knativeerrordata` decodes to the exception's message:

```
$ echo Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0 | base64 -d; echo
cannot process fail/broken.txt
```

To replay or alert on dead letters, consume the topic with any Kafka client. The record identifies the event (`ce_id`, `ce_source` for the trigger, `ce_elementpath` for the object) but not the pipeline, so if several pipelines share a trigger, use `ce_knativeerrordest` or the trigger to tell them apart.

### Ordering

A link's `events_order` decides ordering. With `ordered`, a destination processes one event per topic partition at a time, in order, so its concurrency is capped at the topic's partition count. With `unordered`, events are processed concurrently and can finish in any order. Events for the same object are expected to land in the same partition.

## Scaling and concurrency

Each function deployment is its own Knative service, so each step of a pipeline scales on its own. You set the bounds in the pipeline YAML, under the deployment's `resources`:

- **Instances.** `min_concurrency` and `max_concurrency` are the minimum and maximum number of instances (pods). The autoscaler adds and removes instances between them, using the deployment's `autoscaling` metric: concurrent requests (`concurrency`), requests per second (`rps`), or CPU or memory use (`resource`). If you don't give one, the cluster adds a concurrency-based block when it stores the pipeline.  The autoscaling settings also have a scale-to-zero delay, for deployments allowed to scale down to no instances.
- **Requests per instance.** `max_concurrent_requests_limit` caps how many requests one instance handles at once (Knative's container concurrency; 0 means no limit). If you leave it out, it's computed from `num_workers` and the batch size.
- **Threads.** A synchronous (`def`) handler runs in a pool of `num_workers` threads, so one instance runs at most that many calls at once; it refuses calls beyond that. An `async def` handler runs on the instance's event loop and isn't limited by the pool.
- **CPU, memory, storage** (`min_cpu`, `max_cpu`, `min_memory`, `max_memory`, `storage`) are per instance.
- **Start-up.** `init(ctx)` runs once when an instance starts, and the instance gets no events until `init` has finished.  Put slow set-up (loading a model, opening clients) in `init`, not in the handler.
- **Timeout.** The deployment's `timeout` (seconds) limits how long one call can take. The runtime itself has no handler timeout.  With batching, the timeout must be longer than the batch timeout.

An `ordered` link caps concurrency at the topic's partition count whatever these settings say. See [pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml) for every field and its limits.

### Batching

With `config.event_batching` set on a function deployment (`batch_max_size` and `batch_timeout_ms`), the instance collects events and calls the handler with a list when the batch is full or the timeout passes, whichever comes first. The handler must return a list of the same length, and each item is sent downstream as its own event. A batch has its own trace, and each event in it keeps its own. See [write a function](/develop/docs/dev-dataengine-guides-write-a-function).

## Logs, traces and metrics

The runtime in each function instance sends telemetry over OpenTelemetry to DataEngine's telemetry collector on the compute cluster. The collector writes it into VAST DataBase tables on the VAST cluster, and VMS answers queries over them.

| Signal | What's captured | Read it with |
| --- | --- | --- |
| Logs | `ctx.logger`, any Python `logging` logger, and `print()` (logged at INFO). Output to `stderr` that doesn't go through `logging` isn't captured. Each log has a scope: `user` for your code, `runtime` for the DataEngine runtime (shown as `[vast-runtime]` in `vastde logs get` output). | `vastde logs get`, `vastde logs tail`; the UI's Logs page |
| Traces | One trace per event, across every function it passes through, with spans for each call and any spans you add (`ctx.tracer`, `@span_wrapped`, `@event_span_wrapped`). Each instance's `init` call gets a trace too, so the first event on a new instance shows up as two traces. Batches have their own trace. | `vastde traces list`, `vastde traces get`; the UI's trace views |
| Metrics | Automatic metrics: `dataengine.function.invocations`, `dataengine.function.invoke_duration`, `dataengine.event.total` and `dataengine.event.reception_latency` (and `dataengine.function.errors` for failures ), plus counters, histograms, gauges and up-down counters you create from `ctx`. | `vastde metrics get`, `vastde metrics names`; the UI dashboard |

Telemetry is kept for a limited time, not forever. Copy anything you need to keep into your own tables.

`vastde functions localrun` runs your function in a local container that logs to the console only; nothing reaches the collector. See [observability](/develop/docs/dev-dataengine-guides-observability) and [build and test locally](/develop/docs/dev-dataengine-guides-build-and-test-locally).

## See also

- [Concepts](/develop/docs/dev-dataengine-concepts-index): the objects, revisions and the source of truth.
- [Pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code): resources, autoscaling, batching and links in the YAML.
- [Troubleshooting](/develop/docs/dev-dataengine-troubleshooting-index): deploy errors and failing events.
- [VAST Event Broker architecture](/develop/docs/dev-event-broker-concepts-architecture): how the cluster serves the topics.