Applies to VAST Cluster 5.5.
This guide shows how to manage a DataEngine pipeline from a YAML file kept in Git: create and deploy it, change it with update and deploy, read its revisions, export an existing pipeline into a file, chain functions, route events, and set retries and read the dead-letter topic. It ends with the rule that keeps this working: one source of truth per pipeline.
It builds on the quickstart's thumbnails pipeline, with thumbnailer at revision 2 (publish functions). Commands use the environment variables from the quickstart's before you start. The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6 with vastde v5.5.0-sp2, where the compute cluster is test-cluster and the tenant's namespace is dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91. Your names, GUIDs, addresses and times will differ. Every field of the file is in the pipeline YAML reference.
Lay out a repository
Keep everything a pipeline needs, except secrets, in one repository:
thumbnails/
├── pipeline.yaml # the pipeline: triggers, function deployments, links
├── triggers/ # trigger definitions, or a script of vastde triggers create commands
├── functions/
│ ├── thumbnailer/ # a function project from vastde functions init
│ │ ├── main.py
│ │ └── requirements.txt
│ └── indexer/
└── tests/events/ # event files for vastde functions invoke
Secret files (for --secret-file) stay out of Git; generate them from your secrets store in CI. The pipeline file names secrets but holds no values. See security.
The pipeline file
A pipeline file has four top-level fields and a manifest:
name: thumbnails
description: Make a thumbnail for every new image in docs-src/photos/
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
triggers:
- name: uploads
vrn: vast:dataengine:triggers:docs-src-uploads
function_deployments:
- name: thumbnailer
function_vrn: vast:dataengine:functions:thumbnailer
links:
- source: [uploads]
destination: [thumbnailer]
name,kubernetes_cluster_vrnandnamespaceare fixed when you create the pipeline. To move a pipeline, create a new one from the same file under another name.- The manifest refers to triggers and functions that already exist, by VRN, and gives each a local name. Links use the local names.
- Leave out what you don't need: the cluster fills in default
resourcesfor each function deployment.
Create and deploy
vastde pipelines create --config @pipeline.yaml saves the pipeline as revision 1, with status Draft; --deploy also deploys it. The quickstart shows the output, and the status going from InProgress to Ready.
The cluster checks the file in two stages. Create (and update) check what the file refers to, and refuse a bad file outright:
vastde pipelines create --config @bad-link.yaml --deploy
pipeline creation failed: Links in manifest refer to unknown sources or destinations: {'thumbnail'}
| File problem | Error |
|---|---|
A link names something that isn't in triggers or function_deployments |
pipeline creation failed: Links in manifest refer to unknown sources or destinations: {'thumbnail'} |
| A misspelled function VRN | pipeline creation failed: Functions with VRN {'vast:dataengine:functions:thumbnailr'} not found |
A namespace that isn't in the cluster (here default) |
pipeline creation failed: Namespace default is not in the Kubernetes cluster test-cluster |
No kubernetes_cluster_vrn |
pipeline creation failed: Kubernetes cluster vrn is required to validate namespace |
Deploy checks that the pipeline can run. A pipeline with no trigger, or with no links, is created but fails to deploy:
vastde pipelines create --config @no-link.yaml --deploy
Pipeline created: errors-demo
Name: errors-demo
Tags: []
GUID: 39c285c4-dfc7-4bc0-85d3-3a50d0171600
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:47:44Z
Updated At: 2026-10-08T12:47:44Z
VRN: vast:dataengine:pipelines:errors-demo
Deploying pipeline: errors-demo
pipeline deployment failed: Failed to deploy pipeline. Pipeline GUID: 39c285c4-dfc7-4bc0-85d3-3a50d0171600. Hint: Unexpected error: BadRequest: 400: Pipeline manifest must contain at least 1 link in the 'links' section
Without a trigger, the hint is Issue with trigger or topic configuration. Verify trigger and topic settings. A missing secret also fails here; see write a function. Either way the pipeline is left behind as a Draft. This is the one created without a trigger:
vastde pipelines get errors-demo
Name: errors-demo
Tags: []
GUID: 60edfc3b-5d5b-43fd-be4d-628d242f7621
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:48:38Z
Updated At: 2026-10-08T12:48:38Z
VRN: vast:dataengine:pipelines:errors-demo
Status: Draft
Fix the file and run vastde pipelines update and vastde pipelines deploy, or delete it and create it again:
vastde pipelines delete errors-demo
Deleting pipeline: 60edfc3b-5d5b-43fd-be4d-628d242f7621
Pipeline deleted successfully
In CI, run create (the first time) or update, check the exit status, then deploy, and poll vastde pipelines get <name> until it's Ready or Error.
Add a function to the chain
To change a pipeline, edit the file and run update, then deploy. This edit adds the indexer function (registered here) and a link that sends thumbnailer's return values to it:
name: thumbnails
description: Make a thumbnail for every new image in docs-src/photos/, then index it
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
triggers:
- name: uploads
vrn: vast:dataengine:triggers:docs-src-uploads
function_deployments:
- name: thumbnailer
function_vrn: vast:dataengine:functions:thumbnailer
- name: indexer
function_vrn: vast:dataengine:functions:indexer
links:
- source: [uploads]
destination: [thumbnailer]
- source: [thumbnailer]
destination: [indexer]
vastde pipelines update thumbnails --config @pipeline.yaml
Pipeline updated successfully
Name: thumbnails
Description: Make a thumbnail for every new image in docs-src/photos/, then index it
Tags: []
GUID: c7f63133-3438-4cae-af6a-91291041f1b6
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:25:11Z
Updated At: 2026-10-08T12:43:33Z
VRN: vast:dataengine:pipelines:thumbnails
Status: Ready
Status: Ready is the deployed revision's status: the change isn't running yet. Deploy it:
vastde pipelines deploy thumbnails
Pipeline deployed successfully
Once vastde pipelines get thumbnails shows Ready, --deployed shows what's running, with the resources the cluster filled in:
vastde pipelines get thumbnails --deployed -o yaml
function_deployments:
- function_vrn: vast:dataengine:functions:thumbnailer
name: thumbnailer
resources:
autoscaling:
…
num_workers: 20
revision: null
- function_vrn: vast:dataengine:functions:indexer
name: indexer
resources:
autoscaling:
…
num_workers: 20
revision: null
links:
- destination:
- thumbnailer
source:
- uploads
- destination:
- indexer
source:
- thumbnailer
triggers:
- name: uploads
vrn: vast:dataengine:triggers:docs-src-uploads
An upload now runs both functions; the logs are in write a function. The whole chain is one trace, listed under one of its functions:
vastde traces list thumbnails --since 1m
Trace ID Status Pipeline Function Start Time End Time
--------------------------------------------------------------------------------------------------------------------------------------
0000500500000004d94772b02eb941df OK thumbnails thumbnailer 2026-10-08 12:45:01.778 2026-10-08 12:45:01.961
A function-to-function link uses the tenant's default topic unless you give it a topic. See topics on links.
Conditional routing
A link from a function can carry conditional_trigger_labels. The upstream function sets labels on the events it returns with event.set_trigger_labels({...}), and the event goes down a link only if it has every label the link lists, with the same values:
links:
- source: [thumbnailer]
destination: [indexer]
conditional_trigger_labels:
- triggerLabelName: kind
triggerLabelValue: photo
Labels decide only the next hop, and only links that start at a function can have them. Names are a lowercase letter followed by up to 14 lowercase letters or digits, and values are 1 to 64 letters or digits: no underscores. See conditional routing and the Runtime SDK reference.
Updates, revisions and deploys
Each saved version of the manifest is a revision. Three rules:
- Update saves; deploy runs.
vastde pipelines updatenever changes what's running. The pinning example in publish functions shows it: after an update,--manifestshowed the newrevision: 1while--deployedstill showedrevision: null, until the deploy. - Deploy takes the latest revision. There's no command to deploy an older one. To roll back, check out the earlier file from Git, update and deploy.
- Updates between deploys collapse. An update after a deploy adds a revision; further updates before the next deploy overwrite that same revision.
vastde pipelines get <name> -o yaml shows the latest revision number and the status's reason:
vastde pipelines get thumbnails -o yaml
created_at: "2026-10-08T12:25:11.843Z"
description: Make a thumbnail for every new image in docs-src/photos/
guid: c7f63133-3438-4cae-af6a-91291041f1b6
id: 7.513937540412342e+18
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
last_revision_number: 2
name: thumbnails
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
owner:
id: "4294967295"
id_type: vid
name: root
reason: Pipeline is ready and active.
status: Ready
tenant_guid: da47bae0-aefd-4e85-a5a9-2020d6a2cb91
updated_at: "2026-10-08T12:37:12.279Z"
vrn: vast:dataengine:pipelines:thumbnails
Send the whole file every time. A manifest section you leave out (triggers, function_deployments, links or config) is kept from the latest revision, and a section you include replaces it entirely. Top-level fields aren't kept: an update without description clears the description, as the round trip below shows. A complete file from Git gives the same result whatever the cluster holds. To remove everything in a section, send it empty (links: []).
Export an existing pipeline
To put a pipeline that was built in the VAST Web UI, or by someone else, under Git, export its manifest. vastde pipelines get <name> --manifest -o yaml prints only the manifest, with the resources the cluster filled in:
vastde pipelines get thumbnails --manifest -o yaml
function_deployments:
- function_vrn: vast:dataengine:functions:thumbnailer
name: thumbnailer
resources:
autoscaling:
concurrency:
burst_threshold_percentage: 200
burst_window_percentage: 10
scale_down_delay_seconds: 300
scale_to_zero_delay_seconds: 600
stable_window_seconds: 60
target: 20
target_utilization_percentage: 70
metric: concurrency
autoscaling_concurrency_overload_factor: null
autoscaling_rps_factor: null
max_concurrency: 10
max_cpu: 500m
max_memory: 256Mi
min_concurrency: 1
min_cpu: 100m
min_memory: 128Mi
num_workers: 20
revision: null
links:
- destination:
- thumbnailer
source:
- uploads
triggers:
- name: uploads
vrn: vast:dataengine:triggers:docs-src-uploads
That isn't a file update accepts yet: it has no name, kubernetes_cluster_vrn or namespace. Save it, and wrap it under a manifest: key with those three fields, which vastde pipelines get <name> -o yaml shows:
vastde pipelines get thumbnails --manifest -o yaml > manifest-before.yaml
{ printf 'name: thumbnails\nkubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster\nnamespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91\nmanifest:\n'; sed 's/^/ /' manifest-before.yaml; } > roundtrip.yaml
cat roundtrip.yaml
name: thumbnails
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
function_deployments:
- function_vrn: vast:dataengine:functions:thumbnailer
name: thumbnailer
resources:
autoscaling:
concurrency:
burst_threshold_percentage: 200
burst_window_percentage: 10
scale_down_delay_seconds: 300
scale_to_zero_delay_seconds: 600
stable_window_seconds: 60
target: 20
target_utilization_percentage: 70
metric: concurrency
autoscaling_concurrency_overload_factor: null
autoscaling_rps_factor: null
max_concurrency: 10
max_cpu: 500m
max_memory: 256Mi
min_concurrency: 1
min_cpu: 100m
min_memory: 128Mi
num_workers: 20
revision: null
links:
- destination:
- thumbnailer
source:
- uploads
triggers:
- name: uploads
vrn: vast:dataengine:triggers:docs-src-uploads
Use your pipeline's name, cluster VRN and namespace in the printf. Apply the file once, to check that the cluster accepts it and that nothing changes:
vastde pipelines update thumbnails --config @roundtrip.yaml
Pipeline updated successfully
Name: thumbnails
Tags: []
GUID: c7f63133-3438-4cae-af6a-91291041f1b6
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:25:11Z
Updated At: 2026-10-08T12:41:00Z
VRN: vast:dataengine:pipelines:thumbnails
Status: Ready
vastde pipelines get thumbnails --manifest -o yaml > manifest-after.yaml
diff manifest-before.yaml manifest-after.yaml && echo identical
identical
The manifest round-trips. But the output has no Description: line any more: the wrapped file had no description, so the update cleared it. Add description (and tags, if the pipeline has them) to the file before you apply it. The update also made a new revision:
vastde pipelines get thumbnails -o yaml | grep -E 'last_revision_number|status'
last_revision_number: 4
status: Ready
Then deploy, commit the file, and from then on change the pipeline only through it. You can delete the filled-in resources blocks from the file to keep it short (the cluster adds them back), and delete revision: null or replace it with a number to pin. The Web UI's YAML view can also download the whole file; see coming from the UI.
Retries and the dead-letter topic
When a handler raises, DataEngine retries the delivery, with a growing wait between attempts. When the retries run out, the event goes to your tenant's dead-letter topic. This pipeline links an element trigger (prefix fail/) to the faulty function from write a function, which always raises:
name: errors-demo
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
triggers:
- name: fail-uploads
vrn: vast:dataengine:triggers:docs-src-fail
function_deployments:
- name: faulty
function_vrn: vast:dataengine:functions:faulty
links:
- source: [fail-uploads]
destination: [faulty]
After one upload, fail/broken.txt:
vastde logs get errors-demo --since 30m --severity ERROR
2026-10-08 12:53:07.18 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:09.19 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:13.21 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:21.22 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:37.23 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:54:09.25 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:55:13.27 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:57:21.28 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 13:01:37.30 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 13:10:09.33 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
With no retry setting on the link, the same event was retried again and again, each wait about twice the last, for more than half an hour before it was dead-lettered. All the attempts are one trace, with status ERROR:
vastde traces list errors-demo --since 30m
Trace ID Status Pipeline Function Start Time End Time
--------------------------------------------------------------------------------------------------------------------------------------
05c803a68378573cc7828a9012492150 OK errors-demo faulty 2026-10-08 12:51:24.757 2026-10-08 12:51:24.759
000050150000000a2de0c9d069d74986 ERROR errors-demo faulty 2026-10-08 12:53:07.178 2026-10-08 13:10:09.331
(The OK trace is the instance's init.)
Set the retries
Set retries on the link, in config. Choose it from how long a failure can wait, and set it on every link: the unset behaviour isn't documented.
links:
- source: [fail-uploads]
destination: [faulty]
config:
retries: 2
vastde pipelines update errors-demo --config @errors-demo.yaml
Pipeline updated successfully
Name: errors-demo
Tags: []
GUID: bc606b00-b7a6-4f2a-944a-1fb6783dffc1
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:51:00Z
Updated At: 2026-10-08T13:29:14Z
VRN: vast:dataengine:pipelines:errors-demo
Status: Ready
vastde pipelines deploy errors-demo
Pipeline deployed successfully
After an upload of fail/broken-2.txt, the handler ran three times, the first attempt and two retries:
vastde logs get errors-demo --since 2m --scope user
2026-10-08 13:29:38.88 [faulty] [INFO] [user] faulty initialized
2026-10-08 13:30:12.09 [faulty] [INFO] [user] processing fail/broken-2.txt
2026-10-08 13:30:14.11 [faulty] [INFO] [user] processing fail/broken-2.txt
2026-10-08 13:30:18.12 [faulty] [INFO] [user] processing fail/broken-2.txt
retries is the number of retries after the first attempt. A link's other setting, events_order, is in links[].config.
Read the dead-letter topic
The dead-letter topic is a Kafka topic on your tenant's broker; your platform team chose its name when they enabled DataEngine (in the lab, default-dlt). Nothing consumes it unless you do. This script prints every record in a topic. It needs a Kafka client (pip install kafka-python-ng) and the broker's address in KAFKA_BOOTSTRAP, as <host>:<port> (the broker.url in vastde triggers get <trigger> -o yaml, for example):
# usage: python read_topic.py <topic> -- print every record in a topic, oldest first
import os
import sys
from kafka import KafkaConsumer
topic = sys.argv[1] if len(sys.argv) > 1 else "default-dlt"
consumer = KafkaConsumer(
topic,
bootstrap_servers=os.environ["KAFKA_BOOTSTRAP"],
auto_offset_reset="earliest",
enable_auto_commit=False,
consumer_timeout_ms=10000,
)
n = 0
for rec in consumer:
n += 1
print(f"--- partition={rec.partition} offset={rec.offset} timestamp={rec.timestamp}")
for k, v in rec.headers:
print(f" header {k}: {v.decode(errors='replace')}")
print(f" key: {rec.key.decode(errors='replace') if rec.key else None}")
print(f" value: {rec.value.decode(errors='replace') if rec.value else None}")
print(f"{n} record(s) in {topic}")
After the first event's retries ran out:
python read_topic.py default-dlt
--- partition=6 offset=0 timestamp=1791466033508
header ce_specversion: 1.0
header ce_id: 88051124535306
header ce_source: vastdata.com:docs-src-fail.a17df233-c00b-4d37-b2ae-3b62cdd23d97
header ce_type: vastdata.com:Element.ElementCreated
header content-type: application/json
header ce_subject: vast-broker-docs-kafka.default
header ce_time: 2026-10-08T12:53:06.474661Z
header ce_knativeerrorcode: 500
header ce_elementsourcetype: vast:s3
header ce_knativeerrordata: Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0
header ce_elementparenthandle: 13444190038277752475
header ce_elementpath: docs-src/fail/broken.txt
header ce_knativekafkapartition: 1
header ce_knativekafkaoffset: 1
header ce_knativeerrordest: http://faulty-nbdoisubyaaaa.dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91.svc.cluster.local
header ce_elementhandle: 11801945720009428452
key: None
value: {"Records": [{"eventVersion": "2.2", "eventSource": "vast:s3", "awsRegion": "<cluster name>", "eventTime": "2026-10-08T12:53:06.474661Z", "eventName": "ObjectCreated:Put", "userIdentity": {"principalId": "docs-owner"}, "requestParameters": {"sourceIPAddress": "15.0.0.1"}, "responseElements": {"x-amz-request-id": "0x50110002bd76", "x-amz-id-2": "0x50110002bd76"}, "s3": {"s3SchemaVersion": "1.0", "configurationId": "docs-src-fail.a17df233-c00b-4d37-b2ae-3b62cdd23d97", "bucket": {"name": "docs-src", "ownerIdentity": {"principalId": "docs-owner"}, "arn": "arn:aws:s3:::docs-src"}, "object": {"key": "fail/broken.txt", "size": 15, "eTag": "a60ecbab48ca83bc85122364c8ea5f38", "sequencer": "000200000000000f4242"} } } ] }
1 record(s) in default-dlt
A dead-lettered record is the original event, as a CloudEvent in Kafka binary form: its attributes are ce_* headers and its data is the value. Three headers say what went wrong: ce_knativeerrorcode (the HTTP status the function returned), ce_knativeerrordest (the function's address) and ce_knativeerrordata, the error message, base64-encoded:
echo Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0 | base64 -d; echo
cannot process fail/broken.txt
After the retries: 2 run, the second event joined it:
python read_topic.py default-dlt | grep -E '^---|ce_id|ce_elementpath|knativeerrorcode|record'
--- partition=6 offset=0 timestamp=1791466033508
header ce_id: 88051124535306
header ce_knativeerrorcode: 500
header ce_elementpath: docs-src/fail/broken.txt
--- partition=6 offset=1 timestamp=1791466218279
header ce_id: 87982405058573
header ce_knativeerrorcode: 500
header ce_elementpath: docs-src/fail/broken-2.txt
2 record(s) in default-dlt
The dead-letter topic is shared by every pipeline in the tenant, so filter on ce_source (the trigger) or ce_knativeerrordest (the function) when you alert on it. Use a consumer group of your own to track what you've handled. For consumers, see consuming in the VAST Event Broker section.
One source of truth
The VAST Web UI, the CLI and the REST API all write the same pipeline objects, and none of them checks a version before saving: the last save wins, section by section. So pick one place to change each pipeline:
- For a pipeline in Git, change it only by updating from the file, by hand or from CI, and treat the UI as read-only: use it to watch runs.
- If someone changes it in the UI anyway, export it (above), review the difference against Git, and commit or discard it before the next update.
See source of truth for why, and coming from the UI.
See also
- Pipeline YAML: every field, what the cluster fills in, and what a deploy requires.
- Concepts: pipelines and pipeline revisions.
- Architecture: what deploy does on the compute cluster.
- Publish functions: function revisions and pinning.
- Observability: logs, traces and metrics for a pipeline.
- KB: Building and Deploying a Pipeline on VAST DataEngine