Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Quickstart

Prev Next

Applies to VAST Cluster 5.5.

In this quickstart you'll write a Python function that runs every time an image is uploaded under photos/ in an S3 bucket. You'll scaffold it, test it on your machine with a realistic event, push it to your registry, register it with DataEngine, wire it to an element trigger in a pipeline file, deploy, upload an object and read the function's logs and traces. Everything is done with the vastde CLI and two small files. It takes about 15 minutes once your platform team has set up access.

Run the commands in order, in one shell, from an empty directory. Below each command is the output you should see.

Before you start

You need:

  • DataEngine enabled on your tenant, a compute cluster with a namespace for your tenant, and a container registry linked to that cluster and namespace. See setup prerequisites for the list to send your platform team.
  • A bucket (a view) set up as a source for element triggers, and an S3 access key pair that can write to it.
  • On your machine: Docker (running), the vastde CLI configured for your tenant, push access to the registry (docker login <registry host>), and the AWS CLI for the upload. See your workstation.

Set these for the rest of the page:

export REGISTRY="<registry host>"            # the host you push images to, for example registry.example.com
export REGISTRY_NAME="<registry name>"       # the registry's name in DataEngine (vastde container-registries list)
export BUCKET="<source bucket>"              # the bucket your element trigger watches
export S3_ENDPOINT="http://<S3 address>"     # an S3 endpoint of the cluster
export AWS_ACCESS_KEY_ID="<access key>"
export AWS_SECRET_ACCESS_KEY="<secret key>"

The outputs on this page come from a lab cluster running VAST Cluster 5.5.0-SP6. There, the registry host is 127.0.0.1:5000, the registry's name is docs-local, the bucket is docs-src, the compute cluster is test-cluster and the tenant's namespace is dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91. Your names, GUIDs, IDs, addresses and times will differ.

Check the CLI and its configuration:

vastde version

v5.5.0-sp2

vastde config view

Tenant Name: docs-de
Username: docs-admin
Password: ***
Builder Image URL: vastdataorg/vast-builder:v5.5.0-sp2
VMS URL: https://<VMS address>

Seven items, in order, then vastde connects. 1, platform team: enable DataEngine on your tenant and assign the event broker, a VAST Event Broker view or a third-party Kafka broker, with the tenant's default and dead-letter topics; a topic's partition count can't be changed later. 2, platform team: a compute cluster, a VAST compute cluster or an external Kubernetes cluster, linked to DataEngine with a namespace for your tenant; attaching your tenant to a VAST compute cluster gives it the dataengine-ns-tenant GUID namespace. 3, platform team: a container registry, served over HTTPS with a certificate from a valid CA, linked to the compute cluster and namespace your pipelines use. 4, platform team: a DataEngine user, a tenant admin or an application user with the data-engine-tenant identity policy or your own. 5, platform team: source views for element triggers; application users can't create them. 6, platform team or you: S3 key pairs, for your user and for the user your functions run with. 7, you: Docker, the vastde CLI, a builder image, and docker login to the registry. Then vastde connects: vastde config init with your tenant's VMS URL, tenant and user, and vastde compute-clusters list checks that DataEngine is enabled.

What your platform team sets up for DataEngine, in order, and what you install, before vastde connects. See setup prerequisites for each item.

1. Scaffold the function

Create a Python function project called thumbnailer:

vastde functions init python-pip thumbnailer

Creating pip scaffold for function: thumbnailer
Target folder: /home/centos/de-docs/qs
Handlers file: main.py
Python pip scaffold created successfully in /home/centos/de-docs/qs/thumbnailer
Navigate to /home/centos/de-docs/qs/thumbnailer and start implementing your function

ls thumbnailer

Aptfile
customDeps
main.py
README.md
requirements.txt

main.py is the handler file. requirements.txt, Aptfile and customDeps hold your Python packages, system packages and local libraries; this function needs none. The scaffold's handler looks like this:

cat thumbnailer/main.py


def init(ctx):
    # One time initialization comes here
    ctx.logger.info("Initialized...")

def handler(ctx, event):
    # Events Processing comes here
    ctx.logger.info(f"Handler {event}")
    return "Hello World"

init(ctx) runs once when an instance of the function starts. handler(ctx, event) runs once for every event.

2. Write the handler

Replace thumbnailer/main.py with this. For now it only logs the bucket and key of each new object:

def init(ctx):
    # One-time initialization: runs once when the function instance starts
    ctx.logger.info("thumbnailer initialized")

def handler(ctx, event):
    # An element trigger delivers one event per new object
    ctx.logger.info("new object: bucket=%s key=%s", event.bucket, event.object_key)

An element trigger delivers a CloudEvent, and the SDK gives you its bucket and object key as event.bucket and event.object_key. Keep both functions: vastde functions build refuses a handler file without init. See write a function for the rest of the handler contract.

3. Build and push the image

vastde functions build "$REGISTRY/docs/thumbnailer" --target ./thumbnailer --image-tag v1 --push

Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/docs/thumbnailer:v1
App Path: ./thumbnailer
Handlers File: main.py
2026/10/08 12:20:16 [Started] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1
2026/10/08 12:21:00 [Completed] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1
Build completed: 127.0.0.1:5000/docs/thumbnailer:v1

The image name is your registry host plus a path (docs/thumbnailer here; use any path you can push to). --push pushes it with your Docker credentials. The first build also pulls the builder image, so it takes a few minutes longer.

4. Test it locally

Run the image in a local container on port 18080:

vastde functions localrun "$REGISTRY/docs/thumbnailer" --image-tag v1 --port 18080 --detach

Running function '127.0.0.1:5000/docs/thumbnailer:v1' as Docker container on port 18080

Function is running in detached mode on port 18080!

  # Generate and send a test event:
  vastde functions invoke --generate-event --url http://localhost:18080/

  # Send a custom CloudEvent:
  vastde functions invoke --event cloudevent.yaml --url http://localhost:18080/

e8e703e11c8157e913a2c281868061455c236b107a599cf5b01c33ed7c016a1d
Function '127.0.0.1:5000/docs/thumbnailer' ran successfully.

The long hex string is the container's ID. Save this test event as element-event.yaml. It has the shape of the events an element trigger delivers (see events):

# A test element event shaped like the ones an element trigger delivers
specversion: "1.0"
id: "87982405058561"
type: vastdata.com:Element.ElementCreated
source: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98
subject: vast-broker-docs-kafka.default
time: "2026-10-08T12:10:19.407648+00:00"
datacontenttype: application/json
elementpath: docs-src/photos/cat.jpg
elementsourcetype: vast:s3
data:
  Records:
    - eventVersion: "2.2"
      eventSource: vast:s3
      eventName: ObjectCreated:Put
      eventTime: "2026-10-08T12:10:19.407648Z"
      s3:
        bucket:
          name: docs-src
        object:
          key: photos/cat.jpg
          size: 24

Send it to the local function:

vastde functions invoke --event element-event.yaml --url http://localhost:18080/

Sending CloudEvent to http://localhost:18080/
Event ID: 87982405058561
Event Type: vastdata.com:Element.ElementCreated
CloudEvent sent successfully (Status: 204)

Read the container's log, using the first 12 characters of the container ID that localrun printed. Locally, each line is printed two or three times; the lines that matter are these:

docker logs <container ID>

2026-10-08 12:24:08 [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0
2026-10-08 12:24:08 [INFO] [trace_id=-] Starting VAST DataEngine Runtime
…
2026-10-08 12:24:09 [INFO] [trace_id=-] Call Init Handler: init
thumbnailer initialized
2026-10-08 12:24:09 [INFO] [trace_id=-] thumbnailer initialized
…
2026-10-08 12:24:09 [INFO] [trace_id=-] Handler is now ready to accept requests
…
2026-10-08 12:24:10 [INFO] [trace_id=-] START EventId: 87982405058561, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98, Timestamp: 1791462250.3285668
…
2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] new object: bucket=docs-src key=photos/cat.jpg
…
2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] No sink URL configured
2026-10-08 12:24:10 [INFO] [trace_id=-] END EventId: 87982405058561, TraceId: -, Duration: 0.0017s, Response: <starlette.responses.Response object at 0x7fceb8b53fe0>, Timestamp: 1791462250.330168
2026-10-08 12:24:10 [INFO] [trace_id=-] 172.17.0.1:54444 - "POST / HTTP/1.1" 204

The handler read the bucket and key from the event. Stop the container:

docker stop <container ID>

e8e703e11c81

For more on local testing, and what it doesn't emulate, see build and test locally.

5. Register the function

DataEngine needs to know where the image is. Check your registry's name:

vastde container-registries list

Registry Name    Description                       GUID                  Updated at
---------------------------------------------------------------------------------------
docs-local       Local HTTP registry for the d...  1e286592-0291-46e...  2026-10-08 ...

Create the function. --artifact-source is the image's path in the registry, without the host and without the tag:

vastde functions create --name thumbnailer --description "Makes thumbnails of new images" --artifact-type image --artifact-source docs/thumbnailer --image-tag v1 --container-registry "$REGISTRY_NAME" --publish

Function created: thumbnailer
Name: thumbnailer
Description: Makes thumbnails of new images
Tags: []
GUID: 45f7eccb-c794-4fd0-a7df-7abe09386a26
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:24:33Z
Updated At: 2026-10-08T12:24:33Z
VRN: vast:dataengine:functions:thumbnailer
Last Revision: 1

The function is revision 1, pointing at the v1 image. Its VRN, vast:dataengine:functions:thumbnailer, is how a pipeline refers to it.

6. Create an element trigger

Create a trigger that fires for every new object whose key starts with photos/:

vastde triggers create element --name docs-src-uploads --description "New objects under photos/ in docs-src" --source-bucket "$BUCKET" --event "ObjectCreated:*" --name-prefix photos/

Trigger created: docs-src-uploads
Name: docs-src-uploads
Description: New objects under photos/ in docs-src
Tags: []
GUID: 11f2e798-f6c6-427f-b30f-0f20bdb8172a
Owner: {4294967295 vid 0xc0002da020 map[]}
CreatedAt: 2026-10-08T12:24:39Z
UpdatedAt: 2026-10-08T12:24:39Z
VRN: vast:dataengine:triggers:docs-src-uploads
Type: Element
Events:
  -: ObjectCreated:*
Topic: vast:dataengine:topics:docs-kafka/default
SourceBucket: docs-src

With no topic given, the trigger publishes its events to your tenant's default topic. See triggers for filters, other event types and schedules.

7. Write the pipeline file

A pipeline runs on one compute cluster, in one namespace. Find yours:

vastde compute-clusters list

Compute Cluster            Description      GUID                       Updated at
---------------------------------------------------------------------------------------
test-cluster               None             2a052533-ebb2-4b2e-8b0...  2026-10-08 12:03

vastde compute-clusters get test-cluster -o yaml

created_at: "2026-10-08T12:03:24.742Z"
guid: 2a052533-ebb2-4b2e-8b0d-718c3533d5dc
id: 7.51393205803339e+18
is_internal: true
kube_api_url: https://<compute cluster API address>:6443
mtls_credentials_guid: be58a611-249f-4078-9281-5109264d4d19
name: test-cluster
namespaces:
    - dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
owner:
    id: "4294967295"
    id_type: vid
    name: root
tenant_guid: da47bae0-aefd-4e85-a5a9-2020d6a2cb91
updated_at: "2026-10-08T12:03:24.742Z"
vrn: vast:dataengine:kubernetes-clusters:test-cluster

Use your cluster's name in the command. You need two values from it: vrn, and your tenant's namespace under namespaces (on a VAST compute cluster, dataengine-ns-<tenant GUID>).

Save this as pipeline.yaml, with your cluster's VRN and your namespace in place of the lab's:

name: thumbnails
description: Make a thumbnail for every new image in docs-src/photos/
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
  triggers:
    - name: uploads
      vrn: vast:dataengine:triggers:docs-src-uploads
  function_deployments:
    - name: thumbnailer
      function_vrn: vast:dataengine:functions:thumbnailer
  links:
    - source: [uploads]
      destination: [thumbnailer]

The manifest gives the trigger and the function local names (uploads, thumbnailer) and links them: every event from uploads goes to thumbnailer. The pipeline YAML reference lists every field.

8. Create and deploy the pipeline

vastde pipelines create --config @pipeline.yaml --deploy

Pipeline created: thumbnails
Name: thumbnails
Description: Make a thumbnail for every new image in docs-src/photos/
Tags: []
GUID: c7f63133-3438-4cae-af6a-91291041f1b6
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:25:11Z
Updated At: 2026-10-08T12:25:11Z
VRN: vast:dataengine:pipelines:thumbnails

Deploying pipeline: thumbnails
Pipeline deployed successfully

The command returns once the deploy is accepted. The compute cluster then pulls the image and starts the function, and the status goes from InProgress to Ready:

vastde pipelines get thumbnails

Name: thumbnails
Description: Make a thumbnail for every new image in docs-src/photos/
Tags: []
GUID: c7f63133-3438-4cae-af6a-91291041f1b6
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:25:11Z
Updated At: 2026-10-08T12:25:11Z
VRN: vast:dataengine:pipelines:thumbnails
Status: InProgress

Run it again until the status is Ready. In the lab that took about 30 seconds:

vastde pipelines get thumbnails

Name: thumbnails
Description: Make a thumbnail for every new image in docs-src/photos/
Tags: []
GUID: c7f63133-3438-4cae-af6a-91291041f1b6
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T12:25:11Z
Updated At: 2026-10-08T12:25:11Z
VRN: vast:dataengine:pipelines:thumbnails
Status: Ready

If it shows Error instead, vastde pipelines get thumbnails -o yaml shows the reason.

9. Upload an object

Upload any small file as photos/cat.jpg:

aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 cp cat.jpg "s3://$BUCKET/photos/cat.jpg"

Completed 132 Bytes/132 Bytes (1.7 KiB/s) with 1 file(s) remaining
upload: ./cat.jpg to s3://docs-src/photos/cat.jpg

10. Read the logs and traces

The function ran within a second of the upload. Its logs:

vastde logs get thumbnails --since 10m

2026-10-08 12:25:28.67 [thumbnailer] [INFO]  [vast-runtime] Print function intercepted and redirected to 'user-print' logger
2026-10-08 12:25:28.96 [thumbnailer] [INFO]  [vast-runtime] Runtime is listening on port 8080
2026-10-08 12:25:28.97 [thumbnailer] [INFO]  [vast-runtime] Sync handler detected, concurrency limited to 20 via middleware
2026-10-08 12:25:29.15 [thumbnailer] [INFO]  [vast-runtime] ConcurrencyLimitMiddleware initialized with max_concurrent=20
2026-10-08 12:25:29.15 [thumbnailer] [INFO]  [vast-runtime] Configured private thread pool executor with 20 workers
2026-10-08 12:25:29.83 [thumbnailer] [INFO]  [vast-runtime] Initialized HTTP client with HTTP/2 support for CloudEvent sending
2026-10-08 12:25:29.84 [thumbnailer] [INFO]  [vast-runtime] Call Init Handler: init
2026-10-08 12:25:29.84 [thumbnailer] [INFO]  [user] thumbnailer initialized
2026-10-08 12:25:29.84 [thumbnailer] [INFO]  [vast-runtime] Init Handler Completed, duration: 0:00:00.001268
2026-10-08 12:25:29.84 [thumbnailer] [INFO]  [vast-runtime] Handler is now ready to accept requests
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] START EventId: 88051124535297, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.11f2e798-f6c6-427f-b30f-0f20bdb8172a, Timestamp: 1791462418.9423726
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] Function invoked in timestamp 2026-10-08 12:26:58.942813. Event batching mode: False
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] Starting running function in timestamp 2026-10-08 12:26:58.944075
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [user] new object: bucket=docs-src key=photos/cat.jpg
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] Function returned: None in timestamp 2026-10-08 12:26:58.944853 with duration 0.5662441253662109 milliseconds
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] No sink URL configured
2026-10-08 12:26:58.94 [thumbnailer] [INFO]  [vast-runtime] END EventId: 88051124535297, TraceId: -, Duration: 0.0032s, Response: <starlette.responses.Response object at 0x7efc699747a0>, Timestamp: 1791462418.9454546

[user] lines come from your code and [vast-runtime] lines from the DataEngine runtime around it. Add --scope user to see only yours. The traces:

vastde traces list thumbnails --since 10m

Trace ID                          Status  Pipeline              Function              Start Time               End Time
--------------------------------------------------------------------------------------------------------------------------------------
8d3144dcbc26b05e21a4c6e59554e4fc  OK      thumbnails            thumbnailer           2026-10-08 12:25:29.840  2026-10-08 12:25:29.842
0000501500000001b49102791a8a492a  OK      thumbnails            thumbnailer           2026-10-08 12:26:58.944  2026-10-08 12:26:58.945

The first trace is the instance's init call and the second is the upload. vastde traces get <trace ID> shows one trace's event and function. See observability.

Clean up

The guides in this section build on this pipeline, function and trigger. If you're going on to them, keep everything. Otherwise, delete the pipeline first (it stops the function on the compute cluster), then the trigger and the function, then the test object:

vastde pipelines delete thumbnails
vastde triggers delete docs-src-uploads
vastde functions delete thumbnailer
aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 rm "s3://$BUCKET/photos/cat.jpg"

vastde pipelines delete prints Deleting pipeline: <GUID> and then Pipeline deleted successfully. The image stays in your registry.

Next steps

Related articles