Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Build and test locally

Prev Next

Applies to VAST Cluster 5.5.

This guide shows how to turn a function project into an image with vastde functions build, run that image on your machine with vastde functions localrun, and send it events with vastde functions invoke, so you find most mistakes before anything reaches the cluster. It ends with what a local run doesn't emulate.

You need Docker running and the vastde CLI configured; see your workstation. Commands use the environment variables from the quickstart. The outputs come from a lab machine with VAST Cluster 5.5.0-SP6 and vastde v5.5.0-sp2, where the registry host is 127.0.0.1:5000. Your paths, IDs and times will differ.

Build an image

vastde functions build builds an image from a function project with the builder image, and --push pushes it to your registry:

vastde functions build "$REGISTRY/docs/thumbnailer" --target ./thumbnailer --image-tag v1 --push

Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/docs/thumbnailer:v1
App Path: ./thumbnailer
Handlers File: main.py
2026/10/08 12:20:16 [Started] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1
2026/10/08 12:21:00 [Completed] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1
Build completed: 127.0.0.1:5000/docs/thumbnailer:v1
  • The image name is <registry host>/<path>; --image-tag sets the tag. Use a new tag for every build, so that each function revision points at exactly one image. See publish functions.
  • --target is the project directory. The handler file is main.py unless you pass -H (--handlers).
  • Python version. The build defaults to Python 3.12 and resolves it to a patch release, as above.
  • Network access. The build pulls a base image from Docker Hub and installs your requirements.txt from PyPI, so the machine needs access to both, or to mirrors.
  • --push uses your own Docker login for the registry (docker login), not DataEngine's registry settings.

The build checks the handler file before it builds anything. A file without init fails, and so does a --target with no handler file in it:

vastde functions build "$REGISTRY/docs/noinit" --target ./does-not-exist --image-tag v1

Detected language: python
Validation failed: handler file not found: does-not-exist/main.py
handler file not found: does-not-exist/main.py

The init case is in write a function.

The builder image

The builder is an image too. By default the CLI uses the builder whose tag matches its own version. vastde builders list shows the published tags:

vastde builders list

Fetching available builder images from Docker Hub...

REPOSITORY                    TAG                             SIZE          CREATED
--------------------------------------------------------------------------------------------
vastdataorg/vast-builder      latest                          700.9 MB      1 month ago
vastdataorg/vast-builder      v5.5.0-sp2                      700.9 MB      1 month ago
vastdataorg/vast-builder      v5.4.6-hf6                      700.0 MB      1 month ago
vastdataorg/vast-builder      v5.4.3-sp4-hf21                 700.0 MB      1 month ago
vastdataorg/vast-builder      v5.5.0                          700.0 MB      1 month ago
…

To set a builder image, use: vastde builders set <image-tag>

Use the tag your platform team names for your cluster, and set it with vastde builders set <tag>; vastde config view shows the one in use. Don't use latest or the dev and rc tags: pin a release tag so that builds are repeatable. vastde builders inspect <tag> shows a tag's digest and platform (linux/amd64).

Run it locally

vastde functions localrun runs the image in a local Docker container, with the DataEngine runtime listening on the port you choose. --detach runs it in the background:

vastde functions localrun "$REGISTRY/docs/thumbnailer" --image-tag v1 --port 18080 --detach

Running function '127.0.0.1:5000/docs/thumbnailer:v1' as Docker container on port 18080

Function is running in detached mode on port 18080!

  # Generate and send a test event:
  vastde functions invoke --generate-event --url http://localhost:18080/

  # Send a custom CloudEvent:
  vastde functions invoke --event cloudevent.yaml --url http://localhost:18080/

e8e703e11c8157e913a2c281868061455c236b107a599cf5b01c33ed7c016a1d
Function '127.0.0.1:5000/docs/thumbnailer' ran successfully.

The last hex string is the container ID; docker logs <container ID> shows the runtime's log, and docker stop <container ID> stops and removes the container. Without --detach, the runtime's log goes to your terminal and Ctrl+C stops it; run invoke from a second terminal.

If the image isn't on your machine, localrun pulls it from the registry first.

Send it an event

vastde functions invoke --event <file> sends the event in a YAML file to the local function. Write the file in the shape of a real event. This one is an element event for a new object, modelled on one a 5.5 cluster delivered:

# A test element event shaped like the ones an element trigger delivers
specversion: "1.0"
id: "87982405058561"
type: vastdata.com:Element.ElementCreated
source: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98
subject: vast-broker-docs-kafka.default
time: "2026-10-08T12:10:19.407648+00:00"
datacontenttype: application/json
elementpath: docs-src/photos/cat.jpg
elementsourcetype: vast:s3
data:
  Records:
    - eventVersion: "2.2"
      eventSource: vast:s3
      eventName: ObjectCreated:Put
      eventTime: "2026-10-08T12:10:19.407648Z"
      s3:
        bucket:
          name: docs-src
        object:
          key: photos/cat.jpg
          size: 24

elementpath (<bucket>/<key>) is what event.bucket and event.object_key are read from. Keys other than the standard CloudEvent fields (id, source, specversion, type, subject, time, data) become extension attributes, as on the cluster.

vastde functions invoke --event element-event.yaml --url http://localhost:18080/

Sending CloudEvent to http://localhost:18080/
Event ID: 87982405058561
Event Type: vastdata.com:Element.ElementCreated
CloudEvent sent successfully (Status: 204)

Status 204 means the handler returned without raising. If it raises, invoke prints Failed to send CloudEvent (Status: 500) and the exception's message. The handler's log lines are in the container's log:

docker logs <container ID>

…
2026-10-08 12:24:10 [INFO] [trace_id=-] START EventId: 87982405058561, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98, Timestamp: 1791462250.3285668
…
2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] new object: bucket=docs-src key=photos/cat.jpg
2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] Function returned: None in timestamp 2026-10-08 12:24:10.329816 with duration 0.49614906311035156 milliseconds
…
2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] No sink URL configured
2026-10-08 12:24:10 [INFO] [trace_id=-] END EventId: 87982405058561, TraceId: -, Duration: 0.0017s, Response: <starlette.responses.Response object at 0x7fceb8b53fe0>, Timestamp: 1791462250.330168
2026-10-08 12:24:10 [INFO] [trace_id=-] 172.17.0.1:54444 - "POST / HTTP/1.1" 204

Locally, docker logs prints most lines two or three times; on the cluster each appears once.

Don't rely on --generate-event. vastde functions invoke --generate-event sends a built-in sample event. It's useful to check that the function starts and answers, but it isn't shaped like a real event: it has no elementpath, so event.bucket and event.object_key are None, its type and subject don't match what the cluster sends, and event.get_data() returns the whole event. See test events locally. Keep event files like the one above in your project, next to the code, and use them as test fixtures.

Pass environment variables and secrets

--config gives the local run environment variables (envs) and secrets (secrets, by secret name and key), in place of what the pipeline would attach on the cluster:

envs:
  STAGE: local
  THUMBNAIL_SIZE: "64"
secrets:
  thumbnail-api:
    api-token: "<your API token>"

Keep this file out of Git. Run the function in the foreground with it:

vastde functions localrun "$REGISTRY/docs/envcheck" --image-tag v1 --port 18081 --config localrun-config.yaml

Running function '127.0.0.1:5000/docs/envcheck:v1' as Docker container on port 18081
Loaded configuration from: localrun-config.yaml

Function is starting on port 18081... Once ready, you can invoke it:
In another terminal, run:

  # Generate and send a test event:
  vastde functions invoke --generate-event --url http://localhost:18081/

  # Send a custom CloudEvent:
  vastde functions invoke --event cloudevent.yaml --url http://localhost:18081/

Press Ctrl+C to stop the function

2026-10-08 13:06:32 [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0
…
2026-10-08 13:06:32 [INFO] [trace_id=-] Handler is now ready to accept requests
2026-10-08 13:06:32 [INFO] [trace_id=-] Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2026-10-08 13:06:36 [INFO] [trace_id=-] START EventId: 87982405058561, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98, Timestamp: 1791464796.605323
2026-10-08 13:06:36 [INFO] [trace_id=-] Function invoked in timestamp 2026-10-08 13:06:36.605654. Event batching mode: False
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] Starting running function in timestamp 2026-10-08 13:06:36.606112
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env STAGE='local'
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env THUMBNAIL_SIZE='64'
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env LOG_PREFIX=None
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] secrets: {'thumbnail-api': ['api-token']}
2026-10-08 13:06:36 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] api-token is 48 characters long
…
Received signal interrupt, stopping container...
…
Container stopped gracefully

The event came from vastde functions invoke --event element-event.yaml --url http://localhost:18081/ in a second terminal, which printed CloudEvent sent successfully (Status: 204). The handler is the envcheck function from write a function.

Known issue in vastde v5.5.0-sp2: --detach drops the secrets. With --detach --config, the container starts with the environment variables but with no secrets, because the CLI removes its temporary copy of them when it exits. A handler that reads a secret then fails:

vastde functions localrun "$REGISTRY/docs/envcheck" --image-tag v1 --port 18081 --config localrun-config.yaml --detach

Running function '127.0.0.1:5000/docs/envcheck:v1' as Docker container on port 18081
Loaded configuration from: localrun-config.yaml

Function is running in detached mode on port 18081!

  # Generate and send a test event:
  vastde functions invoke --generate-event --url http://localhost:18081/

  # Send a custom CloudEvent:
  vastde functions invoke --event cloudevent.yaml --url http://localhost:18081/

732e93c808a5e5102a8c4e253e16c1201eef2c23cc3d1ee9e85dc5ca5fb901ea
Function '127.0.0.1:5000/docs/envcheck' ran successfully.

vastde functions invoke --event element-event.yaml --url http://localhost:18081/

Sending CloudEvent to http://localhost:18081/
Event ID: 87982405058561
Event Type: vastdata.com:Element.ElementCreated
Failed to send CloudEvent (Status: 500)
Error Response: 'thumbnail-api'
HTTP request failed with status 500

docker logs <container ID> 2>&1 | grep -E '^20.*(env |secrets|api-token)'

2026-10-08 13:05:19 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env STAGE='local'
2026-10-08 13:05:19 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env THUMBNAIL_SIZE='64'
2026-10-08 13:05:19 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] env LOG_PREFIX=None
2026-10-08 13:05:19 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] secrets: {}

So when you test with secrets, run localrun in the foreground.

Catch start-up errors

Errors when the handler file is imported (a missing package, a syntax error, a failing import at the top of the file) don't show up in the build. On the cluster, they leave the pipeline InProgress for minutes with nothing in vastde logs (see write a function). Locally you see them at once, because the container exits with the traceback:

vastde functions localrun "$REGISTRY/docs/badimport" --image-tag v1 --port 18082

Running function '127.0.0.1:5000/docs/badimport:v1' as Docker container on port 18082

Function is starting on port 18082... Once ready, you can invoke it:
In another terminal, run:

  # Generate and send a test event:
  vastde functions invoke --generate-event --url http://localhost:18082/

  # Send a custom CloudEvent:
  vastde functions invoke --event cloudevent.yaml --url http://localhost:18082/

Press Ctrl+C to stop the function

2026-10-08 13:20:56 [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0
2026-10-08 13:20:56 [INFO] [trace_id=-] Starting VAST DataEngine Runtime
2026-10-08 13:20:56 [INFO] [trace_id=-] App Path: /workspace
2026-10-08 13:20:56 [INFO] [trace_id=-] Handler: main.py
2026-10-08 13:20:56 [INFO] [trace_id=-] Loading handlers from /workspace
Container error output:
Unable to find image '127.0.0.1:5000/docs/badimport:v1' locally
…
Status: Downloaded newer image for 127.0.0.1:5000/docs/badimport:v1
Traceback (most recent call last):
…
  File "/workspace/main.py", line 1, in <module>
    from PIL import Image  # Pillow is not listed in requirements.txt
    ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'PIL'

docker container failed: exit status 1

Run every new image locally once, with at least one event file, before you create or update the function.

What a local run doesn't do

A local run is the same image and the same runtime as on the cluster, but nothing around it:

  • No downstream events. There's no link, so the return value goes nowhere: the runtime logs No sink URL configured. Test a chain by sending the upstream function's return value, as the data of a function event file, to the downstream function.
  • No triggers, retries or dead-letter topic. Events arrive only when you send them, once. A handler that raises returns a 500 to invoke, and that's all.
  • Logs stay local. They go to the container's output, not to DataEngine: vastde logs, vastde traces and metrics show nothing from a local run.
  • No cluster configuration. Environment variables and secrets come only from --config, not from the pipeline file. Resources, scaling and timeouts don't apply.
  • Event data may differ. invoke sends the event file in a way that can make event.get_data() return the whole event rather than only data. Read objects with event.bucket and event.object_key, which behave the same in both places, and check code that reads get_data() against a deployed pipeline.

See also

Related articles