Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Troubleshooting VAST DataEngine

Prev Next

Applies to VAST Cluster 5.5.

Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the details. If a command printed an error, search the error reference for it: it has every message on this page verbatim. For problems with the topics themselves, see troubleshooting VAST Event Broker.

Where to look first

What you want to know Command
Is the pipeline running, and if not, why? vastde pipelines get <pipeline> -o yaml: read status and reason. The default output shows Status but not the reason.
What did my function log? vastde logs get <pipeline> --since 30m. Add --scope user for your own records only, --severity ERROR for errors, --function <deployment> for one function.
Did an event arrive, and did it fail? vastde traces list <pipeline> --since 30m, then vastde traces get <trace-id>
Which revision is live? vastde pipelines get <pipeline> --deployed -o yaml
Where did failed events go? The tenant's dead-letter topic. See retries and the dead-letter topic.

Building functions

Symptom Likely cause Fix
functions build fails with Handler file validation failed and Missing required components main.py has no top-level init (or handler). The CLI requires both. Add def init(ctx): pass. See errors.
functions build fails with handler file not found --target doesn't point at the directory that holds main.py Fix the path. See errors.
The build succeeds, but the pipeline later goes to Error with ModuleNotFoundError The build doesn't import your code, so a missing package in requirements.txt only shows when the function starts Add the package, rebuild. Run the image with vastde functions localrun first: an import error shows in seconds. See below.
The build fails or stalls without network access The build pulls a base image from Docker Hub and Python packages from PyPI as it runs Build where Docker Hub and PyPI (or mirrors of them) are reachable.

Registering functions and triggers

Symptom Likely cause Fix
functions create fails with Image is unreachable: … status_code: 404 The repository or tag isn't in the registry: not pushed, a typo, or --artifact-source includes the registry address Push with functions build --push; pass the repository path only, such as docs/thumbnailer. See errors.
functions create fails with container registry with name '…' not found The registry isn't linked to your tenant under that name vastde container-registries list. See errors.
functions update fails with 422 and last_published_revision_number A vastde v5.5.0-sp2 bug on functions with a published revision Update through the API. See errors.
triggers create element rejects ObjectCreated:Put Triggers take ObjectCreated:*, ObjectRemoved:*, ObjectTagging:Put, ObjectTagging:Delete See errors.
triggers create schedule fails with Invalid cron schedule The expression isn't 5-field cron or 6/7-field Quartz, or a field is out of range. The message says expected 5 even though 6 and 7 fields work. See errors.

Creating and deploying pipelines

Symptom Likely cause Fix
pipeline creation failed: … The file has a mistake the API checks on create: unknown link names, an unknown function, a namespace outside the cluster, a reserved environment variable, a secret named twice Fix the file; nothing was created. Each message is in errors: pipelines.
pipelines create --deploy prints Pipeline created, then pipeline deployment failed The deploy checks failed: no trigger, no link, or a secret that doesn't exist The pipeline exists, in Draft. Fix the file, then pipelines update and pipelines deploy, or delete it. See errors.
create --deploy fails with Function with … already exists or Trigger with … already exists on a second run Functions and triggers are separate objects, created once Reuse them. Only the pipeline file needs to change between runs.
Pipeline deployed successfully, then Status: InProgress Normal for up to a minute or so: the compute cluster is starting the function services Wait, and check vastde pipelines get again before you send events.
InProgress for several minutes, then Error A function's container exits at start-up See below.

The pipeline stays InProgress, then goes to Error

vastde pipelines deploy returns Pipeline deployed successfully as soon as the cluster accepts the deploy. If a function's container then crashes while it starts, for example because main.py raises at import, the pipeline shows InProgress with the reason Waiting for pipeline to become ready... for several minutes, and then changes to Error.

While it's InProgress, vastde logs get has nothing for that function (No logs found. Try increasing the start time or changing the scope.): the process exits before it sends any logs. The traceback is in two places you can reach without access to the compute cluster:

  • The pipeline's reason, once the status is Error. It quotes the container's output:

    $ vastde pipelines get errors-init -o yaml
…
reason: |-
    Subresource failed: Function 'badimport' deployment failed: RevisionFailed - Revision "badimport-<id>-00001" failed with message: Container failed with: … VAST DataEngine Runtime, Version: 0.2.0
    …
      File "/workspace/main.py", line 1, in <module>
        from PIL import Image  # Pillow is not listed in requirements.txt
        ^^^^^^^^^^^^^^^^^^^^^
    ModuleNotFoundError: No module named 'PIL'
    .
status: Error

  • A local run of the same image, which fails at once with the same traceback, so you don't have to wait:

    $ vastde functions localrun <registry>/docs/badimport --image-tag v1 --port 18082
…
ModuleNotFoundError: No module named 'PIL'

docker container failed: exit status 1

Someone with kubectl access to the pipeline's namespace sees the function's pod in CrashLoopBackOff, with the same traceback in its logs.

To fix it, correct the code or requirements.txt, rebuild and push a new tag, add a function revision with that tag, and deploy again.

Events don't arrive

The pipeline is Ready, you upload an object or wait for the schedule, and the function's logs show nothing.

Symptom Likely cause Fix
Nothing for some objects, events for others The trigger's name filters (--name-prefix, --name-suffix) don't match the key vastde triggers get <trigger> -o yaml shows the filters. A prefix is matched against the object key, for example photos/, without the bucket.
Nothing at all after a file change You ran pipelines update but not pipelines deploy. Update saves a revision; the deployed one keeps running. vastde pipelines deploy <pipeline>.
The events arrive, but the logs don't show them vastde logs get returns the oldest 100 records in the window by default, so on a busy pipeline the newest are cut off Add --order-by desc, narrow --since, or raise --limit.
A schedule trigger never fires The expression is valid but its next time is far off (a Quartz year field, for example 0 0 12 ? * MON-FRI 2027), or the cluster's management service is down: schedule triggers don't fire then Check the expression. Element triggers aren't affected by a management-service outage.
A schedule-triggered and an upload-triggered event look as if they came from different brokers event.broker is <view> on schedule events and vast-broker-<view> on element events, on the same topic Compare event.topic or event.type instead. See events.
An upload runs functions in two pipelines Two pipelines' triggers watch the same bucket, with overlapping filters. Each delivers its own copy. Give the triggers disjoint prefixes, or delete the pipeline you don't need.
The function ran, but its logs show no handler output, only malformed vast event It's a local test with vastde functions invoke --generate-event, whose sample event has no elementpath Test with an event file shaped like a real one. See test events locally.

If the trace list shows the event with status ERROR, the event arrived and the handler failed: see the next section.

Handler errors, retries and the dead-letter topic

When your handler raises, the delivery fails and DataEngine retries it. What you see:

  • Logs. Each attempt logs your code's records again, and one runtime record at ERROR with the exception's message:

    $ vastde logs get errors-demo --since 30m --severity ERROR
2026-10-08 12:53:07.18 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:09.19 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
2026-10-08 12:53:13.21 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt
…

  • Traces. One trace per event, with status ERROR, lasting from the first attempt to the last: vastde traces list <pipeline> --status ERROR.

  • Metrics. dataengine.function.errors appears in vastde metrics names -p <pipeline> once a handler has failed.

  • The dead-letter topic. When the retries run out, the original event is written to the tenant's dead-letter topic with the error message added, base64-encoded, in the ce_knativeerrordata header. See the dead-lettered record.

Symptom Likely cause Fix
The same failing event runs again and again, with longer and longer gaps, for a long time The link has no retries setting, so a server default applies Set config.retries on every link. With retries: 2, the handler runs three times, then the event is dead-lettered. See pipeline YAML.
Duplicate output after an error A retry runs the whole handler again, including the work it did before it raised Make handlers idempotent: derive output names from the input, use upserts. See retries.
You want to skip an event, not retry it Every exception fails the delivery Catch it, log it, and return. Raise only for events that should be retried and dead-lettered.
The dead-letter topic has records from other pipelines It's shared by every pipeline in the tenant Filter on ce_source (the trigger) or ce_knativeerrordest (the failing function).
A handler that reads a secret fails with KeyError The secret isn't attached to this function deployment or the pipeline, or (locally) localrun --detach dropped the secrets Attach it in the pipeline YAML, or run localrun --config in the foreground. See errors.

Changes don't take effect

Symptom Likely cause Fix
A new function revision isn't running Deploy resolves the revision. Unpinned deployments take the function's default or latest revision when you deploy, not when the revision is created; pinned ones (revision: <n>) always run n. vastde pipelines deploy <pipeline>. Check pipelines get --deployed -o yaml for pins. See pinning a revision.
The pipeline's description vanished after an update pipelines update clears a top-level description the file leaves out Keep description in the file. See updates and revisions.
ctx.function_name isn't the name you gave the function It's the deployed service name, for example thumbnailer-nbdn4ypiyaaaa Don't use it as a stable identifier. See Context.
A reserved environment variable such as VAST_ENDPOINT can't be set Names starting VAST_, VASTPIPELINE_ or K_, and a few others, are reserved Use another name. See errors.

CLI quirks

Behaviour of vastde v5.5.0-sp2 that can look like a problem with your setup:

What you see What's going on Workaround
vastde config init --vms-url … --tenant … --username … --password … prints Setting local tenant to <tenant> and nothing else, and no config file appears With no config file yet, config init with flags exits with status 1 and writes nothing Run vastde config init with no flags first, then repeat your command with --force. See errors.
vastde setup-dataengine says Data engine setup completed successfully, then other commands fail with Tenant is not provisioned for Data Engine Provisioning carries on in the background for about a minute Wait and retry.
vastde status prints release: not found It reports a Helm release from vastde install, not whether DataEngine is enabled Use vastde compute-clusters list to check access instead.
vastde topics list fails with database-name is required v5.5.0-sp2 needs the broker's database Add --database-name <view bucket>.
vastde traces get shows a duration such as Duration: 2.148µs The value is in milliseconds; the unit label is wrong Read it as ms, or subtract the start and end times, which are in µs.
IDs print as floats, for example id: 7.513950817699758e+18 The CLI converts the numeric ID Use the guid or vrn fields instead.
Owner: prints as {4294967295 vid 0xc000a0a620 map[]} after triggers create Cosmetic None needed.
vastde logs tail first prints the pipeline's whole history Without --since, it starts from the beginning Pass --since 5m.
Every log line appears twice under functions localrun Locally, the container prints each record to stdout and through the log interceptor None needed; in the cluster each appears once.
vastde metrics get prints JSON It's the command's only output format in this version Pipe it through jq.

Before you ask for help

Collect these for your platform team or VAST support:

  • vastde version and the cluster version.
  • vastde pipelines get <pipeline> -o yaml, which includes status and reason.
  • The pipeline file, and the function and trigger names it uses, without secrets.
  • The failing command and its full output, or the trace ID from vastde traces list.
  • For start-up failures: whether vastde functions localrun of the same image works.

See also

Related articles