--- title: "Troubleshooting VAST DataEngine" slug: "dev-dataengine-troubleshooting-index" description: "Troubleshoot pipeline errors, function crashes, and handler issues with detailed fixes and logs to ensure smooth deployment and operation." tags: ["DataEngine"] updated: 2026-10-30T18:19:06Z published: 2026-10-30T18:19:06Z canonical: "docs.vastdata.com/dev-dataengine-troubleshooting-index" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting VAST DataEngine *Applies to VAST Cluster 5.5.* Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the details. If a command printed an error, search the [error reference](/develop/docs/dev-dataengine-troubleshooting-errors) for it: it has every message on this page verbatim. For problems with the topics themselves, see [troubleshooting VAST Event Broker](/develop/docs/dev-event-broker-troubleshooting-index). ## Where to look first | What you want to know | Command | | --- | --- | | Is the pipeline running, and if not, why? | `vastde pipelines get <pipeline> -o yaml`: read `status` and `reason`. The default output shows `Status` but not the reason. | | What did my function log? | `vastde logs get <pipeline> --since 30m`. Add `--scope user` for your own records only, `--severity ERROR` for errors, `--function <deployment>` for one function. | | Did an event arrive, and did it fail? | `vastde traces list <pipeline> --since 30m`, then `vastde traces get <trace-id>` | | Which revision is live? | `vastde pipelines get <pipeline> --deployed -o yaml` | | Where did failed events go? | The tenant's dead-letter topic. See [retries and the dead-letter topic](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic). | ## Building functions | Symptom | Likely cause | Fix | | --- | --- | --- | | `functions build` fails with `Handler file validation failed` and `Missing required components` | `main.py` has no top-level `init` (or `handler`). The CLI requires both. | Add `def init(ctx): pass`. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#missing-required-components-async-def-initctx). | | `functions build` fails with `handler file not found` | `--target` doesn't point at the directory that holds `main.py` | Fix the path. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#handler-file-not-found). | | The build succeeds, but the pipeline later goes to `Error` with `ModuleNotFoundError` | The build doesn't import your code, so a missing package in `requirements.txt` only shows when the function starts | Add the package, rebuild. Run the image with `vastde functions localrun` first: an import error shows in seconds. See [below](/develop/docs/dev-dataengine-troubleshooting-index#the-pipeline-stays-inprogress-then-goes-to-error). | | The build fails or stalls without network access | The build pulls a base image from Docker Hub and Python packages from PyPI as it runs | Build where Docker Hub and PyPI (or mirrors of them) are reachable. | ## Registering functions and triggers | Symptom | Likely cause | Fix | | --- | --- | --- | | `functions create` fails with `Image is unreachable: … status_code: 404` | The repository or tag isn't in the registry: not pushed, a typo, or `--artifact-source` includes the registry address | Push with `functions build --push`; pass the repository path only, such as `docs/thumbnailer`. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#image-is-unreachable). | | `functions create` fails with `container registry with name '…' not found` | The registry isn't linked to your tenant under that name | `vastde container-registries list`. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#container-registry-with-name-not-found). | | `functions update` fails with `422` and `last_published_revision_number` | A `vastde` v5.5.0-sp2 bug on functions with a published revision | Update through the API. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#lastpublishedrevisionnumber-extra-inputs-are-not-permitted). | | `triggers create element` rejects `ObjectCreated:Put` | Triggers take `ObjectCreated:*`, `ObjectRemoved:*`, `ObjectTagging:Put`, `ObjectTagging:Delete` | See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#-event-list-invalid). | | `triggers create schedule` fails with `Invalid cron schedule` | The expression isn't 5-field cron or 6/7-field Quartz, or a field is out of range. The message says `expected 5` even though 6 and 7 fields work. | See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#invalid-cron-schedule-wrong-number-of-fields). | ## Creating and deploying pipelines | Symptom | Likely cause | Fix | | --- | --- | --- | | `pipeline creation failed: …` | The file has a mistake the API checks on create: unknown link names, an unknown function, a namespace outside the cluster, a reserved environment variable, a secret named twice | Fix the file; nothing was created. Each message is in [errors: pipelines](/develop/docs/dev-dataengine-troubleshooting-errors#pipelines). | | `pipelines create --deploy` prints `Pipeline created`, then `pipeline deployment failed` | The deploy checks failed: no trigger, no link, or a secret that doesn't exist | The pipeline exists, in `Draft`. Fix the file, then `pipelines update` and `pipelines deploy`, or delete it. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#hint-issue-with-trigger-or-topic-configuration). | | `create --deploy` fails with `Function with … already exists` or `Trigger with … already exists` on a second run | Functions and triggers are separate objects, created once | Reuse them. Only the pipeline file needs to change between runs. | | `Pipeline deployed successfully`, then `Status: InProgress` | Normal for up to a minute or so: the compute cluster is starting the function services | Wait, and check `vastde pipelines get` again before you send events. | | `InProgress` for several minutes, then `Error` | A function's container exits at start-up | See [below](/develop/docs/dev-dataengine-troubleshooting-index#the-pipeline-stays-inprogress-then-goes-to-error). | ### The pipeline stays InProgress, then goes to Error `vastde pipelines deploy` returns `Pipeline deployed successfully` as soon as the cluster accepts the deploy. If a function's container then crashes while it starts, for example because `main.py` raises at import, the pipeline shows `InProgress` with the reason `Waiting for pipeline to become ready...` for several minutes, and then changes to `Error`. While it's `InProgress`, `vastde logs get` has nothing for that function (`No logs found. Try increasing the start time or changing the scope.`): the process exits before it sends any logs. The traceback is in two places you can reach without access to the compute cluster: - **The pipeline's reason**, once the status is `Error`. It quotes the container's output: ``` $ vastde pipelines get errors-init -o yaml … reason: |- Subresource failed: Function 'badimport' deployment failed: RevisionFailed - Revision "badimport--00001" failed with message: Container failed with: … VAST DataEngine Runtime, Version: 0.2.0 … File "/workspace/main.py", line 1, in from PIL import Image # Pillow is not listed in requirements.txt ^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'PIL' . status: Error ``` - **A local run of the same image**, which fails at once with the same traceback, so you don't have to wait: ``` $ vastde functions localrun /docs/badimport --image-tag v1 --port 18082 … ModuleNotFoundError: No module named 'PIL' docker container failed: exit status 1 ``` Someone with `kubectl` access to the pipeline's namespace sees the function's pod in `CrashLoopBackOff`, with the same traceback in its logs. To fix it, correct the code or `requirements.txt`, rebuild and push a new tag, add a function revision with that tag, and deploy again. ## Events don't arrive The pipeline is `Ready`, you upload an object or wait for the schedule, and the function's logs show nothing. | Symptom | Likely cause | Fix | | --- | --- | --- | | Nothing for some objects, events for others | The trigger's name filters (`--name-prefix`, `--name-suffix`) don't match the key | `vastde triggers get <trigger> -o yaml` shows the filters. A prefix is matched against the object key, for example `photos/`, without the bucket. | | Nothing at all after a file change | You ran `pipelines update` but not `pipelines deploy`. Update saves a revision; the deployed one keeps running. | `vastde pipelines deploy <pipeline>`. | | The events arrive, but the logs don't show them | `vastde logs get` returns the **oldest** 100 records in the window by default, so on a busy pipeline the newest are cut off | Add `--order-by desc`, narrow `--since`, or raise `--limit`. | | A schedule trigger never fires | The expression is valid but its next time is far off (a Quartz year field, for example `0 0 12 ? * MON-FRI 2027`), or the cluster's management service is down: schedule triggers don't fire then | Check the expression. Element triggers aren't affected by a management-service outage. | | A schedule-triggered and an upload-triggered event look as if they came from different brokers | `event.broker` is `<view>` on schedule events and `vast-broker-<view>` on element events, on the same topic | Compare `event.topic` or `event.type` instead. See [events](/develop/docs/dev-dataengine-reference-events#schedule-events). | | An upload runs functions in two pipelines | Two pipelines' triggers watch the same bucket, with overlapping filters. Each delivers its own copy. | Give the triggers disjoint prefixes, or delete the pipeline you don't need. | | The function ran, but its logs show no handler output, only `malformed vast event` | It's a local test with `vastde functions invoke --generate-event`, whose sample event has no `elementpath` | Test with an event file shaped like a real one. See [test events locally](/develop/docs/dev-dataengine-reference-events#test-events-locally). | If the trace list shows the event with status `ERROR`, the event arrived and the handler failed: see the next section. ## Handler errors, retries and the dead-letter topic When your handler raises, the delivery fails and DataEngine retries it. What you see: - **Logs.** Each attempt logs your code's records again, and one runtime record at `ERROR` with the exception's message: ``` $ vastde logs get errors-demo --since 30m --severity ERROR 2026-10-08 12:53:07.18 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt 2026-10-08 12:53:09.19 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt 2026-10-08 12:53:13.21 [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt … ``` - **Traces.** One trace per event, with status `ERROR`, lasting from the first attempt to the last: `vastde traces list <pipeline> --status ERROR`. - **Metrics.** `dataengine.function.errors` appears in `vastde metrics names -p <pipeline>` once a handler has failed. - **The dead-letter topic.** When the retries run out, the original event is written to the tenant's dead-letter topic with the error message added, base64-encoded, in the `ce_knativeerrordata` header. See [the dead-lettered record](/develop/docs/dev-dataengine-concepts-architecture#the-dead-lettered-record). | Symptom | Likely cause | Fix | | --- | --- | --- | | The same failing event runs again and again, with longer and longer gaps, for a long time | The link has no `retries` setting, so a server default applies | Set `config.retries` on every link. With `retries: 2`, the handler runs three times, then the event is dead-lettered. See [pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml#linksconfig). | | Duplicate output after an error | A retry runs the whole handler again, including the work it did before it raised | Make handlers idempotent: derive output names from the input, use upserts. See [retries](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic). | | You want to skip an event, not retry it | Every exception fails the delivery | Catch it, log it, and return. Raise only for events that should be retried and dead-lettered. | | The dead-letter topic has records from other pipelines | It's shared by every pipeline in the tenant | Filter on `ce_source` (the trigger) or `ce_knativeerrordest` (the failing function). | | A handler that reads a secret fails with `KeyError` | The secret isn't attached to this function deployment or the pipeline, or (locally) `localrun --detach` dropped the secrets | Attach it in the pipeline YAML, or run `localrun --config` in the foreground. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#failed-to-send-cloudevent-status-500). | ## Changes don't take effect | Symptom | Likely cause | Fix | | --- | --- | --- | | A new function revision isn't running | Deploy resolves the revision. Unpinned deployments take the function's default or latest revision when you deploy, not when the revision is created; pinned ones (`revision: <n>`) always run *n*. | `vastde pipelines deploy <pipeline>`. Check `pipelines get --deployed -o yaml` for pins. See [pinning a revision](/develop/docs/dev-dataengine-reference-pipeline-yaml#pinning-a-revision). | | The pipeline's description vanished after an update | `pipelines update` clears a top-level `description` the file leaves out | Keep `description` in the file. See [updates and revisions](/develop/docs/dev-dataengine-reference-pipeline-yaml#updates-and-revisions). | | `ctx.function_name` isn't the name you gave the function | It's the deployed service name, for example `thumbnailer-nbdn4ypiyaaaa` | Don't use it as a stable identifier. See [Context](/develop/docs/dev-dataengine-reference-runtime-sdk#context). | | A reserved environment variable such as `VAST_ENDPOINT` can't be set | Names starting `VAST_`, `VASTPIPELINE_` or `K_`, and a few others, are reserved | Use another name. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#environment-variable-names-are-reserved-for-internal-use). | ## CLI quirks Behaviour of `vastde` v5.5.0-sp2 that can look like a problem with your setup: | What you see | What's going on | Workaround | | --- | --- | --- | | `vastde config init --vms-url … --tenant … --username … --password …` prints `Setting local tenant to <tenant>` and nothing else, and no config file appears | With no config file yet, `config init` with flags exits with status 1 and writes nothing | Run `vastde config init` with no flags first, then repeat your command with `--force`. See [errors](/develop/docs/dev-dataengine-troubleshooting-errors#cannot-read-config-file). | | `vastde setup-dataengine` says `Data engine setup completed successfully`, then other commands fail with `Tenant is not provisioned for Data Engine` | Provisioning carries on in the background for about a minute | Wait and retry. | | `vastde status` prints `release: not found` | It reports a Helm release from `vastde install`, not whether DataEngine is enabled | Use `vastde compute-clusters list` to check access instead. | | `vastde topics list` fails with `database-name is required` | v5.5.0-sp2 needs the broker's database | Add `--database-name <view bucket>`. | | `vastde traces get` shows a duration such as `Duration: 2.148µs` | The value is in milliseconds; the unit label is wrong | Read it as ms, or subtract the start and end times, which are in µs. | | IDs print as floats, for example `id: 7.513950817699758e+18` | The CLI converts the numeric ID | Use the `guid` or `vrn` fields instead. | | `Owner:` prints as `{4294967295 vid 0xc000a0a620 map[]}` after `triggers create` | Cosmetic | None needed. | | `vastde logs tail` first prints the pipeline's whole history | Without `--since`, it starts from the beginning | Pass `--since 5m`. | | Every log line appears twice under `functions localrun` | Locally, the container prints each record to stdout and through the log interceptor | None needed; in the cluster each appears once. | | `vastde metrics get` prints JSON | It's the command's only output format in this version | Pipe it through `jq`. | ## Before you ask for help Collect these for your platform team or VAST support: - `vastde version` and the cluster version. - `vastde pipelines get <pipeline> -o yaml`, which includes `status` and `reason`. - The pipeline file, and the function and trigger names it uses, without secrets. - The failing command and its full output, or the trace ID from `vastde traces list`. - For start-up failures: whether `vastde functions localrun` of the same image works. ## See also - [Error reference](/develop/docs/dev-dataengine-troubleshooting-errors) - [Retries and the dead-letter topic](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic) and [what deploy does](/develop/docs/dev-dataengine-concepts-architecture#what-deploy-does) - [Events](/develop/docs/dev-dataengine-reference-events) and the [Runtime SDK reference](/develop/docs/dev-dataengine-reference-runtime-sdk) - [Pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml) - [Setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites) - [Troubleshooting VAST Event Broker](/develop/docs/dev-event-broker-troubleshooting-index)