--- title: "DataEngine error reference" slug: "dev-dataengine-troubleshooting-errors" description: "Explore VASTDE error messages, their causes, and fixes for building, registering, and deploying functions in DataEngine. Troubleshoot effectively!" tags: ["DataEngine"] updated: 2026-10-28T02:14:59Z published: 2026-10-28T02:14:59Z canonical: "docs.vastdata.com/dev-dataengine-troubleshooting-errors" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # DataEngine error reference *Applies to VAST Cluster 5.5.* This page lists the error messages you can get from `vastde`, from the cluster and from the DataEngine runtime while you build, register, wire and deploy functions, what causes each one, and how to fix it. Search the page for the message you see. For symptoms with no message, such as a pipeline that never becomes `Ready` or events that never arrive, start at [troubleshooting](/develop/docs/dev-dataengine-troubleshooting-index). **How the messages were produced:** every message on this page was captured on VAST Cluster 5.5.0-SP6 with `vastde` v5.5.0-sp2, by running the command shown. They're quoted verbatim, except that `<placeholders>` replace values that vary: GUIDs, addresses, timestamps and your own names. ## Reading a DataEngine error Errors come from four places, and the prefix tells you which: | Prefix | From | What it means | | --- | --- | --- | | `invalid arguments:`, `failed to resolve …`, `Validation failed:`, `Handler file validation failed:` | `vastde` itself | The CLI rejected the command before it sent anything. Nothing changed. | | An HTTP status, such as `400 Bad Request:`, `409 Conflict:` or `422 Unprocessable Entity:` | The DataEngine API on the cluster | The request was rejected. Nothing changed. | | `pipeline creation failed:` | The API, on `pipelines create` | The pipeline wasn't created. | | `pipeline deployment failed:` | The API, on `pipelines deploy` or the deploy step of `pipelines create --deploy` | The pipeline exists, but this revision wasn't deployed. After `create --deploy`, the pipeline is in `Draft`. | Errors that happen after a deploy, in the compute cluster or in your code, don't fail a command. They appear in the pipeline's status `reason` (`vastde pipelines get <pipeline> -o yaml`), in the function's logs (`vastde logs get <pipeline>`) and in its traces (`vastde traces list <pipeline>`). ## CLI setup ### `Cannot read config file` ``` Cannot read config file: /.vast/config.toml Call 'vastde config init' to create a new config file ``` `vastde` has no configuration file yet. Every command prints this first, even `vastde config init --help`. The file is `~/.vast/config.toml`. **Fix:** run `vastde config init` with no flags, which creates the file, then set your values with `vastde config init --force` and the flags, or with `vastde config set`. Running `vastde config init` with flags when no file exists fails without an error: it prints this banner and `Setting local tenant to <tenant>`, exits with status 1, and writes nothing. See [CLI quirks](/develop/docs/dev-dataengine-troubleshooting-index#cli-quirks). ### `has no permissions to perform GET action on /api/latest/tenants/` ``` Setting up data engine provisioning... Failed to setup data engine: 403 Forbidden: Not authorized! (error: {"detail":" (
) has no permissions to perform GET action on /api/latest/tenants/","code":"permission_denied"}) ``` From `vastde setup-dataengine`. The tenant admin's VMS role has no permissions. **Fix:** a platform-team task. The role of the user running `setup-dataengine` needs permissions; see [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites) and the KB procedure linked there. ### `Tenant is not provisioned for Data Engine` ``` 403 Forbidden: Tenant is not provisioned for Data Engine. Complete data engine setup first. ``` DataEngine isn't enabled on your tenant yet. You also get this for about a minute after `vastde setup-dataengine` reports `Data engine setup completed successfully`, because provisioning continues in the background. **Fix:** if `setup-dataengine` has just run, wait a minute and try again. Otherwise ask your platform team to enable DataEngine on the tenant; see [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites). ### `database-name is required` ``` database-name is required ``` From `vastde topics list` with no flags. In `vastde` v5.5.0-sp2, `topics list` needs the broker's database. **Fix:** pass `--database-name <broker view's bucket>`, for example the Kafka view your triggers use. ### `accepts 1 arg(s), received 2` ``` accepts 1 arg(s), received 2 ``` From `vastde traces get <pipeline> <trace-id>`. `traces get` takes only the trace ID. **Fix:** `vastde traces get <trace-id>`. Get the ID from `vastde traces list <pipeline>`. ## Building functions ### `Missing required components: [async] def init(ctx)` ``` Detected language: python Handler file validation failed: /main.py Missing required components: - [async] def init(ctx) - Initialization function Please fix the handler file and try again. Handler file validation failed: /main.py Requirements: - [async] def init(ctx) - Initialization function ``` From `vastde functions build`. The CLI checks `main.py` for a top-level `init` and `handler` before it builds, and this file has no `init`. The check is the CLI's own: nothing is built or pushed. **Fix:** add an `init`, even an empty one: ``` def init(ctx): pass ``` ### `handler file not found` ``` Detected language: python Validation failed: handler file not found: /main.py handler file not found: /main.py ``` From `vastde functions build --target <dir>`. There's no `main.py` in `<dir>`, usually because the path is wrong. **Fix:** point `--target` at the directory `vastde functions init` created, the one that holds `main.py`. A build that succeeds can still produce an image that fails when it starts, for example when `main.py` imports a package that isn't in `requirements.txt`. The build doesn't import your code. See [ModuleNotFoundError](/develop/docs/dev-dataengine-troubleshooting-errors#modulenotfounderror-no-module-named). ## Functions ### `Image is unreachable` ``` Failed to create function: 400 Bad Request: Image is unreachable: http:///:, registry: , status_code: 404, error: Unknown error, checked_at: ``` From `vastde functions create` (and the same check on `update`). The cluster looked for the image in the linked registry and the registry answered 404: the repository (`--artifact-source`) or the tag (`--image-tag`) doesn't exist there. The same message appears for a wrong tag and for a wrong repository. **Fix:** - Check that the image was pushed: `vastde functions build … --push` prints `Build completed: <registry>/<repo>:<tag>`. - `--artifact-source` is the repository path inside the registry, without the registry address and without a leading slash, for example `docs/thumbnailer`. - The address in the message is the registry's address as it was linked. If you push to the registry by another address, check that both reach the same registry. ### `container registry with name '…' not found` ``` failed to resolve container registry 'no-such-registry': container registry with name 'no-such-registry' not found ``` From `vastde functions create --container-registry <name>`. No registry with that name is linked to your tenant. **Fix:** `vastde container-registries list` shows the linked registries. Linking one is a platform-team task; see [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites). ### `Name must be 2-40 characters` ``` Failed to create function: 422 Unprocessable Entity: {"name": "Name must be 2-40 characters: lowercase letters, numbers, and hyphens only."} ``` From `vastde functions create --name Ghost_Fn`. Function names are 2 to 40 characters of lowercase letters, digits and hyphens. **Fix:** rename the function, for example `ghost-fn`. ### `Function with (name=…) already exists!` ``` Failed to create function: 409 Conflict: Function with (name=thumbnailer,tenant_guid=) already exists! ``` A function with that name already exists in your tenant. **Fix:** to ship new code under the same function, add a revision with `vastde functions update` instead (but see the next entry). To start again, delete the old function first with `vastde functions delete <name>`. ### `last_published_revision_number: Extra inputs are not permitted` ``` Failed to update function: 422 Unprocessable Entity: {"detail":"{\"last_published_revision_number\": \"Extra inputs are not permitted\"}"} ``` From `vastde functions update` on a function that has a published revision, for example one created with `--publish`. It's a `vastde` v5.5.0-sp2 bug: the CLI copies `last_published_revision_number` from the function into the update request, and the API rejects it. `--dry-run` shows the field in the request body. Functions with no published revision update normally. **Fix:** until a fixed CLI ships, update the function through the DataEngine API with a body that leaves the field out. This `PUT` added revision 2 with the new tag and published it: ``` { "name": "thumbnailer", "description": "Makes thumbnails of new images", "artifact_type": "image", "artifact_source": "docs/thumbnailer", "image_tag": "v2", "container_registry_vrn": "vast:dataengine:container-registries:docs-local", "revision_description": "Return the thumbnail key", "is_published": true } ``` Send it to `PUT https://<VMS address>/api/latest/serverless/functions/<function GUID>`, with a bearer token from `POST https://<VMS address>/api/token/<tenant>`. `vastde functions get <name> -o json` shows the GUID. The API is documented in the [VAST DataEngine API reference](https://kb.vastdata.com/api-5-5/apidocs) on the KB. ## Triggers ### `--event list invalid` ``` invalid arguments: - --event list invalid: 'ObjectCreated:Put' not allowed. Available events: - ObjectCreated:* - ObjectRemoved:* - ObjectTagging:Put - ObjectTagging:Delete ``` From `vastde triggers create element --event`. Element triggers take only the four event types listed, with wildcards. `ObjectCreated:Put` is the name that appears inside a delivered event's `data`, not a trigger setting. **Fix:** use `--event "ObjectCreated:*"`. To act only on some kinds of create, check `event.get_data()["Records"][0]["eventName"]` in the handler. See [events](/develop/docs/dev-dataengine-reference-events#element-events). ### `events are required` ``` invalid arguments: - events are required ``` From `vastde triggers create element` with no `--event`. **Fix:** add at least one `--event`, for example `--event "ObjectCreated:*"`. ### `Failed to create trigger: Bucket not found` ``` 400 Bad Request: Failed to create trigger: Bucket not found ``` From `vastde triggers create element --source-bucket <bucket>`. The cluster has no bucket with that name in your tenant. **Fix:** check the bucket name: it's the view's bucket name, not its path. Creating the view is a platform-team task. ### `Invalid cron schedule: Wrong number of fields` ``` 400 Bad Request: Invalid cron schedule: Wrong number of fields; got 3, expected 5 ``` From `vastde triggers create schedule --cron-schedule "every 5 minutes"`. The schedule isn't a cron expression. The message says `expected 5` because it reports the standard-cron parser's error, but 6- and 7-field Quartz expressions are accepted too. **Fix:** use 5-field cron (`*/5 * * * *`) or Quartz (`0 0/5 * * * ?`). See [schedule triggers](/develop/docs/dev-dataengine-reference-function-and-trigger-files#schedule-trigger-file). ### `Invalid cron schedule: Error validating expression` ``` 400 Bad Request: Invalid cron schedule: Error validating expression '61': the last value (61) is higher than the maximum value (59) ``` From `--cron-schedule "61 * * * *"`. A field is out of range: minutes are 0 to 59. **Fix:** correct the field the message names. ### `Trigger with (name=…) already exists!` ``` 409 Conflict: Trigger with (name=docs-src-uploads) already exists! ``` A trigger with that name already exists in your tenant. Triggers are reusable: several pipelines can use the same one. **Fix:** use the existing trigger in your pipeline, change it with `vastde triggers update`, or pick another name. ## Pipelines ### `Links in manifest refer to unknown sources or destinations` ``` pipeline creation failed: Links in manifest refer to unknown sources or destinations: {'thumbnail'} ``` A link's `source` or `destination` names something that isn't in `manifest.triggers` or `manifest.function_deployments`. Links use the local `name` you give each entry in the file, not the trigger's or function's own name. Here the deployment is `thumbnailer` and the link says `thumbnail`. **Fix:** make every name in `links` match a `name` under `triggers` or `function_deployments`. See [pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml#manifestlinks). ### `Functions with VRN … not found` ``` pipeline creation failed: Functions with VRN {'vast:dataengine:functions:thumbnailr'} not found ``` A `function_vrn` names a function that doesn't exist in your tenant, here because of a typo. **Fix:** `vastde functions list` shows your functions. The VRN is `vast:dataengine:functions:<function name>`. ### `Namespace … is not in the Kubernetes cluster` ``` pipeline creation failed: Namespace default is not in the Kubernetes cluster test-cluster ``` `namespace` isn't one of your tenant's namespaces on that compute cluster. Examples that use `default` don't work on a cluster where your tenant has its own namespace. **Fix:** use your tenant's DataEngine namespace, `dataengine-ns-<tenant GUID>`. `vastde compute-clusters get <cluster> -o yaml` lists it under `namespaces`. ### `Kubernetes cluster vrn is required to validate namespace` ``` pipeline creation failed: Kubernetes cluster vrn is required to validate namespace ``` The file has no `kubernetes_cluster_vrn`. **Fix:** add `kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:<cluster name>`. `vastde compute-clusters list` shows the cluster names. ### `Hint: Issue with trigger or topic configuration` ``` pipeline deployment failed: Failed to deploy pipeline. Pipeline GUID: . Hint: Issue with trigger or topic configuration. Verify trigger and topic settings. ``` From the deploy step. Here the manifest had no `triggers` section: a pipeline needs at least one trigger, and a link from it, to deploy. The pipeline was created, and stays in `Draft`. **Fix:** add the trigger and a link from it, then `vastde pipelines update <pipeline> --config <file>` and `vastde pipelines deploy <pipeline>`. Or delete the draft with `vastde pipelines delete <pipeline>` and run `create --deploy` again. ### `Pipeline manifest must contain at least 1 link` ``` pipeline deployment failed: Failed to deploy pipeline. Pipeline GUID: . Hint: Unexpected error: BadRequest: 400: Pipeline manifest must contain at least 1 link in the 'links' section ``` From the deploy step. The manifest has triggers and function deployments but no `links`, so nothing connects them. The pipeline was created, and stays in `Draft`. **Fix:** add a link from the trigger to the function deployment, then update and deploy, as above. ## Secrets and environment variables ### `environment variable names are reserved for internal use` ``` pipeline creation failed: The following environment variable names are reserved for internal use and cannot be set by users: VAST_ENDPOINT. Reserved prefixes: VASTPIPELINE_, K_, VAST_. Reserved names: HOSTNAME, PERFORMANCE, PORT, VAST_RUNTIME_HANDLER_THREADS ``` An environment variable in the pipeline or a function deployment has a reserved name. The same message names `PORT` or `K_SINK` when you set those. It applies to the `VAST_ENDPOINT`, `VAST_ACCESS_KEY` and similar names. **Fix:** rename the variable, for example `DB_ENDPOINT`, and read that name in your code. See [environment variables](/develop/docs/dev-dataengine-reference-runtime-sdk#environment-variables). ### `Secret(s) already defined at pipeline level` ``` pipeline creation failed: Secret(s) already defined at pipeline level cannot be redefined at function deployment level. Duplicates found in: 'envcheck': thumbnail-api. Remove the duplicate from either the pipeline or the function deployment config. ``` The same secret is listed in `manifest.config.secrets` and in a function deployment's `config.secrets`. **Fix:** list it once. At the pipeline level, every function deployment gets it. ### `Hint: Required secrets are missing` ``` pipeline deployment failed: Failed to deploy pipeline. Pipeline: 'env-errors' (Kubernetes name: 'env-errors', vast-id: '', GUID: ). Hint: Required secrets are missing. Ensure all secrets referenced in the pipeline manifest exist in the target namespace. ``` From the deploy step. The manifest names a secret that doesn't exist in the pipeline's namespace. The create step doesn't check this, so after `create --deploy` the pipeline exists, in `Draft`. **Fix:** create the secret, for example with `vastde pipelines update <pipeline> --config <file> --secret-file <secret file>` or `pipelines create … --secret-file`, check the name matches, then deploy. See [secrets](/develop/docs/dev-dataengine-operations-security#secrets). ## Pipeline status These don't fail a command. Read them in the `reason` field of `vastde pipelines get <pipeline> -o yaml`. ### `Waiting for pipeline to become ready...` ``` reason: Waiting for pipeline to become ready... status: InProgress ``` The deploy was accepted and the compute cluster is still starting the pipeline: pulling images and starting the function services. It normally changes to `Ready` within a minute or so. If it stays for several minutes, a function is probably failing to start; the next entry is what you see when the cluster gives up. ### `Subresource failed: Function '…' deployment failed: RevisionFailed` ``` reason: |- Subresource failed: Function 'badimport' deployment failed: RevisionFailed - Revision "badimport--00001" failed with message: Container failed with: [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0 … Traceback (most recent call last): … File "/workspace/main.py", line 1, in from PIL import Image # Pillow is not listed in requirements.txt ^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'PIL' . status: Error ``` The function's container exited while the runtime was loading `main.py`, so the function never started. The `reason` carries the container's output, including the Python traceback; its last line is the exception. The pipeline shows `InProgress` for several minutes before it changes to `Error`. **Fix:** fix the exception the traceback names, rebuild, push, update the function and deploy again. For `ModuleNotFoundError`, see the next entry. ### `ModuleNotFoundError: No module named` ``` ModuleNotFoundError: No module named 'PIL' ``` `main.py` imports a package that isn't installed in the image. The build doesn't import your code, so it succeeds anyway. You see the traceback in the pipeline's `reason` once the pipeline goes to `Error`, and locally under `vastde functions localrun`, which fails straight away: ``` Traceback (most recent call last): … File "/workspace/main.py", line 1, in from PIL import Image # Pillow is not listed in requirements.txt ^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'PIL' docker container failed: exit status 1 ``` **Fix:** add the package to `requirements.txt` (here, `Pillow`), rebuild and redeploy. Run the image with `vastde functions localrun` before you create the function: an import error shows up in seconds. ### `No logs found` ``` No logs found. Try increasing the start time or changing the scope. ``` From `vastde logs get <pipeline>`. Nothing in the window matched. For a function that crashes at start-up, this is expected: the process exits before it sends any logs, so read the pipeline's `reason` instead. **Fix:** widen `--since`, remove `--scope` or `--severity`, or check the `reason`. ## Running functions ### `Error in handler execution` ``` [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt ``` In the function's logs (`vastde logs get <pipeline> --severity ERROR`). Your handler raised an exception; the text after the colon is its message. The runtime logs one of these per attempt: the event is retried, then sent to the dead-letter topic. **Fix:** fix the handler, or catch the exception if the event should be skipped rather than retried. See [handler errors and retries](/develop/docs/dev-dataengine-troubleshooting-index#handler-errors-retries-and-the-dead-letter-topic). ### `malformed vast event: object_key` ``` malformed vast event: object_key: 'NoneType' object has no attribute 'split' ``` Logged at `ERROR` when code reads `event.bucket` or `event.object_key` on an element event with no `elementpath` attribute. Events from a real element trigger always have one. The sample event that `vastde functions invoke --generate-event` sends doesn't. **Fix:** test locally with an event file shaped like a real element event. See [test events locally](/develop/docs/dev-dataengine-reference-events#test-events-locally). ### `Internal function event does not have a trigger` ``` vast.dataengine.sdk.event: subtype is not available for function event type vast.dataengine.sdk.event: Internal function event does not have a trigger, use function_trigger property instead vast.dataengine.sdk.event: Internal function event does not have a trigger_id, use function_trigger property instead ``` Warnings, in the `user` scope. Code read `event.subtype`, `event.trigger` or `event.trigger_id` on an event from an upstream function. Those properties belong to trigger events and return `None` here. **Fix:** use `event.function_trigger`, or check `event.type == "Function"` first. See [function events](/develop/docs/dev-dataengine-reference-events#function-events). ## Local runs ### `Failed to send CloudEvent (Status: 500)` ``` Sending CloudEvent to http://localhost:/ Event ID: Event Type: vastdata.com:Element.ElementCreated Failed to send CloudEvent (Status: 500) Error Response: 'thumbnail-api' HTTP request failed with status 500 ``` From `vastde functions invoke` against a function under `vastde functions localrun`. The handler raised; `Error Response` is the exception's message. Here it's a `KeyError` for the secret `thumbnail-api`, because the function was started with `localrun --config <file> --detach`: with `--detach`, the secrets in the config file don't reach the container, and `ctx.secrets` is empty. **Fix:** for secrets, run `localrun --config <file>` in the foreground and invoke from a second terminal. For other exceptions, read the container's output for the traceback. ## Logs, traces and metrics ### `All metrics must have the same type` ``` 422 Unprocessable Entity ({"metric_type": "All metrics must have the same type. Found types: Counter, Histogram"}) ``` From `vastde metrics get -m <a>,<b>` with a counter and a histogram in one query. **Fix:** query metrics of different types separately. `vastde metrics names -p <pipeline>` shows each metric's type. ## See also - [Troubleshooting](/develop/docs/dev-dataengine-troubleshooting-index): symptoms, including the ones with no error message - [Retries and the dead-letter topic](/develop/docs/dev-dataengine-concepts-architecture#retries-and-the-dead-letter-topic) - [Pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml) and [function and trigger files](/develop/docs/dev-dataengine-reference-function-and-trigger-files) - [Runtime SDK reference](/develop/docs/dev-dataengine-reference-runtime-sdk) - [Setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites): what your platform team sets up