DataEngine error reference

Applies to VAST Cluster 5.5.

This page lists the error messages you can get from vastde, from the cluster and from the DataEngine runtime while you build, register, wire and deploy functions, what causes each one, and how to fix it. Search the page for the message you see. For symptoms with no message, such as a pipeline that never becomes Ready or events that never arrive, start at troubleshooting.

How the messages were produced: every message on this page was captured on VAST Cluster 5.5.0-SP6 with vastde v5.5.0-sp2, by running the command shown. They're quoted verbatim, except that <placeholders> replace values that vary: GUIDs, addresses, timestamps and your own names.

Reading a DataEngine error

Errors come from four places, and the prefix tells you which:

Errors that happen after a deploy, in the compute cluster or in your code, don't fail a command. They appear in the pipeline's status reason (vastde pipelines get <pipeline> -o yaml), in the function's logs (vastde logs get <pipeline>) and in its traces (vastde traces list <pipeline>).

CLI setup

Cannot read config file

Cannot read config file: <home>/.vast/config.toml
Call 'vastde config init' to create a new config file

vastde has no configuration file yet. Every command prints this first, even vastde config init --help. The file is ~/.vast/config.toml.

Fix: run vastde config init with no flags, which creates the file, then set your values with vastde config init --force and the flags, or with vastde config set. Running vastde config init with flags when no file exists fails without an error: it prints this banner and Setting local tenant to <tenant>, exits with status 1, and writes nothing. See CLI quirks.

has no permissions to perform GET action on /api/latest/tenants/

Setting up data engine provisioning...
Failed to setup data engine:
403 Forbidden: Not authorized! (error: {"detail":"<user> (<address>) has no permissions to perform GET action on /api/latest/tenants/","code":"permission_denied"})

From vastde setup-dataengine. The tenant admin's VMS role has no permissions.

Fix: a platform-team task. The role of the user running setup-dataengine needs permissions; see setup prerequisites and the KB procedure linked there.

Tenant is not provisioned for Data Engine

403 Forbidden: Tenant is not provisioned for Data Engine. Complete data engine setup first.

DataEngine isn't enabled on your tenant yet. You also get this for about a minute after vastde setup-dataengine reports Data engine setup completed successfully, because provisioning continues in the background.

Fix: if setup-dataengine has just run, wait a minute and try again. Otherwise ask your platform team to enable DataEngine on the tenant; see setup prerequisites.

database-name is required

database-name is required

From vastde topics list with no flags. In vastde v5.5.0-sp2, topics list needs the broker's database.

Fix: pass --database-name <broker view's bucket>, for example the Kafka view your triggers use.

accepts 1 arg(s), received 2

accepts 1 arg(s), received 2

From vastde traces get <pipeline> <trace-id>. traces get takes only the trace ID.

Fix: vastde traces get <trace-id>. Get the ID from vastde traces list <pipeline>.

Building functions

Missing required components: [async] def init(ctx)

Detected language: python
Handler file validation failed: <dir>/main.py
Missing required components:
  - [async] def init(ctx) - Initialization function
Please fix the handler file and try again.
Handler file validation failed: <dir>/main.py
Requirements:
  - [async] def init(ctx) - Initialization function

From vastde functions build. The CLI checks main.py for a top-level init and handler before it builds, and this file has no init. The check is the CLI's own: nothing is built or pushed.

Fix: add an init, even an empty one:

def init(ctx):
    pass

handler file not found

Detected language: python
Validation failed: handler file not found: <dir>/main.py
handler file not found: <dir>/main.py

From vastde functions build --target <dir>. There's no main.py in <dir>, usually because the path is wrong.

Fix: point --target at the directory vastde functions init created, the one that holds main.py.

A build that succeeds can still produce an image that fails when it starts, for example when main.py imports a package that isn't in requirements.txt. The build doesn't import your code. See ModuleNotFoundError.

Functions

Image is unreachable

Failed to create function:
400 Bad Request: Image is unreachable: http://<registry-address>/<artifact-source>:<tag>, registry: <registry-address>, status_code: 404, error: Unknown error, checked_at: <timestamp>

From vastde functions create (and the same check on update). The cluster looked for the image in the linked registry and the registry answered 404: the repository (--artifact-source) or the tag (--image-tag) doesn't exist there. The same message appears for a wrong tag and for a wrong repository.

Fix:

• Check that the image was pushed: vastde functions build … --push prints Build completed: <registry>/<repo>:<tag>.
• --artifact-source is the repository path inside the registry, without the registry address and without a leading slash, for example docs/thumbnailer.
• The address in the message is the registry's address as it was linked. If you push to the registry by another address, check that both reach the same registry.

container registry with name '…' not found

failed to resolve container registry 'no-such-registry': container registry with name 'no-such-registry' not found

From vastde functions create --container-registry <name>. No registry with that name is linked to your tenant.

Fix: vastde container-registries list shows the linked registries. Linking one is a platform-team task; see setup prerequisites.

Name must be 2-40 characters

Failed to create function:
422 Unprocessable Entity: {"name": "Name must be 2-40 characters: lowercase letters, numbers, and hyphens only."}

From vastde functions create --name Ghost_Fn. Function names are 2 to 40 characters of lowercase letters, digits and hyphens.

Fix: rename the function, for example ghost-fn.

Function with (name=…) already exists!

Failed to create function:
409 Conflict: Function with (name=thumbnailer,tenant_guid=<tenant-guid>) already exists!

A function with that name already exists in your tenant.

Fix: to ship new code under the same function, add a revision with vastde functions update instead (but see the next entry). To start again, delete the old function first with vastde functions delete <name>.

last_published_revision_number: Extra inputs are not permitted

Failed to update function:
422 Unprocessable Entity: {"detail":"{\"last_published_revision_number\": \"Extra inputs are not permitted\"}"}

From vastde functions update on a function that has a published revision, for example one created with --publish. It's a vastde v5.5.0-sp2 bug: the CLI copies last_published_revision_number from the function into the update request, and the API rejects it. --dry-run shows the field in the request body. Functions with no published revision update normally.

Fix: until a fixed CLI ships, update the function through the DataEngine API with a body that leaves the field out. This PUT added revision 2 with the new tag and published it:

{
  "name": "thumbnailer",
  "description": "Makes thumbnails of new images",
  "artifact_type": "image",
  "artifact_source": "docs/thumbnailer",
  "image_tag": "v2",
  "container_registry_vrn": "vast:dataengine:container-registries:docs-local",
  "revision_description": "Return the thumbnail key",
  "is_published": true
}

Send it to PUT https://<VMS address>/api/latest/serverless/functions/<function GUID>, with a bearer token from POST https://<VMS address>/api/token/<tenant>. vastde functions get <name> -o json shows the GUID. The API is documented in the VAST DataEngine API reference on the KB.

Triggers

--event list invalid

invalid arguments:
  - --event list invalid: 'ObjectCreated:Put' not allowed. Available events:
    - ObjectCreated:*
    - ObjectRemoved:*
    - ObjectTagging:Put
    - ObjectTagging:Delete

From vastde triggers create element --event. Element triggers take only the four event types listed, with wildcards. ObjectCreated:Put is the name that appears inside a delivered event's data, not a trigger setting.

Fix: use --event "ObjectCreated:*". To act only on some kinds of create, check event.get_data()["Records"][0]["eventName"] in the handler. See events.

events are required

invalid arguments:
  - events are required

From vastde triggers create element with no --event.

Fix: add at least one --event, for example --event "ObjectCreated:*".

Failed to create trigger: Bucket not found

400 Bad Request: Failed to create trigger: Bucket not found

From vastde triggers create element --source-bucket <bucket>. The cluster has no bucket with that name in your tenant.

Fix: check the bucket name: it's the view's bucket name, not its path. Creating the view is a platform-team task.

Invalid cron schedule: Wrong number of fields

400 Bad Request: Invalid cron schedule: Wrong number of fields; got 3, expected 5

From vastde triggers create schedule --cron-schedule "every 5 minutes". The schedule isn't a cron expression. The message says expected 5 because it reports the standard-cron parser's error, but 6- and 7-field Quartz expressions are accepted too.

Fix: use 5-field cron (*/5 * * * *) or Quartz (0 0/5 * * * ?). See schedule triggers.

Invalid cron schedule: Error validating expression

400 Bad Request: Invalid cron schedule: Error validating expression '61': the last value (61) is higher than the maximum value (59)

From --cron-schedule "61 * * * *". A field is out of range: minutes are 0 to 59.

Fix: correct the field the message names.

Trigger with (name=…) already exists!

409 Conflict: Trigger with (name=docs-src-uploads) already exists!

A trigger with that name already exists in your tenant. Triggers are reusable: several pipelines can use the same one.

Fix: use the existing trigger in your pipeline, change it with vastde triggers update, or pick another name.

Pipelines

Links in manifest refer to unknown sources or destinations

pipeline creation failed: Links in manifest refer to unknown sources or destinations: {'thumbnail'}

A link's source or destination names something that isn't in manifest.triggers or manifest.function_deployments. Links use the local name you give each entry in the file, not the trigger's or function's own name. Here the deployment is thumbnailer and the link says thumbnail.

Fix: make every name in links match a name under triggers or function_deployments. See pipeline YAML.

Functions with VRN … not found

pipeline creation failed: Functions with VRN {'vast:dataengine:functions:thumbnailr'} not found

A function_vrn names a function that doesn't exist in your tenant, here because of a typo.

Fix: vastde functions list shows your functions. The VRN is vast:dataengine:functions:<function name>.

Namespace … is not in the Kubernetes cluster

pipeline creation failed: Namespace default is not in the Kubernetes cluster test-cluster

namespace isn't one of your tenant's namespaces on that compute cluster. Examples that use default don't work on a cluster where your tenant has its own namespace.

Fix: use your tenant's DataEngine namespace, dataengine-ns-<tenant GUID>. vastde compute-clusters get <cluster> -o yaml lists it under namespaces.

Kubernetes cluster vrn is required to validate namespace

pipeline creation failed: Kubernetes cluster vrn is required to validate namespace

The file has no kubernetes_cluster_vrn.

Fix: add kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:<cluster name>. vastde compute-clusters list shows the cluster names.

Hint: Issue with trigger or topic configuration

pipeline deployment failed: Failed to deploy pipeline. Pipeline GUID: <pipeline-guid>. Hint: Issue with trigger or topic configuration. Verify trigger and topic settings.

From the deploy step. Here the manifest had no triggers section: a pipeline needs at least one trigger, and a link from it, to deploy. The pipeline was created, and stays in Draft.

Fix: add the trigger and a link from it, then vastde pipelines update <pipeline> --config <file> and vastde pipelines deploy <pipeline>. Or delete the draft with vastde pipelines delete <pipeline> and run create --deploy again.

Pipeline manifest must contain at least 1 link

pipeline deployment failed: Failed to deploy pipeline. Pipeline GUID: <pipeline-guid>. Hint: Unexpected error: BadRequest: 400: Pipeline manifest must contain at least 1 link in the 'links' section

From the deploy step. The manifest has triggers and function deployments but no links, so nothing connects them. The pipeline was created, and stays in Draft.

Fix: add a link from the trigger to the function deployment, then update and deploy, as above.

Secrets and environment variables

environment variable names are reserved for internal use

pipeline creation failed: The following environment variable names are reserved for internal use and cannot be set by users: VAST_ENDPOINT. Reserved prefixes: VASTPIPELINE_, K_, VAST_. Reserved names: HOSTNAME, PERFORMANCE, PORT, VAST_RUNTIME_HANDLER_THREADS

An environment variable in the pipeline or a function deployment has a reserved name. The same message names PORT or K_SINK when you set those. It applies to the VAST_ENDPOINT, VAST_ACCESS_KEY and similar names.

Fix: rename the variable, for example DB_ENDPOINT, and read that name in your code. See environment variables.

Secret(s) already defined at pipeline level

pipeline creation failed: Secret(s) already defined at pipeline level cannot be redefined at function deployment level. Duplicates found in: 'envcheck': thumbnail-api. Remove the duplicate from either the pipeline or the function deployment config.

The same secret is listed in manifest.config.secrets and in a function deployment's config.secrets.

Fix: list it once. At the pipeline level, every function deployment gets it.

Hint: Required secrets are missing

pipeline deployment failed: Failed to deploy pipeline. Pipeline: 'env-errors' (Kubernetes name: 'env-errors', vast-id: '<id>', GUID: <pipeline-guid>). Hint: Required secrets are missing. Ensure all secrets referenced in the pipeline manifest exist in the target namespace.

From the deploy step. The manifest names a secret that doesn't exist in the pipeline's namespace. The create step doesn't check this, so after create --deploy the pipeline exists, in Draft.

Fix: create the secret, for example with vastde pipelines update <pipeline> --config <file> --secret-file <secret file> or pipelines create … --secret-file, check the name matches, then deploy. See secrets.

Pipeline status

These don't fail a command. Read them in the reason field of vastde pipelines get <pipeline> -o yaml.

Waiting for pipeline to become ready...

reason: Waiting for pipeline to become ready...
status: InProgress

The deploy was accepted and the compute cluster is still starting the pipeline: pulling images and starting the function services. It normally changes to Ready within a minute or so. If it stays for several minutes, a function is probably failing to start; the next entry is what you see when the cluster gives up.

Subresource failed: Function '…' deployment failed: RevisionFailed

reason: |-
    Subresource failed: Function 'badimport' deployment failed: RevisionFailed - Revision "badimport-<id>-00001" failed with message: Container failed with: <timestamp> [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0
    …
    Traceback (most recent call last):
      …
      File "/workspace/main.py", line 1, in <module>
        from PIL import Image  # Pillow is not listed in requirements.txt
        ^^^^^^^^^^^^^^^^^^^^^
    ModuleNotFoundError: No module named 'PIL'
    .
status: Error

The function's container exited while the runtime was loading main.py, so the function never started. The reason carries the container's output, including the Python traceback; its last line is the exception. The pipeline shows InProgress for several minutes before it changes to Error.

Fix: fix the exception the traceback names, rebuild, push, update the function and deploy again. For ModuleNotFoundError, see the next entry.

ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'PIL'

main.py imports a package that isn't installed in the image. The build doesn't import your code, so it succeeds anyway. You see the traceback in the pipeline's reason once the pipeline goes to Error, and locally under vastde functions localrun, which fails straight away:

Traceback (most recent call last):
  …
  File "/workspace/main.py", line 1, in <module>
    from PIL import Image  # Pillow is not listed in requirements.txt
    ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'PIL'

docker container failed: exit status 1

Fix: add the package to requirements.txt (here, Pillow), rebuild and redeploy. Run the image with vastde functions localrun before you create the function: an import error shows up in seconds.

No logs found

No logs found. Try increasing the start time or changing the scope.

From vastde logs get <pipeline>. Nothing in the window matched. For a function that crashes at start-up, this is expected: the process exits before it sends any logs, so read the pipeline's reason instead.

Fix: widen --since, remove --scope or --severity, or check the reason.

Running functions

Error in handler execution

<timestamp> [faulty] [ERROR] [vast-runtime] Error in handler execution: cannot process fail/broken.txt

In the function's logs (vastde logs get <pipeline> --severity ERROR). Your handler raised an exception; the text after the colon is its message. The runtime logs one of these per attempt: the event is retried, then sent to the dead-letter topic.

Fix: fix the handler, or catch the exception if the event should be skipped rather than retried. See handler errors and retries.

malformed vast event: object_key

malformed vast event: object_key: 'NoneType' object has no attribute 'split'

Logged at ERROR when code reads event.bucket or event.object_key on an element event with no elementpath attribute. Events from a real element trigger always have one. The sample event that vastde functions invoke --generate-event sends doesn't.

Fix: test locally with an event file shaped like a real element event. See test events locally.

Internal function event does not have a trigger

vast.dataengine.sdk.event: subtype is not available for function event type
vast.dataengine.sdk.event: Internal function event does not have a trigger, use function_trigger property instead
vast.dataengine.sdk.event: Internal function event does not have a trigger_id, use function_trigger property instead

Warnings, in the user scope. Code read event.subtype, event.trigger or event.trigger_id on an event from an upstream function. Those properties belong to trigger events and return None here.

Fix: use event.function_trigger, or check event.type == "Function" first. See function events.

Local runs

Failed to send CloudEvent (Status: 500)

Sending CloudEvent to http://localhost:<port>/
Event ID: <event-id>
Event Type: vastdata.com:Element.ElementCreated
Failed to send CloudEvent (Status: 500)
Error Response: 'thumbnail-api'
HTTP request failed with status 500

From vastde functions invoke against a function under vastde functions localrun. The handler raised; Error Response is the exception's message. Here it's a KeyError for the secret thumbnail-api, because the function was started with localrun --config <file> --detach: with --detach, the secrets in the config file don't reach the container, and ctx.secrets is empty.

Fix: for secrets, run localrun --config <file> in the foreground and invoke from a second terminal. For other exceptions, read the container's output for the traceback.

Logs, traces and metrics

All metrics must have the same type

422 Unprocessable Entity  ({"metric_type": "All metrics must have the same type. Found types: Counter, Histogram"})

From vastde metrics get -m <a>,<b> with a counter and a histogram in one query.

Fix: query metrics of different types separately. vastde metrics names -p <pipeline> shows each metric's type.

See also

• Troubleshooting: symptoms, including the ones with no error message
• Retries and the dead-letter topic
• Pipeline YAML and function and trigger files
• Runtime SDK reference
• Setup prerequisites: what your platform team sets up