--- title: "VAST DataEngine" slug: "dev-dataengine" tags: ["DataEngine"] updated: 2026-10-30T18:19:15Z published: 2026-10-30T18:19:15Z canonical: "docs.vastdata.com/dev-dataengine" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # VAST DataEngine VAST DataEngine runs your Python functions on Kubernetes when something happens to your data on the cluster: an S3 object or an NFSv4 file is created or deleted in a view, an S3 object's tags change, or a schedule fires. Start with Setup prerequisites to get DataEngine from your platform team, then run the Quickstart to deploy a pipeline that runs on an S3 upload. Read Concepts and Architecture before you design a pipeline, then Pipelines as code to keep it in Git. If your team built pipelines in the Web UI, read Coming from the Web UI first. ## Overview and concepts Where to begin: the overview and quickstart, then the concepts and architecture the rest of the section relies on, and the release notes. - **[Overview](/develop/docs/dev-dataengine-overview)**: VAST DataEngine runs your Python functions on Kubernetes when data changes or a schedule fires, for developers building event-driven pipelines. - **[Quickstart](/develop/docs/dev-dataengine-quickstart)**: Write a Python function that runs on each image upload, test it locally, then deploy it with vastde and read its logs. - **[Concepts](/develop/docs/dev-dataengine-concepts-index)**: DataEngine's objects and how they relate, including which function revision a pipeline runs and what happens when UI and CLI edit it. - **[Architecture](/develop/docs/dev-dataengine-concepts-architecture)**: Follow one event from an S3 upload through your functions, and see what deploy, retries, scaling and telemetry do on the cluster. ## Guides Step-by-step instructions for building and running pipelines from the command line. Start with Write a function and Build and test locally; then Publish functions, Triggers and Pipelines as code take you to a deployed pipeline, and Observability shows what it's doing. - **[Write a function](/develop/docs/dev-dataengine-guides-write-a-function)**: Write a DataEngine handler in Python: init and handler, reading each event type, chaining with return values, secrets, environment variables and dependencies. - **[Build and test locally](/develop/docs/dev-dataengine-guides-build-and-test-locally)**: Build a function image with vastde functions build, run it locally with localrun and send it test events with invoke before deploying. - **[Publish functions](/develop/docs/dev-dataengine-guides-publish-functions)**: Register a function image, release new versions as revisions, and choose whether a pipeline follows the latest revision or stays pinned. - **[Triggers](/develop/docs/dev-dataengine-guides-triggers)**: Create element triggers for bucket changes and schedule triggers on a cron schedule, with the filters and syntaxes the cluster accepts. - **[Pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code)**: Keep a pipeline in a Git-tracked YAML file: create, update and deploy it, chain functions, route events, and set retries. - **[Observability](/develop/docs/dev-dataengine-guides-observability)**: Read and follow logs, inspect traces, and query built-in and custom metrics for a pipeline with vastde logs, traces and metrics. ## Integrations Patterns for connecting DataEngine to other systems with code you write, since none of them is built in. Start with VAST DataBase and Vector Search from a function if your functions write tables, and Consume the dead-letter topic before you go to production. - **[Integrations](/develop/docs/dev-dataengine-integrations-index)**: Patterns for connecting DataEngine to VAST DataBase, the dead-letter topic, OpenTelemetry, orchestrators and lineage tools, none of them built in. - **[VAST DataBase and Vector Search from a function](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search)**: Write VAST DataBase rows and embeddings from a function with the vastdb SDK, with S3 keys passed as a secret. - **[Consume the dead-letter topic](/develop/docs/dev-dataengine-integrations-dead-letter-topic)**: Read your tenant's dead-letter topic with a Kafka consumer, decode why each event failed, and alert on it or replay it. - **[Your own OpenTelemetry backend](/develop/docs/dev-dataengine-integrations-opentelemetry)**: Send a function's traces and logs to your own OpenTelemetry collector over OTLP, alongside DataEngine's telemetry and with the same trace ID. - **[Orchestrators (Airflow and similar)](/develop/docs/dev-dataengine-integrations-orchestrators)**: Start a pipeline from an Airflow or similar task by writing an object, wait for its run, and fail on errors. - **[OpenLineage and OpenMetadata](/develop/docs/dev-dataengine-integrations-openlineage-and-openmetadata)**: Emit OpenLineage run events from a function with openlineage-python, over HTTP or an Event Broker topic, for Marquez or OpenMetadata. ## Examples Complete applications built on DataEngine, published by VAST on GitHub. Read Reference applications to see how pipelines, VAST DataBase and Kubernetes services fit together in a real system. - **[Reference applications](/develop/docs/dev-dataengine-examples-reference-applications)**: Two complete VAST applications on GitHub, video search and a RAG assistant, that combine DataEngine pipelines with VAST DataBase and Kubernetes. ## Reference Every field, attribute and SDK call, for looking things up while you work. Use Pipeline YAML and Runtime SDK reference most; Events and Function and trigger files cover the rest, and Coming from the Web UI maps UI forms to code. - **[Pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml)**: Every field of the pipeline file, what the server fills in, what can't change after creation, and what a deploy requires. - **[Function and trigger files](/develop/docs/dev-dataengine-reference-function-and-trigger-files)**: Every field of the YAML files that create functions, element triggers and schedule triggers, and how updates and revisions behave. - **[Runtime SDK reference](/develop/docs/dev-dataengine-reference-runtime-sdk)**: The vast.dataengine.sdk Python API: the init and handler contract, Context, event classes, metrics, tracing, secrets, logging and errors. - **[Events](/develop/docs/dev-dataengine-reference-events)**: The CloudEvents attributes each element, schedule, function and manual event carries, and the Runtime SDK property that reads each one. - **[Coming from the Web UI](/develop/docs/dev-dataengine-reference-coming-from-the-ui)**: Map Web UI forms and Visual Builder pipelines to vastde flags and YAML keys, and move a UI-built pipeline into Git. ## Operations What your platform team sets up for you, how access and credentials work, and how to release from CI. Start with Setup prerequisites, read Security before you pass credentials to functions, and use CI/CD and automation when you automate releases. - **[Setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites)**: What to ask your platform team for before you deploy a pipeline, a request to copy, and what to install locally. - **[Security](/develop/docs/dev-dataengine-operations-security)**: Who can do what in DataEngine by user type and identity policy, and how to handle secrets, registry credentials and S3 keys. - **[CI/CD and automation](/develop/docs/dev-dataengine-operations-ci-cd)**: Release functions and pipelines from Git with a vastde deploy script in GitHub Actions or GitLab CI, or with the DataEngine API. ## Troubleshooting Where to look when a pipeline misbehaves. Start with Troubleshooting VAST DataEngine for symptoms without a message, and search the DataEngine error reference when a command prints an error. - **[Troubleshooting VAST DataEngine](/develop/docs/dev-dataengine-troubleshooting-index)**: Diagnose pipelines stuck in InProgress, events that don't arrive, handler failures and changes that don't take effect, by symptom. - **[DataEngine error reference](/develop/docs/dev-dataengine-troubleshooting-errors)**: Error messages from vastde, the cluster and the runtime while you build, register and deploy, with the cause and fix of each.