--- title: "Quickstart" slug: "dev-dataengine-quickstart" description: "Learn to create a Python function for S3 image uploads, test locally, and deploy with DataEngine using the vastde CLI in just 15 minutes." tags: ["DataEngine"] updated: 2026-10-30T18:19:01Z published: 2026-10-30T18:19:01Z canonical: "docs.vastdata.com/dev-dataengine-quickstart" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Quickstart *Applies to VAST Cluster 5.5.* In this quickstart you'll write a Python function that runs every time an image is uploaded under `photos/` in an S3 bucket. You'll scaffold it, test it on your machine with a realistic event, push it to your registry, register it with DataEngine, wire it to an element trigger in a pipeline file, deploy, upload an object and read the function's logs and traces. Everything is done with the `vastde` CLI and two small files. It takes about 15 minutes once your platform team has set up access. Run the commands in order, in one shell, from an empty directory. Below each command is the output you should see. ## Before you start You need: - DataEngine enabled on your tenant, a compute cluster with a namespace for your tenant, and a container registry linked to that cluster and namespace. See [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites) for the list to send your platform team. - A bucket (a view) set up as a source for element triggers, and an S3 access key pair that can write to it. - On your machine: Docker (running), the `vastde` CLI configured for your tenant, push access to the registry (`docker login <registry host>`), and the AWS CLI for the upload. See [your workstation](/develop/docs/dev-dataengine-operations-setup-prerequisites#your-workstation). Set these for the rest of the page: ``` export REGISTRY="" # the host you push images to, for example registry.example.com export REGISTRY_NAME="" # the registry's name in DataEngine (vastde container-registries list) export BUCKET="" # the bucket your element trigger watches export S3_ENDPOINT="http://" # an S3 endpoint of the cluster export AWS_ACCESS_KEY_ID="" export AWS_SECRET_ACCESS_KEY="" ``` The outputs on this page come from a lab cluster running VAST Cluster 5.5.0-SP6. There, the registry host is `127.0.0.1:5000`, the registry's name is `docs-local`, the bucket is `docs-src`, the compute cluster is `test-cluster` and the tenant's namespace is `dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91`. Your names, GUIDs, IDs, addresses and times will differ. Check the CLI and its configuration: ``` vastde version ``` ``` v5.5.0-sp2 ``` ``` vastde config view ``` ``` Tenant Name: docs-de Username: docs-admin Password: *** Builder Image URL: vastdataorg/vast-builder:v5.5.0-sp2 VMS URL: https:// ``` ![Seven items, in order, then vastde connects. 1, platform team: enable DataEngine on your tenant and assign the event broker, a VAST Event Broker view or a third-party Kafka broker, with the tenant's default and dead-letter topics; a topic's partition count can't be changed later. 2, platform team: a compute cluster, a VAST compute cluster or an external Kubernetes cluster, linked to DataEngine with a namespace for your tenant; attaching your tenant to a VAST compute cluster gives it the dataengine-ns-tenant GUID namespace. 3, platform team: a container registry, served over HTTPS with a certificate from a valid CA, linked to the compute cluster and namespace your pipelines use. 4, platform team: a DataEngine user, a tenant admin or an application user with the data-engine-tenant identity policy or your own. 5, platform team: source views for element triggers; application users can't create them. 6, platform team or you: S3 key pairs, for your user and for the user your functions run with. 7, you: Docker, the vastde CLI, a builder image, and docker login to the registry. Then vastde connects: vastde config init with your tenant's VMS URL, tenant and user, and vastde compute-clusters list checks that DataEngine is enabled.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/de-setup-flow.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A54%3A35Z&se=2026-11-08T09%3A54%3A35Z&sr=c&sp=r&sig=JEu3vq8TckmrxqmLhZDccCdNTHCVDrCKmI0E8%2B46kS8%3D) *What your platform team sets up for DataEngine, in order, and what you install, before vastde connects.* See [setup prerequisites](/develop/docs/dev-dataengine-operations-setup-prerequisites) for each item. ## 1. Scaffold the function Create a Python function project called `thumbnailer`: ``` vastde functions init python-pip thumbnailer ``` ``` Creating pip scaffold for function: thumbnailer Target folder: /home/centos/de-docs/qs Handlers file: main.py Python pip scaffold created successfully in /home/centos/de-docs/qs/thumbnailer Navigate to /home/centos/de-docs/qs/thumbnailer and start implementing your function ``` ``` ls thumbnailer ``` ``` Aptfile customDeps main.py README.md requirements.txt ``` `main.py` is the handler file. `requirements.txt`, `Aptfile` and `customDeps` hold your Python packages, system packages and local libraries; this function needs none. The scaffold's handler looks like this: ``` cat thumbnailer/main.py ``` ``` def init(ctx): # One time initialization comes here ctx.logger.info("Initialized...") def handler(ctx, event): # Events Processing comes here ctx.logger.info(f"Handler {event}") return "Hello World" ``` `init(ctx)` runs once when an instance of the function starts. `handler(ctx, event)` runs once for every event. ## 2. Write the handler Replace `thumbnailer/main.py` with this. For now it only logs the bucket and key of each new object: ``` def init(ctx): # One-time initialization: runs once when the function instance starts ctx.logger.info("thumbnailer initialized") def handler(ctx, event): # An element trigger delivers one event per new object ctx.logger.info("new object: bucket=%s key=%s", event.bucket, event.object_key) ``` An element trigger delivers a CloudEvent, and the SDK gives you its bucket and object key as `event.bucket` and `event.object_key`. Keep both functions: `vastde functions build` refuses a handler file without `init`. See [write a function](/develop/docs/dev-dataengine-guides-write-a-function) for the rest of the handler contract. ## 3. Build and push the image ``` vastde functions build "$REGISTRY/docs/thumbnailer" --target ./thumbnailer --image-tag v1 --push ``` ``` Detected language: python Validating Python version 3.12.*... Python version 3.12.* resolved to 3.12.13 Building 127.0.0.1:5000/docs/thumbnailer:v1 App Path: ./thumbnailer Handlers File: main.py 2026/10/08 12:20:16 [Started] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1 2026/10/08 12:21:00 [Completed] Python Builder: 127.0.0.1:5000/docs/thumbnailer:v1 Build completed: 127.0.0.1:5000/docs/thumbnailer:v1 ``` The image name is your registry host plus a path (`docs/thumbnailer` here; use any path you can push to). `--push` pushes it with your Docker credentials. The first build also pulls the builder image, so it takes a few minutes longer. ## 4. Test it locally Run the image in a local container on port 18080: ``` vastde functions localrun "$REGISTRY/docs/thumbnailer" --image-tag v1 --port 18080 --detach ``` ``` Running function '127.0.0.1:5000/docs/thumbnailer:v1' as Docker container on port 18080 Function is running in detached mode on port 18080! # Generate and send a test event: vastde functions invoke --generate-event --url http://localhost:18080/ # Send a custom CloudEvent: vastde functions invoke --event cloudevent.yaml --url http://localhost:18080/ e8e703e11c8157e913a2c281868061455c236b107a599cf5b01c33ed7c016a1d Function '127.0.0.1:5000/docs/thumbnailer' ran successfully. ``` The long hex string is the container's ID. Save this test event as `element-event.yaml`. It has the shape of the events an element trigger delivers (see [events](/develop/docs/dev-dataengine-reference-events#element-events)): ``` # A test element event shaped like the ones an element trigger delivers specversion: "1.0" id: "87982405058561" type: vastdata.com:Element.ElementCreated source: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98 subject: vast-broker-docs-kafka.default time: "2026-10-08T12:10:19.407648+00:00" datacontenttype: application/json elementpath: docs-src/photos/cat.jpg elementsourcetype: vast:s3 data: Records: - eventVersion: "2.2" eventSource: vast:s3 eventName: ObjectCreated:Put eventTime: "2026-10-08T12:10:19.407648Z" s3: bucket: name: docs-src object: key: photos/cat.jpg size: 24 ``` Send it to the local function: ``` vastde functions invoke --event element-event.yaml --url http://localhost:18080/ ``` ``` Sending CloudEvent to http://localhost:18080/ Event ID: 87982405058561 Event Type: vastdata.com:Element.ElementCreated CloudEvent sent successfully (Status: 204) ``` Read the container's log, using the first 12 characters of the container ID that `localrun` printed. Locally, each line is printed two or three times; the lines that matter are these: ``` docker logs ``` ``` 2026-10-08 12:24:08 [INFO] [trace_id=-] VAST DataEngine Runtime, Version: 0.2.0 2026-10-08 12:24:08 [INFO] [trace_id=-] Starting VAST DataEngine Runtime … 2026-10-08 12:24:09 [INFO] [trace_id=-] Call Init Handler: init thumbnailer initialized 2026-10-08 12:24:09 [INFO] [trace_id=-] thumbnailer initialized … 2026-10-08 12:24:09 [INFO] [trace_id=-] Handler is now ready to accept requests … 2026-10-08 12:24:10 [INFO] [trace_id=-] START EventId: 87982405058561, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.7d5e6227-d1bf-4bf9-90c6-084333be5a98, Timestamp: 1791462250.3285668 … 2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] new object: bucket=docs-src key=photos/cat.jpg … 2026-10-08 12:24:10 [INFO] [trace_id=00005005-0000-0001-0000-000000000000] No sink URL configured 2026-10-08 12:24:10 [INFO] [trace_id=-] END EventId: 87982405058561, TraceId: -, Duration: 0.0017s, Response: , Timestamp: 1791462250.330168 2026-10-08 12:24:10 [INFO] [trace_id=-] 172.17.0.1:54444 - "POST / HTTP/1.1" 204 ``` The handler read the bucket and key from the event. Stop the container: ``` docker stop ``` ``` e8e703e11c81 ``` For more on local testing, and what it doesn't emulate, see [build and test locally](/develop/docs/dev-dataengine-guides-build-and-test-locally). ## 5. Register the function DataEngine needs to know where the image is. Check your registry's name: ``` vastde container-registries list ``` ``` Registry Name Description GUID Updated at --------------------------------------------------------------------------------------- docs-local Local HTTP registry for the d... 1e286592-0291-46e... 2026-10-08 ... ``` Create the function. `--artifact-source` is the image's path in the registry, without the host and without the tag: ``` vastde functions create --name thumbnailer --description "Makes thumbnails of new images" --artifact-type image --artifact-source docs/thumbnailer --image-tag v1 --container-registry "$REGISTRY_NAME" --publish ``` ``` Function created: thumbnailer Name: thumbnailer Description: Makes thumbnails of new images Tags: [] GUID: 45f7eccb-c794-4fd0-a7df-7abe09386a26 Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T12:24:33Z Updated At: 2026-10-08T12:24:33Z VRN: vast:dataengine:functions:thumbnailer Last Revision: 1 ``` The function is revision 1, pointing at the `v1` image. Its VRN, `vast:dataengine:functions:thumbnailer`, is how a pipeline refers to it. ## 6. Create an element trigger Create a trigger that fires for every new object whose key starts with `photos/`: ``` vastde triggers create element --name docs-src-uploads --description "New objects under photos/ in docs-src" --source-bucket "$BUCKET" --event "ObjectCreated:*" --name-prefix photos/ ``` ``` Trigger created: docs-src-uploads Name: docs-src-uploads Description: New objects under photos/ in docs-src Tags: [] GUID: 11f2e798-f6c6-427f-b30f-0f20bdb8172a Owner: {4294967295 vid 0xc0002da020 map[]} CreatedAt: 2026-10-08T12:24:39Z UpdatedAt: 2026-10-08T12:24:39Z VRN: vast:dataengine:triggers:docs-src-uploads Type: Element Events: -: ObjectCreated:* Topic: vast:dataengine:topics:docs-kafka/default SourceBucket: docs-src ``` With no topic given, the trigger publishes its events to your tenant's default topic. See [triggers](/develop/docs/dev-dataengine-guides-triggers) for filters, other event types and schedules. ## 7. Write the pipeline file A pipeline runs on one compute cluster, in one namespace. Find yours: ``` vastde compute-clusters list ``` ``` Compute Cluster Description GUID Updated at --------------------------------------------------------------------------------------- test-cluster None 2a052533-ebb2-4b2e-8b0... 2026-10-08 12:03 ``` ``` vastde compute-clusters get test-cluster -o yaml ``` ``` created_at: "2026-10-08T12:03:24.742Z" guid: 2a052533-ebb2-4b2e-8b0d-718c3533d5dc id: 7.51393205803339e+18 is_internal: true kube_api_url: https://:6443 mtls_credentials_guid: be58a611-249f-4078-9281-5109264d4d19 name: test-cluster namespaces: - dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91 owner: id: "4294967295" id_type: vid name: root tenant_guid: da47bae0-aefd-4e85-a5a9-2020d6a2cb91 updated_at: "2026-10-08T12:03:24.742Z" vrn: vast:dataengine:kubernetes-clusters:test-cluster ``` Use your cluster's name in the command. You need two values from it: `vrn`, and your tenant's namespace under `namespaces` (on a VAST compute cluster, `dataengine-ns-<tenant GUID>`). Save this as `pipeline.yaml`, with your cluster's VRN and your namespace in place of the lab's: ``` name: thumbnails description: Make a thumbnail for every new image in docs-src/photos/ kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91 manifest: triggers: - name: uploads vrn: vast:dataengine:triggers:docs-src-uploads function_deployments: - name: thumbnailer function_vrn: vast:dataengine:functions:thumbnailer links: - source: [uploads] destination: [thumbnailer] ``` The manifest gives the trigger and the function local names (`uploads`, `thumbnailer`) and links them: every event from `uploads` goes to `thumbnailer`. The [pipeline YAML](/develop/docs/dev-dataengine-reference-pipeline-yaml) reference lists every field. ## 8. Create and deploy the pipeline ``` vastde pipelines create --config @pipeline.yaml --deploy ``` ``` Pipeline created: thumbnails Name: thumbnails Description: Make a thumbnail for every new image in docs-src/photos/ Tags: [] GUID: c7f63133-3438-4cae-af6a-91291041f1b6 Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T12:25:11Z Updated At: 2026-10-08T12:25:11Z VRN: vast:dataengine:pipelines:thumbnails Deploying pipeline: thumbnails Pipeline deployed successfully ``` The command returns once the deploy is accepted. The compute cluster then pulls the image and starts the function, and the status goes from `InProgress` to `Ready`: ``` vastde pipelines get thumbnails ``` ``` Name: thumbnails Description: Make a thumbnail for every new image in docs-src/photos/ Tags: [] GUID: c7f63133-3438-4cae-af6a-91291041f1b6 Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T12:25:11Z Updated At: 2026-10-08T12:25:11Z VRN: vast:dataengine:pipelines:thumbnails Status: InProgress ``` Run it again until the status is `Ready`. In the lab that took about 30 seconds: ``` vastde pipelines get thumbnails ``` ``` Name: thumbnails Description: Make a thumbnail for every new image in docs-src/photos/ Tags: [] GUID: c7f63133-3438-4cae-af6a-91291041f1b6 Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T12:25:11Z Updated At: 2026-10-08T12:25:11Z VRN: vast:dataengine:pipelines:thumbnails Status: Ready ``` If it shows `Error` instead, `vastde pipelines get thumbnails -o yaml` shows the `reason`. ## 9. Upload an object Upload any small file as `photos/cat.jpg`: ``` aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 cp cat.jpg "s3://$BUCKET/photos/cat.jpg" ``` ``` Completed 132 Bytes/132 Bytes (1.7 KiB/s) with 1 file(s) remaining upload: ./cat.jpg to s3://docs-src/photos/cat.jpg ``` ## 10. Read the logs and traces The function ran within a second of the upload. Its logs: ``` vastde logs get thumbnails --since 10m ``` ``` 2026-10-08 12:25:28.67 [thumbnailer] [INFO] [vast-runtime] Print function intercepted and redirected to 'user-print' logger 2026-10-08 12:25:28.96 [thumbnailer] [INFO] [vast-runtime] Runtime is listening on port 8080 2026-10-08 12:25:28.97 [thumbnailer] [INFO] [vast-runtime] Sync handler detected, concurrency limited to 20 via middleware 2026-10-08 12:25:29.15 [thumbnailer] [INFO] [vast-runtime] ConcurrencyLimitMiddleware initialized with max_concurrent=20 2026-10-08 12:25:29.15 [thumbnailer] [INFO] [vast-runtime] Configured private thread pool executor with 20 workers 2026-10-08 12:25:29.83 [thumbnailer] [INFO] [vast-runtime] Initialized HTTP client with HTTP/2 support for CloudEvent sending 2026-10-08 12:25:29.84 [thumbnailer] [INFO] [vast-runtime] Call Init Handler: init 2026-10-08 12:25:29.84 [thumbnailer] [INFO] [user] thumbnailer initialized 2026-10-08 12:25:29.84 [thumbnailer] [INFO] [vast-runtime] Init Handler Completed, duration: 0:00:00.001268 2026-10-08 12:25:29.84 [thumbnailer] [INFO] [vast-runtime] Handler is now ready to accept requests 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] START EventId: 88051124535297, TraceId: -, EventType: vastdata.com:Element.ElementCreated, EventSource: vastdata.com:docs-src-uploads.11f2e798-f6c6-427f-b30f-0f20bdb8172a, Timestamp: 1791462418.9423726 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] Function invoked in timestamp 2026-10-08 12:26:58.942813. Event batching mode: False 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] Starting running function in timestamp 2026-10-08 12:26:58.944075 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [user] new object: bucket=docs-src key=photos/cat.jpg 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] Function returned: None in timestamp 2026-10-08 12:26:58.944853 with duration 0.5662441253662109 milliseconds 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] No sink URL configured 2026-10-08 12:26:58.94 [thumbnailer] [INFO] [vast-runtime] END EventId: 88051124535297, TraceId: -, Duration: 0.0032s, Response: , Timestamp: 1791462418.9454546 ``` `[user]` lines come from your code and `[vast-runtime]` lines from the DataEngine runtime around it. Add `--scope user` to see only yours. The traces: ``` vastde traces list thumbnails --since 10m ``` ``` Trace ID Status Pipeline Function Start Time End Time -------------------------------------------------------------------------------------------------------------------------------------- 8d3144dcbc26b05e21a4c6e59554e4fc OK thumbnails thumbnailer 2026-10-08 12:25:29.840 2026-10-08 12:25:29.842 0000501500000001b49102791a8a492a OK thumbnails thumbnailer 2026-10-08 12:26:58.944 2026-10-08 12:26:58.945 ``` The first trace is the instance's `init` call and the second is the upload. `vastde traces get <trace ID>` shows one trace's event and function. See [observability](/develop/docs/dev-dataengine-guides-observability). ## Clean up The guides in this section build on this pipeline, function and trigger. If you're going on to them, keep everything. Otherwise, delete the pipeline first (it stops the function on the compute cluster), then the trigger and the function, then the test object: ``` vastde pipelines delete thumbnails vastde triggers delete docs-src-uploads vastde functions delete thumbnailer aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 rm "s3://$BUCKET/photos/cat.jpg" ``` `vastde pipelines delete` prints `Deleting pipeline: <GUID>` and then `Pipeline deleted successfully`. The image stays in your registry. ## Next steps - [Write a function](/develop/docs/dev-dataengine-guides-write-a-function): events, return values, chaining, secrets and dependencies. - [Pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code): keep `pipeline.yaml` in Git, and update and deploy it. - [Publish functions](/develop/docs/dev-dataengine-guides-publish-functions): ship a new version of `thumbnailer`. - [Concepts](/develop/docs/dev-dataengine-concepts-index): functions, revisions, triggers and pipelines.