Step-by-step instructions for building and running pipelines from the command line. Start with Write a function and Build and test locally; then Publish functions, Triggers and Pipelines as code take you to a deployed pipeline, and Observability shows what it's doing.
- Write a function: Write a DataEngine handler in Python: init and handler, reading each event type, chaining with return values, secrets, environment variables and dependencies.
- Build and test locally: Build a function image with vastde functions build, run it locally with localrun and send it test events with invoke before deploying.
- Publish functions: Register a function image, release new versions as revisions, and choose whether a pipeline follows the latest revision or stays pinned.
- Triggers: Create element triggers for bucket changes and schedule triggers on a cron schedule, with the filters and syntaxes the cluster accepts.
- Pipelines as code: Keep a pipeline in a Git-tracked YAML file: create, update and deploy it, chain functions, route events, and set retries.
- Observability: Read and follow logs, inspect traces, and query built-in and custom metrics for a pipeline with vastde logs, traces and metrics.