Integrations

Applies to VAST Cluster 5.5.

These pages show how to connect DataEngine functions and pipelines to other systems: VAST DataBase tables, the dead-letter topic, your own observability stack, workflow orchestrators and lineage tools.

None of these is a built-in DataEngine integration in VAST Cluster 5.5. DataEngine has no connectors for them. Each page is a pattern: code you write in your handler, or a client you run next to DataEngine, using the SDKs, the REST API and the Kafka protocol.

Page	What it shows
VAST DataBase and Vector Search	Writing rows to a VAST DataBase table from a handler with the `vastdb` SDK: dependencies, S3 keys as a secret, connecting in `init`, and reading the table back. Embeddings follow the same pattern.
Dead-letter topic	A Kafka consumer that reads the tenant's dead-letter topic, decodes why each event failed, and alerts on it or replays it.
OpenTelemetry	Sending your function's telemetry to an OpenTelemetry backend you run, as well as to DataEngine.
Orchestrators	Driving and watching pipelines from Airflow and similar orchestrators, through the REST API.
OpenLineage and OpenMetadata	Emitting OpenLineage run events from a handler, from the object that triggered it to the table or object it wrote.

Before you start

Every pattern runs inside your function or next to it, so it needs what a function needs to reach the other system:

Network access from the compute cluster. A function's pods connect to databases, brokers and backends from the compute cluster, not from your machine. Ask your platform team which addresses the compute cluster can reach.
Credentials as secrets. Pass keys and tokens as DataEngine secrets, and settings as environment variables without the reserved VAST_ prefix. See security and write a function.
Dependencies in the image. Clients such as vastdb or openlineage-python go in requirements.txt, under the builder's constraints for the runtime's own packages. See write a function.

Integrations

Before you start

See also