Integrations

Patterns for connecting DataEngine to other systems with code you write, since none of them is built in. Start with VAST DataBase and Vector Search from a function if your functions write tables, and Consume the dead-letter topic before you go to production.

• Integrations: Patterns for connecting DataEngine to VAST DataBase, the dead-letter topic, OpenTelemetry, orchestrators and lineage tools, none of them built in.
• VAST DataBase and Vector Search from a function: Write VAST DataBase rows and embeddings from a function with the vastdb SDK, with S3 keys passed as a secret.
• Consume the dead-letter topic: Read your tenant's dead-letter topic with a Kafka consumer, decode why each event failed, and alert on it or replay it.
• Your own OpenTelemetry backend: Send a function's traces and logs to your own OpenTelemetry collector over OTLP, alongside DataEngine's telemetry and with the same trace ID.
• Orchestrators (Airflow and similar): Start a pipeline from an Airflow or similar task by writing an object, wait for its run, and fail on errors.
• OpenLineage and OpenMetadata: Emit OpenLineage run events from a function with openlineage-python, over HTTP or an Event Broker topic, for Marquez or OpenMetadata.