--- title: "Your own OpenTelemetry backend" slug: "dev-dataengine-integrations-opentelemetry" description: "Integrate DataEngine with your OpenTelemetry backend to track logs, traces, and metrics seamlessly, ensuring consistent trace IDs across platforms." tags: ["DataEngine"] updated: 2026-10-28T02:14:59Z published: 2026-10-28T02:14:59Z canonical: "docs.vastdata.com/dev-dataengine-integrations-opentelemetry" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Your own OpenTelemetry backend *Applies to VAST Cluster 5.5.* DataEngine collects your functions' logs, traces and metrics for you, and you read them with `vastde logs`, `vastde traces` and `vastde metrics` (see [observability](/develop/docs/dev-dataengine-guides-observability)). If your team also watches its services in its own OpenTelemetry backend (an OpenTelemetry Collector in front of Jaeger, Tempo, Grafana, Datadog or similar), this page shows how to send your function's telemetry there too, with the same trace ID that DataEngine uses, so you can find one event's run in both places. **This isn't a built-in DataEngine integration in 5.5.** DataEngine can't forward its own telemetry to another backend, and you can't point it somewhere else. What you can do is run a second OpenTelemetry pipeline inside your function, next to DataEngine's: your own tracer provider and log handler, exporting over OTLP to your collector. The rest of this page shows that pattern and what does and doesn't reach each side. Commands use the environment variables from the [DataEngine quickstart](/develop/docs/dev-dataengine-quickstart#before-you-start). The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6 with `vastde` v5.5.0-sp2, where the registry host is `127.0.0.1:5000` and the trigger's source bucket is `docs-src`. Your names, GUIDs, addresses and times will differ. ## What reaches where | Telemetry | DataEngine (`vastde logs`, `traces`, `metrics`) | Your backend | | --- | --- | --- | | The runtime's own logs, traces and metrics | Yes | No. Setting `OTEL_EXPORTER_OTLP_ENDPOINT` in the pipeline doesn't redirect them. | | Spans you create with `ctx.tracer` (and `@span_wrapped`, `@event_span_wrapped`) | Yes | No | | Spans you create with your own tracer provider | No | Yes, in DataEngine's trace | | Records you log with `ctx.logger`, after you add your own log handler | Yes | Yes, with DataEngine's trace ID | DataEngine's runtime builds its exporters with the address of DataEngine's own collector, which an endpoint you set in an environment variable doesn't override. It doesn't register its tracer provider or logger provider as the process-wide default, so yours can be the default without affecting DataEngine's. It does register the process-wide meter provider, so metrics you want in your own backend need a separate `MeterProvider` that you call directly; this page doesn't cover metrics. ## Before you start You need: - An OpenTelemetry Collector, or a backend that accepts OTLP over HTTP, at an address your function's pods can reach. The pods connect to it from the compute cluster, not from your machine, so ask your platform team to confirm the route. See [what a function can reach](/develop/docs/dev-dataengine-integrations-openlineage-and-openmetadata#what-a-function-can-reach). - The OpenTelemetry Python packages at the versions the DataEngine runtime uses. The builder pins `opentelemetry-api`, `opentelemetry-sdk` and `opentelemetry-exporter-otlp` to the same version as the runtime (1.39.0 with builder `vast-builder:v5.5.0-sp2`); the build log lists the pins under `Runtime dependency constraints applied` (see [VAST DataBase and Vector Search](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search#build-the-image)). Pin your OpenTelemetry packages to the same version. For a test, a Collector that prints everything it receives is enough. This is the configuration used for the outputs below; it listens for OTLP over HTTP on port 14318 (OTLP's usual HTTP port is 4318): ``` receivers: otlp: protocols: http: endpoint: 0.0.0.0:14318 exporters: debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] exporters: [debug] logs: receivers: [otlp] exporters: [debug] metrics: receivers: [otlp] exporters: [debug] ``` Save it as `collector.yaml` on a host the pods can reach, and run the Collector: ``` docker run -d --name otel-collector --network host -v $PWD/collector.yaml:/etc/otelcol/config.yaml:ro otel/opentelemetry-collector:0.110.0 --config /etc/otelcol/config.yaml ``` ``` Unable to find image 'otel/opentelemetry-collector:0.110.0' locally 0.110.0: Pulling from otel/opentelemetry-collector … Status: Downloaded newer image for otel/opentelemetry-collector:0.110.0 060d70e31e9e6f43f947b213f5b9dbe59c6504f665955c4355c1779fe67977f0 ``` ``` sleep 5; docker logs otel-collector 2>&1 | tail -3 ``` ``` 2026-10-08T14:14:25.104Z warn internal@v0.110.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2026-10-08T14:14:25.104Z info otlpreceiver@v0.110.0/otlp.go:169 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:14318"} 2026-10-08T14:14:25.104Z info service@v0.110.0/service.go:234 Everything is ready. Begin running and processing data. ``` ## Write the function The function sets up its own tracer provider and logger provider when the module loads, each with an OTLP/HTTP exporter, and makes the tracer provider the process-wide default. In `init`, it adds its log handler to `ctx.logger`, so everything the function logs goes to both DataEngine and your Collector. Save this as `otelfn/main.py`: ``` import logging from opentelemetry import trace from opentelemetry._logs import set_logger_provider from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor # Our own OTel pipeline, separate from the DataEngine runtime's. The exporters take their # endpoint from OTEL_EXPORTER_OTLP_ENDPOINT (set in the pipeline's environment variables). resource = Resource.create({"service.name": "otel-demo"}) tracer_provider = TracerProvider(resource=resource) tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(tracer_provider) logger_provider = LoggerProvider(resource=resource) logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter())) set_logger_provider(logger_provider) external_logs = LoggingHandler(level=logging.INFO, logger_provider=logger_provider) tracer = trace.get_tracer("otel-demo") def init(ctx): ctx.logger.addHandler(external_logs) ctx.logger.info("otel-demo ready, logger %s", ctx.logger.name) def handler(ctx, event): current = trace.get_current_span().get_span_context() ctx.logger.info("event %s DataEngine trace %s, current span trace %032x", event.id, event.get_hex_trace_id(), current.trace_id) with tracer.start_as_current_span("external-span") as span: span.set_attribute("object.key", event.object_key) ctx.logger.info("inside external-span trace %032x span %016x", span.get_span_context().trace_id, span.get_span_context().span_id) with ctx.tracer.start_as_current_span("ctx-tracer-span"): ctx.logger.info("inside ctx.tracer span") tracer_provider.force_flush() logger_provider.force_flush() ``` The handler logs two candidate trace IDs for comparison, creates one span with its own tracer and one with `ctx.tracer`, and flushes its exporters so the test shows results straight away. Flushing in every call adds the export's round trip to each event; without it, the batch processors export in the background, and records still buffered when an instance shuts down can be lost. Its `otelfn/requirements.txt`: ``` opentelemetry-sdk==1.39.0 opentelemetry-exporter-otlp-proto-http==1.39.0 ``` Build and push the image, and register the function: ``` vastde functions build "$REGISTRY/docs/otelfn" --target ./otelfn --image-tag v1 --push ``` ``` Detected language: python Validating Python version 3.12.*... Python version 3.12.* resolved to 3.12.13 Building 127.0.0.1:5000/docs/otelfn:v1 App Path: ./otelfn Handlers File: main.py 2026/10/08 14:15:03 [Started] Python Builder: 127.0.0.1:5000/docs/otelfn:v1 2026/10/08 14:15:50 [Completed] Python Builder: 127.0.0.1:5000/docs/otelfn:v1 Build completed: 127.0.0.1:5000/docs/otelfn:v1 ``` ``` vastde functions create --name otel-demo --artifact-type image --artifact-source docs/otelfn --image-tag v1 --container-registry "$REGISTRY_NAME" --publish ``` ``` Function created: otel-demo Name: otel-demo Tags: [] GUID: d252a601-f3cd-4efb-b289-e0d3eb3b6fdc Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T14:15:54Z Updated At: 2026-10-08T14:15:54Z VRN: vast:dataengine:functions:otel-demo Last Revision: 1 ``` ## Create and deploy the pipeline Create an element trigger for new objects under `otel/`: ``` vastde triggers create element --name otel-uploads --source-bucket "$BUCKET" --event "ObjectCreated:*" --name-prefix otel/ ``` ``` Trigger created: otel-uploads Name: otel-uploads Description: N/A Tags: [] GUID: 12cf59ac-14c1-4d15-8484-64398e12179a Owner: {4294967295 vid 0xc00005f5b0 map[]} CreatedAt: 2026-10-08T14:16:02Z UpdatedAt: 2026-10-08T14:16:02Z VRN: vast:dataengine:triggers:otel-uploads Type: Element Events: -: ObjectCreated:* Topic: vast:dataengine:topics:docs-kafka/default SourceBucket: docs-src ``` Save this as `otel-demo.yaml`, with your compute cluster, namespace and Collector address. The Collector's address goes in `OTEL_EXPORTER_OTLP_ENDPOINT`, which your exporters read; the exporters add the `/v1/traces` and `/v1/logs` paths themselves: ``` name: otel-demo description: user code exports its own traces and logs to an external OTLP collector kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91 manifest: triggers: - name: uploads vrn: vast:dataengine:triggers:otel-uploads function_deployments: - name: otelfn function_vrn: vast:dataengine:functions:otel-demo config: environment_variables: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://:14318 links: - source: [uploads] destination: [otelfn] ``` ``` vastde pipelines create --config @otel-demo.yaml --deploy ``` ``` Pipeline created: otel-demo Name: otel-demo Description: user code exports its own traces and logs to an external OTLP collector Tags: [] GUID: 3f751ed4-497f-4bac-a260-c99ac854e10d Owner: [id: 4294967295, id-type: vid] Created At: 2026-10-08T14:16:15Z Updated At: 2026-10-08T14:16:15Z VRN: vast:dataengine:pipelines:otel-demo Deploying pipeline: otel-demo Pipeline deployed successfully ``` Wait for it to be ready: ``` until s=$(vastde pipelines get otel-demo -o json | jq -r .status); [ "$s" = Ready ] || [ "$s" = Error ]; do sleep 5; done; vastde pipelines get otel-demo -o json | jq -c "{status, reason}" ``` ``` {"status":"Ready","reason":"Pipeline is ready and active."} ``` `OTEL_EXPORTER_OTLP_ENDPOINT` affects only exporters that read it, which here are yours. Settings that DataEngine's runtime doesn't set itself, such as `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_COMPRESSION` or `OTEL_SDK_DISABLED`, may also apply to the runtime's own exporters if you set them for the whole function. Pass headers, such as your backend's API key, to your exporters in code (`OTLPSpanExporter(headers=...)`), and keep the key in a [secret](/develop/docs/dev-dataengine-guides-write-a-function#environment-variables-and-secrets). ## Run it Upload an object, and read the function's logs in DataEngine: ``` echo otel > otel-1.txt aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 cp otel-1.txt "s3://$BUCKET/otel/otel-1.txt" >/dev/null; sleep 15; vastde logs get otel-demo --since 3m --scope user ``` ``` 2026-10-08 14:16:44.27 [otel-demo] [INFO] [user] otel-demo ready, logger user 2026-10-08 14:17:12.44 [otel-demo] [INFO] [user] event 87982405058583 DataEngine trace 00000000000000000000500500000017, current span trace 00005005000000176b3fa2c25502414f 2026-10-08 14:17:12.44 [otel-demo] [INFO] [user] inside external-span trace 00005005000000176b3fa2c25502414f span 3e4b75affef6456f 2026-10-08 14:17:12.44 [otel-demo] [INFO] [user] inside ctx.tracer span ``` ``` vastde traces list otel-demo --since 5m ``` ``` Trace ID Status Pipeline Function Start Time End Time -------------------------------------------------------------------------------------------------------------------------------------- 91c9a3c4002598fb6c5f39431cc7fac8 OK otel-demo otel-demo 2026-10-08 14:16:44.275 2026-10-08 14:16:44.277 00005005000000176b3fa2c25502414f OK otel-demo otel-demo 2026-10-08 14:17:12.446 2026-10-08 14:17:12.453 ``` The first trace is the instance's `init`; the second is the event. The event's trace ID, `00005005000000176b3fa2c25502414f`, is the one the handler read from `trace.get_current_span()`, and the one its own span `external-span` belongs to. The value from `event.get_hex_trace_id()`, `00000000000000000000500500000017`, isn't: see [get the trace ID](/develop/docs/dev-dataengine-integrations-opentelemetry#get-the-trace-id). DataEngine's spans for the event, from the trace-tree endpoint of the DataEngine telemetry API (see [orchestrators](/develop/docs/dev-dataengine-integrations-orchestrators#authenticate) for authentication). Set `VMS_URL` to `https://<VMS address>`, and `VAST_TENANT`, `VAST_USERNAME` and `VAST_PASSWORD` to your tenant and DataEngine user. `-k` skips the certificate check, for a lab VMS with a self-signed certificate; leave it out if your VMS certificate is trusted: ``` curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VMS_URL/api/dataengine/telemetries/trace-tree?trace_id=00005005000000176b3fa2c25502414f" | jq -c '.. | objects | select(has("span_name")) | {span_name, span_id, parent_span_id, scope}' ``` ``` {"span_name":"vastfunction-execution","span_id":"9a205f3d216ee443","parent_span_id":"0d6cfa7e25208732","scope":"vast-runtime"} {"span_name":"ctx-tracer-span","span_id":"082d57040126582f","parent_span_id":"9a205f3d216ee443","scope":"user"} ``` DataEngine has the runtime's span for the handler call and the `ctx.tracer` span, but not `external-span`. What the Collector received: ``` docker logs otel-collector 2>&1 | grep -E "^(Span|LogRecord) #|Trace ID|Parent ID|^ ID|Name |service.name|Body|InstrumentationScope" ``` ``` -> service.name: Str(otel-demo) InstrumentationScope user LogRecord #0 Body: Str(otel-demo ready, logger user) Trace ID: -> service.name: Str(otel-demo) InstrumentationScope otel-demo Span #0 Trace ID : 00005005000000176b3fa2c25502414f Parent ID : 9a205f3d216ee443 ID : 3e4b75affef6456f Name : external-span -> service.name: Str(otel-demo) InstrumentationScope user LogRecord #0 Body: Str(event 87982405058583 DataEngine trace 00000000000000000000500500000017, current span trace 00005005000000176b3fa2c25502414f) Trace ID: 00005005000000176b3fa2c25502414f LogRecord #1 Body: Str(inside external-span trace 00005005000000176b3fa2c25502414f span 3e4b75affef6456f) Trace ID: 00005005000000176b3fa2c25502414f LogRecord #2 Body: Str(inside ctx.tracer span) Trace ID: 00005005000000176b3fa2c25502414f ``` And nothing from DataEngine's runtime, which reports itself as `service.name` `vast-function`: ``` docker logs otel-collector 2>&1 | grep -c "service.name: Str(vast-function)" ``` ``` 0 ``` So, in your backend: - `external-span` is in DataEngine's trace, `00005005000000176b3fa2c25502414f`, and its parent is `9a205f3d216ee443`, DataEngine's `vastfunction-execution` span. That parent exists only in DataEngine, so your backend shows `external-span` with a missing parent. Search either system for the trace ID to find the same event. - The log records the handler wrote during the event carry the event's trace ID. The `init` record has none: `init` runs before any event. - The `ctx.tracer` span isn't there. Create spans with your own tracer for anything you want to see in your backend, and with `ctx.tracer` for anything you want in DataEngine's trace views; a span created with one isn't sent to the other. ## Get the trace ID To log the trace ID, or pass it to another system (for example as an OpenLineage run ID, as in [OpenLineage and OpenMetadata](/develop/docs/dev-dataengine-integrations-openlineage-and-openmetadata)), read it from the current span inside the handler: ``` from opentelemetry import trace def handler(ctx, event): trace_id = trace.get_current_span().get_span_context().trace_id # an int ctx.logger.info("trace %032x", trace_id) # 32 hex digits, as vastde traces list shows it ``` In VAST Cluster 5.5.0-SP6, `event.get_hex_trace_id()` called without an argument doesn't return the event's trace ID: in the run above it returned `00000000000000000000500500000017` for the trace `00005005000000176b3fa2c25502414f`. Use the current span instead. ## Clean up Delete the pipeline, the trigger and the function (`vastde pipelines delete otel-demo`, `vastde triggers delete otel-uploads`, `vastde functions delete otel-demo`), and stop the test Collector with `docker rm -f otel-collector`. ## See also - [Observability](/develop/docs/dev-dataengine-guides-observability): logs, traces and metrics in DataEngine. - [Logs, traces and metrics](/develop/docs/dev-dataengine-concepts-architecture#logs-traces-and-metrics): where DataEngine's telemetry goes. - [Runtime SDK: tracing](/develop/docs/dev-dataengine-reference-runtime-sdk#tracing): `ctx.tracer` and the span decorators. - [OpenLineage and OpenMetadata](/develop/docs/dev-dataengine-integrations-openlineage-and-openmetadata): what a function's pods can reach, and the trace ID as a lineage run ID. - [Integrations](/develop/docs/dev-dataengine-integrations-index)