Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Your own OpenTelemetry backend

Prev Next

Applies to VAST Cluster 5.5.

DataEngine collects your functions' logs, traces and metrics for you, and you read them with vastde logs, vastde traces and vastde metrics (see observability). If your team also watches its services in its own OpenTelemetry backend (an OpenTelemetry Collector in front of Jaeger, Tempo, Grafana, Datadog or similar), this page shows how to send your function's telemetry there too, with the same trace ID that DataEngine uses, so you can find one event's run in both places.

This isn't a built-in DataEngine integration in 5.5. DataEngine can't forward its own telemetry to another backend, and you can't point it somewhere else. What you can do is run a second OpenTelemetry pipeline inside your function, next to DataEngine's: your own tracer provider and log handler, exporting over OTLP to your collector. The rest of this page shows that pattern and what does and doesn't reach each side.

Commands use the environment variables from the DataEngine quickstart. The outputs come from a lab cluster running VAST Cluster 5.5.0-SP6 with vastde v5.5.0-sp2, where the registry host is 127.0.0.1:5000 and the trigger's source bucket is docs-src. Your names, GUIDs, addresses and times will differ.

What reaches where

Telemetry DataEngine (vastde logs, traces, metrics) Your backend
The runtime's own logs, traces and metrics Yes No. Setting OTEL_EXPORTER_OTLP_ENDPOINT in the pipeline doesn't redirect them.
Spans you create with ctx.tracer (and @span_wrapped, @event_span_wrapped) Yes No
Spans you create with your own tracer provider No Yes, in DataEngine's trace
Records you log with ctx.logger, after you add your own log handler Yes Yes, with DataEngine's trace ID

DataEngine's runtime builds its exporters with the address of DataEngine's own collector, which an endpoint you set in an environment variable doesn't override. It doesn't register its tracer provider or logger provider as the process-wide default, so yours can be the default without affecting DataEngine's. It does register the process-wide meter provider, so metrics you want in your own backend need a separate MeterProvider that you call directly; this page doesn't cover metrics.

Before you start

You need:

  • An OpenTelemetry Collector, or a backend that accepts OTLP over HTTP, at an address your function's pods can reach. The pods connect to it from the compute cluster, not from your machine, so ask your platform team to confirm the route. See what a function can reach.
  • The OpenTelemetry Python packages at the versions the DataEngine runtime uses. The builder pins opentelemetry-api, opentelemetry-sdk and opentelemetry-exporter-otlp to the same version as the runtime (1.39.0 with builder vast-builder:v5.5.0-sp2); the build log lists the pins under Runtime dependency constraints applied (see VAST DataBase and Vector Search). Pin your OpenTelemetry packages to the same version.

For a test, a Collector that prints everything it receives is enough. This is the configuration used for the outputs below; it listens for OTLP over HTTP on port 14318 (OTLP's usual HTTP port is 4318):

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:14318
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
    logs:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      exporters: [debug]

Save it as collector.yaml on a host the pods can reach, and run the Collector:

docker run -d --name otel-collector --network host -v $PWD/collector.yaml:/etc/otelcol/config.yaml:ro otel/opentelemetry-collector:0.110.0 --config /etc/otelcol/config.yaml

Unable to find image 'otel/opentelemetry-collector:0.110.0' locally
0.110.0: Pulling from otel/opentelemetry-collector
…
Status: Downloaded newer image for otel/opentelemetry-collector:0.110.0
060d70e31e9e6f43f947b213f5b9dbe59c6504f665955c4355c1779fe67977f0

sleep 5; docker logs otel-collector 2>&1 | tail -3

2026-10-08T14:14:25.104Z	warn	internal@v0.110.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2026-10-08T14:14:25.104Z	info	otlpreceiver@v0.110.0/otlp.go:169	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:14318"}
2026-10-08T14:14:25.104Z	info	service@v0.110.0/service.go:234	Everything is ready. Begin running and processing data.

Write the function

The function sets up its own tracer provider and logger provider when the module loads, each with an OTLP/HTTP exporter, and makes the tracer provider the process-wide default. In init, it adds its log handler to ctx.logger, so everything the function logs goes to both DataEngine and your Collector. Save this as otelfn/main.py:

import logging

from opentelemetry import trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Our own OTel pipeline, separate from the DataEngine runtime's. The exporters take their
# endpoint from OTEL_EXPORTER_OTLP_ENDPOINT (set in the pipeline's environment variables).
resource = Resource.create({"service.name": "otel-demo"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tracer_provider)
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
set_logger_provider(logger_provider)
external_logs = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)

tracer = trace.get_tracer("otel-demo")

def init(ctx):
    ctx.logger.addHandler(external_logs)
    ctx.logger.info("otel-demo ready, logger %s", ctx.logger.name)

def handler(ctx, event):
    current = trace.get_current_span().get_span_context()
    ctx.logger.info("event %s DataEngine trace %s, current span trace %032x",
                    event.id, event.get_hex_trace_id(), current.trace_id)
    with tracer.start_as_current_span("external-span") as span:
        span.set_attribute("object.key", event.object_key)
        ctx.logger.info("inside external-span trace %032x span %016x",
                        span.get_span_context().trace_id, span.get_span_context().span_id)
    with ctx.tracer.start_as_current_span("ctx-tracer-span"):
        ctx.logger.info("inside ctx.tracer span")
    tracer_provider.force_flush()
    logger_provider.force_flush()

The handler logs two candidate trace IDs for comparison, creates one span with its own tracer and one with ctx.tracer, and flushes its exporters so the test shows results straight away. Flushing in every call adds the export's round trip to each event; without it, the batch processors export in the background, and records still buffered when an instance shuts down can be lost.

Its otelfn/requirements.txt:

opentelemetry-sdk==1.39.0
opentelemetry-exporter-otlp-proto-http==1.39.0

Build and push the image, and register the function:

vastde functions build "$REGISTRY/docs/otelfn" --target ./otelfn --image-tag v1 --push

Detected language: python
Validating Python version 3.12.*...
Python version 3.12.* resolved to 3.12.13
Building 127.0.0.1:5000/docs/otelfn:v1
App Path: ./otelfn
Handlers File: main.py
2026/10/08 14:15:03 [Started] Python Builder: 127.0.0.1:5000/docs/otelfn:v1
2026/10/08 14:15:50 [Completed] Python Builder: 127.0.0.1:5000/docs/otelfn:v1
Build completed: 127.0.0.1:5000/docs/otelfn:v1

vastde functions create --name otel-demo --artifact-type image --artifact-source docs/otelfn --image-tag v1 --container-registry "$REGISTRY_NAME" --publish

Function created: otel-demo
Name: otel-demo
Tags: []
GUID: d252a601-f3cd-4efb-b289-e0d3eb3b6fdc
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T14:15:54Z
Updated At: 2026-10-08T14:15:54Z
VRN: vast:dataengine:functions:otel-demo
Last Revision: 1

Create and deploy the pipeline

Create an element trigger for new objects under otel/:

vastde triggers create element --name otel-uploads --source-bucket "$BUCKET" --event "ObjectCreated:*" --name-prefix otel/

Trigger created: otel-uploads
Name: otel-uploads
Description: N/A
Tags: []
GUID: 12cf59ac-14c1-4d15-8484-64398e12179a
Owner: {4294967295 vid 0xc00005f5b0 map[]}
CreatedAt: 2026-10-08T14:16:02Z
UpdatedAt: 2026-10-08T14:16:02Z
VRN: vast:dataengine:triggers:otel-uploads
Type: Element
Events:
  -: ObjectCreated:*
Topic: vast:dataengine:topics:docs-kafka/default
SourceBucket: docs-src

Save this as otel-demo.yaml, with your compute cluster, namespace and Collector address. The Collector's address goes in OTEL_EXPORTER_OTLP_ENDPOINT, which your exporters read; the exporters add the /v1/traces and /v1/logs paths themselves:

name: otel-demo
description: user code exports its own traces and logs to an external OTLP collector
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:test-cluster
namespace: dataengine-ns-da47bae0-aefd-4e85-a5a9-2020d6a2cb91
manifest:
  triggers:
    - name: uploads
      vrn: vast:dataengine:triggers:otel-uploads
  function_deployments:
    - name: otelfn
      function_vrn: vast:dataengine:functions:otel-demo
      config:
        environment_variables:
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: http://<collector host>:14318
  links:
    - source: [uploads]
      destination: [otelfn]

vastde pipelines create --config @otel-demo.yaml --deploy

Pipeline created: otel-demo
Name: otel-demo
Description: user code exports its own traces and logs to an external OTLP collector
Tags: []
GUID: 3f751ed4-497f-4bac-a260-c99ac854e10d
Owner: [id: 4294967295, id-type: vid]
Created At: 2026-10-08T14:16:15Z
Updated At: 2026-10-08T14:16:15Z
VRN: vast:dataengine:pipelines:otel-demo

Deploying pipeline: otel-demo
Pipeline deployed successfully

Wait for it to be ready:

until s=$(vastde pipelines get otel-demo -o json | jq -r .status); [ "$s" = Ready ] || [ "$s" = Error ]; do sleep 5; done; vastde pipelines get otel-demo -o json | jq -c "{status, reason}"

{"status":"Ready","reason":"Pipeline is ready and active."}

OTEL_EXPORTER_OTLP_ENDPOINT affects only exporters that read it, which here are yours. Settings that DataEngine's runtime doesn't set itself, such as OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_COMPRESSION or OTEL_SDK_DISABLED, may also apply to the runtime's own exporters if you set them for the whole function. Pass headers, such as your backend's API key, to your exporters in code (OTLPSpanExporter(headers=...)), and keep the key in a secret.

Run it

Upload an object, and read the function's logs in DataEngine:

echo otel > otel-1.txt
aws --region us-east-1 --endpoint-url "$S3_ENDPOINT" s3 cp otel-1.txt "s3://$BUCKET/otel/otel-1.txt" >/dev/null; sleep 15; vastde logs get otel-demo --since 3m --scope user

2026-10-08 14:16:44.27 [otel-demo] [INFO]  [user] otel-demo ready, logger user
2026-10-08 14:17:12.44 [otel-demo] [INFO]  [user] event 87982405058583 DataEngine trace 00000000000000000000500500000017, current span trace 00005005000000176b3fa2c25502414f
2026-10-08 14:17:12.44 [otel-demo] [INFO]  [user] inside external-span trace 00005005000000176b3fa2c25502414f span 3e4b75affef6456f
2026-10-08 14:17:12.44 [otel-demo] [INFO]  [user] inside ctx.tracer span

vastde traces list otel-demo --since 5m

Trace ID                          Status  Pipeline              Function              Start Time               End Time               
--------------------------------------------------------------------------------------------------------------------------------------
91c9a3c4002598fb6c5f39431cc7fac8  OK      otel-demo             otel-demo             2026-10-08 14:16:44.275  2026-10-08 14:16:44.277
00005005000000176b3fa2c25502414f  OK      otel-demo             otel-demo             2026-10-08 14:17:12.446  2026-10-08 14:17:12.453

The first trace is the instance's init; the second is the event. The event's trace ID, 00005005000000176b3fa2c25502414f, is the one the handler read from trace.get_current_span(), and the one its own span external-span belongs to. The value from event.get_hex_trace_id(), 00000000000000000000500500000017, isn't: see get the trace ID.

DataEngine's spans for the event, from the trace-tree endpoint of the DataEngine telemetry API (see orchestrators for authentication). Set VMS_URL to https://<VMS address>, and VAST_TENANT, VAST_USERNAME and VAST_PASSWORD to your tenant and DataEngine user. -k skips the certificate check, for a lab VMS with a self-signed certificate; leave it out if your VMS certificate is trusted:

curl -sk -u "$VAST_USERNAME:$VAST_PASSWORD" -H "X-Tenant-Name: $VAST_TENANT" "$VMS_URL/api/dataengine/telemetries/trace-tree?trace_id=00005005000000176b3fa2c25502414f" | jq -c '.. | objects | select(has("span_name")) | {span_name, span_id, parent_span_id, scope}'

{"span_name":"vastfunction-execution","span_id":"9a205f3d216ee443","parent_span_id":"0d6cfa7e25208732","scope":"vast-runtime"}
{"span_name":"ctx-tracer-span","span_id":"082d57040126582f","parent_span_id":"9a205f3d216ee443","scope":"user"}

DataEngine has the runtime's span for the handler call and the ctx.tracer span, but not external-span. What the Collector received:

docker logs otel-collector 2>&1 | grep -E "^(Span|LogRecord) #|Trace ID|Parent ID|^    ID|Name  |service.name|Body|InstrumentationScope"

     -> service.name: Str(otel-demo)
InstrumentationScope user 
LogRecord #0
Body: Str(otel-demo ready, logger user)
Trace ID: 
     -> service.name: Str(otel-demo)
InstrumentationScope otel-demo 
Span #0
    Trace ID       : 00005005000000176b3fa2c25502414f
    Parent ID      : 9a205f3d216ee443
    ID             : 3e4b75affef6456f
    Name           : external-span
     -> service.name: Str(otel-demo)
InstrumentationScope user 
LogRecord #0
Body: Str(event 87982405058583 DataEngine trace 00000000000000000000500500000017, current span trace 00005005000000176b3fa2c25502414f)
Trace ID: 00005005000000176b3fa2c25502414f
LogRecord #1
Body: Str(inside external-span trace 00005005000000176b3fa2c25502414f span 3e4b75affef6456f)
Trace ID: 00005005000000176b3fa2c25502414f
LogRecord #2
Body: Str(inside ctx.tracer span)
Trace ID: 00005005000000176b3fa2c25502414f

And nothing from DataEngine's runtime, which reports itself as service.name vast-function:

docker logs otel-collector 2>&1 | grep -c "service.name: Str(vast-function)"

0

So, in your backend:

  • external-span is in DataEngine's trace, 00005005000000176b3fa2c25502414f, and its parent is 9a205f3d216ee443, DataEngine's vastfunction-execution span. That parent exists only in DataEngine, so your backend shows external-span with a missing parent. Search either system for the trace ID to find the same event.
  • The log records the handler wrote during the event carry the event's trace ID. The init record has none: init runs before any event.
  • The ctx.tracer span isn't there. Create spans with your own tracer for anything you want to see in your backend, and with ctx.tracer for anything you want in DataEngine's trace views; a span created with one isn't sent to the other.

Get the trace ID

To log the trace ID, or pass it to another system (for example as an OpenLineage run ID, as in OpenLineage and OpenMetadata), read it from the current span inside the handler:

from opentelemetry import trace

def handler(ctx, event):
    trace_id = trace.get_current_span().get_span_context().trace_id   # an int
    ctx.logger.info("trace %032x", trace_id)                          # 32 hex digits, as vastde traces list shows it

In VAST Cluster 5.5.0-SP6, event.get_hex_trace_id() called without an argument doesn't return the event's trace ID: in the run above it returned 00000000000000000000500500000017 for the trace 00005005000000176b3fa2c25502414f. Use the current span instead.

Clean up

Delete the pipeline, the trigger and the function (vastde pipelines delete otel-demo, vastde triggers delete otel-uploads, vastde functions delete otel-demo), and stop the test Collector with docker rm -f otel-collector.

See also

Related articles