Architecture

Applies to VAST Cluster 5.5.

This page explains what VAST DataEngine runs where, the path one event takes from an S3 upload through your functions, what vastde pipelines deploy does, and how retries, scaling and telemetry work. You don't need it to run the quickstart, but read it before you size a pipeline or debug one that's stuck or dropping events.

What runs where

VMS on the VAST cluster controls everything; your functions run on the compute cluster, fed from topics on the event broker.

Schedule triggers depend on VMS: in 5.5 they don't fire while the VMS is down, for example during an upgrade or an HA failover. Element triggers are unaffected.

The path of one event

Take a pipeline with an element trigger on bucket images, a function deployment thumbnail linked from it, and a function deployment index linked from thumbnail:

1. An object lands. A client uploads images/cat.jpg over S3.
2. The trigger publishes. The object's name passes the trigger's filters, so the cluster publishes a CloudEvent of type vastdata.com:Element.ElementCreated to the trigger's topic, with elementpath set to images/cat.jpg. Its data is an S3-style notification record for the upload. See events.
3. The compute cluster consumes. In the pipeline's namespace, a consumer reads the trigger's topic, and the link's delivery rule sends the event over HTTP to one instance of thumbnail. If no instance is free, the service scales up.
4. The handler runs. The DataEngine runtime in the container turns the request into a VastEvent and calls your handler(ctx, event). If the handler returns normally, the delivery succeeds. If it raises, the delivery fails and is retried (see retries).
5. The return value becomes the next event. The runtime sends what the handler returned as a new CloudEvent of type vastdata.com:Function, whose source is thumbnail. It keeps the incoming event's subject, ID and partition key, and carries any labels the handler set. A handler that returns None still sends an event, with no data. If no link starts at the function, the return value is discarded.
6. The next link routes it. The event travels on the link's topic (the default topic if the link names none). The delivery rule for the thumbnail → index link matches events from thumbnail that carry all of the link's labels, and sends them to index.
7. The trace follows it. Each step adds spans to the event's trace, so vastde traces get <trace-id> and the UI's trace view show the whole path.

Nothing goes back to the original writer: the S3 upload completed at step 1. Element events are asynchronous.

Each hop is a topic, a delivery with retries, and an HTTP call to a function instance.

If thumbnail is configured for batching, step 4 waits until enough events arrive or the batch timeout passes, then calls the handler once with the whole batch. Each item in the list it returns is sent on as its own event.

What deploy does

vastde pipelines deploy <pipeline> (or pipelines create --deploy) deploys the pipeline's latest revision:

1. Checks. The revision must have at least one trigger, at least one link, and at least one link from a trigger. Every secret it names must exist in the pipeline's namespace, and every function's container registry must be linked to the pipeline's cluster and namespace. If a check fails, the command fails and the pipeline stays as it was; after pipelines create --deploy, that means the pipeline exists, in Draft. Other mistakes, such as a link that names an unknown function deployment, are caught earlier, when you create or update the pipeline. See the error reference.
2. Resolves revisions. Each function deployment gets its pinned revision, or the function's default revision if it's unpinned. See which revision runs.
3. Applies it to the compute cluster. VMS creates or updates the pipeline's Kubernetes resources in its namespace, and marks this revision as the deployed one.
4. Status InProgress. The command returns once the request is accepted; Kubernetes is still pulling images and starting services, so check the status before you send events.
5. Status Ready or Error. In the background, the cluster checks the pipeline's resources on the compute cluster and updates its status.

Draft ──deploy──▶ InProgress ──▶ Ready
                       │
                       └──────▶ Error (reason)

Ready means every part of the pipeline reports ready: the function services, the consumers on the trigger topics and the delivery rules for the links. Error means at least one doesn't, and the pipeline's reason carries that part's own message. For a function whose container exits at start-up, for example because main.py imports a package that isn't installed, the reason includes the container's output with the Python traceback. The pipeline stays InProgress for several minutes before it goes to Error, and vastde logs shows nothing for that function, because it never starts. See troubleshooting. If the pipeline's resources have disappeared from the namespace, the reason says the pipeline wasn't found there.

Read the status with vastde pipelines get <pipeline>, and the reason with vastde pipelines get <pipeline> -o yaml (the default output doesn't show it). A check that fails in step 1 fails the command itself, and nothing changes on the compute cluster.

Updating a pipeline doesn't touch what's running. The deployed revision keeps running until you deploy again. Deleting a pipeline sets it to Deleting, removes its resources from the compute cluster, and then removes the pipeline.

Retries and the dead-letter topic

Each link delivers events to its destination functions over HTTP. A delivery fails when the handler raises an exception, when the call times out, or when the runtime can't send the handler's result on to the next link.

1. A failed delivery is retried, up to the link's retries setting, with a wait between attempts that grows each time. retries counts the deliveries after the first: with retries: 2, the handler ran three times. If you leave retries out, a server default applies, which can keep retrying a failing event for a long time before it's dead-lettered, so set retries explicitly on every link.
2. When the retries run out, the event goes to the tenant's dead-letter topic, with the other function, routing and dispatching failures.

Every attempt runs in the same trace, so vastde traces list shows one trace with status ERROR that lasts from the first attempt to the last. Each attempt also logs an Error in handler execution: <message> record at ERROR.

Consequences for your code:

• A handler can see the same event more than once. A retry calls the handler again, even if the first attempt did its work and only failed afterwards. Make handlers idempotent: write output under names derived from the input, use upserts, or check before you write.
• Exceptions are your failure signal. Raise when you want the event retried and, eventually, dead-lettered. Catch and log when you don't.
• Watch the dead-letter topic. It's a topic on the tenant's broker, so you can consume it with a Kafka client. Every pipeline in the tenant shares it. See the dead-lettered record.

On a 5.5 cluster, an event whose handler failed was delivered again, with the same id, on every retry, and then reached the dead-letter topic; no events were lost. Design for at-least-once delivery, and don't rely on events never being lost or never being repeated.

The dead-lettered record

A dead-lettered record is the original event, unchanged, written to the dead-letter topic as a Kafka record whose headers hold the CloudEvent attributes (ce_id, ce_type, ce_source, ce_subject, ce_time, the extensions such as ce_elementpath) and whose value is the original data. Three headers are added:

For the handler above, which raised ValueError(f"cannot process {event.object_key}"), the record for fail/broken.txt had these headers, among others:

ce_id: 88051124535306
ce_type: vastdata.com:Element.ElementCreated
ce_elementpath: docs-src/fail/broken.txt
ce_knativeerrorcode: 500
ce_knativeerrordata: Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0

ce_knativeerrordata decodes to the exception's message:

$ echo Y2Fubm90IHByb2Nlc3MgZmFpbC9icm9rZW4udHh0 | base64 -d; echo
cannot process fail/broken.txt

To replay or alert on dead letters, consume the topic with any Kafka client. The record identifies the event (ce_id, ce_source for the trigger, ce_elementpath for the object) but not the pipeline, so if several pipelines share a trigger, use ce_knativeerrordest or the trigger to tell them apart.

Ordering

A link's events_order decides ordering. With ordered, a destination processes one event per topic partition at a time, in order, so its concurrency is capped at the topic's partition count. With unordered, events are processed concurrently and can finish in any order. Events for the same object are expected to land in the same partition.

Scaling and concurrency

Each function deployment is its own Knative service, so each step of a pipeline scales on its own. You set the bounds in the pipeline YAML, under the deployment's resources:

• Instances. min_concurrency and max_concurrency are the minimum and maximum number of instances (pods). The autoscaler adds and removes instances between them, using the deployment's autoscaling metric: concurrent requests (concurrency), requests per second (rps), or CPU or memory use (resource). If you don't give one, the cluster adds a concurrency-based block when it stores the pipeline. The autoscaling settings also have a scale-to-zero delay, for deployments allowed to scale down to no instances.
• Requests per instance. max_concurrent_requests_limit caps how many requests one instance handles at once (Knative's container concurrency; 0 means no limit). If you leave it out, it's computed from num_workers and the batch size.
• Threads. A synchronous (def) handler runs in a pool of num_workers threads, so one instance runs at most that many calls at once; it refuses calls beyond that. An async def handler runs on the instance's event loop and isn't limited by the pool.
• CPU, memory, storage (min_cpu, max_cpu, min_memory, max_memory, storage) are per instance.
• Start-up. init(ctx) runs once when an instance starts, and the instance gets no events until init has finished. Put slow set-up (loading a model, opening clients) in init, not in the handler.
• Timeout. The deployment's timeout (seconds) limits how long one call can take. The runtime itself has no handler timeout. With batching, the timeout must be longer than the batch timeout.

An ordered link caps concurrency at the topic's partition count whatever these settings say. See pipeline YAML for every field and its limits.

Batching

With config.event_batching set on a function deployment (batch_max_size and batch_timeout_ms), the instance collects events and calls the handler with a list when the batch is full or the timeout passes, whichever comes first. The handler must return a list of the same length, and each item is sent downstream as its own event. A batch has its own trace, and each event in it keeps its own. See write a function.

Logs, traces and metrics

The runtime in each function instance sends telemetry over OpenTelemetry to DataEngine's telemetry collector on the compute cluster. The collector writes it into VAST DataBase tables on the VAST cluster, and VMS answers queries over them.

Telemetry is kept for a limited time, not forever. Copy anything you need to keep into your own tables.

vastde functions localrun runs your function in a local container that logs to the console only; nothing reaches the collector. See observability and build and test locally.

See also

• Concepts: the objects, revisions and the source of truth.
• Pipelines as code: resources, autoscaling, batching and links in the YAML.
• Troubleshooting: deploy errors and failing events.
• VAST Event Broker architecture: how the cluster serves the topics.