Pipeline YAML

Applies to VAST Cluster 5.5.

A pipeline is defined by one file: the file that vastde pipelines create --config and vastde pipelines update --config take. It names the pipeline, says which compute cluster and namespace it runs in, and holds the manifest: the triggers, the function deployments, and the links that connect them. Keep this file in Git. It's the source of truth for the pipeline.

This page lists every field in the file, what the server fills in, what you can't change after creation, what a deploy needs, and how updates and revisions behave. For a walkthrough, see pipelines as code. For the trigger and function files, see function and trigger files.

The file at a glance

name: media-pipeline                     # required
description: Thumbnails and index entries for new images
tags: [media, dev]
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:compute-1   # required
namespace: dataengine-ns-<tenant-guid>  # required: your tenant's DataEngine namespace
manifest:
  config:                                # applies to every function deployment
    environment_variables: []
    secrets: []
  triggers: []                           # triggers this pipeline listens to
  function_deployments: []               # functions this pipeline runs, and how
  links: []                              # who invokes whom

The file is YAML or JSON. Keys are the API's field names, in snake_case, except the two keys of a conditional trigger label (triggerLabelName, triggerLabelValue). --config takes a file path, a path prefixed with @, or inline JSON (a value starting with { or [).

A complete example is at the end of the page.

Top-level fields

The API documents kubernetes_cluster_vrn and namespace as optional; the server requires both on create.

On pipelines create, the --name, --description and --tags flags override the values in the file.

manifest.config

Settings for every function deployment in the pipeline.

Environment variable names that start with VAST_, VASTPIPELINE_ or K_, and a few other names the runtime uses (HOSTNAME, PORT, PERFORMANCE), are rejected. VAST_FUNC_HANDLER is allowed.

Secrets:

• A secret is checked for existence only when you deploy, not when you create or update the pipeline.
• The same secret name can't be listed both here and in a function deployment's config.secrets. The server rejects the manifest.
• You create secrets with vastde pipelines create|update --secret-file <file> (repeatable). The file holds name, kubernetes_cluster_vrn, namespace and entries: [{key, value}]. The CLI creates or replaces the secret before it sends the pipeline, but it doesn't add the secret's name to the manifest: list it in secrets yourself. The CLI warns about a name in the manifest that it didn't create. Don't commit a secret file to Git. See security.

manifest.triggers

The triggers this pipeline listens to. A trigger is a separate object that you create first (see function and trigger files); the manifest refers to it.

  triggers:
    - name: uploads
      vrn: vast:dataengine:triggers:uploads-created

manifest.function_deployments

One entry per function the pipeline runs. A function deployment is a function plus the settings it runs with in this pipeline. The same function can appear twice under different names.

Pinning a revision

• revision: <n> runs revision n of the function on every deploy.
• revision omitted or null runs the function's default revision. If the function has no default revision set, it runs the function's latest revision. The API documents this for default_revision_number.

An unpinned deployment is resolved when you deploy, not when you save the pipeline. Every deploy picks up whatever the function's default or latest revision is at that moment. Creating a function revision doesn't change a running pipeline until the next deploy.

Pin the revision in Git for anything you promote between environments, so that a deploy runs exactly what the file says. For how revisions are created, see function and trigger files.

function_deployments[].config

event_batching:

• If you set event_batching, set both fields. The API requires them.
• With batch_max_size above 1, batch_timeout_ms must be above 0. With batch_max_size: 1, it must be 0. Any other combination is rejected.
• The function's handler must be written to take a batch. Turning batching on without a batch handler is liable to cause runtime failures. See write a function.
• Set the deployment's resources.timeout higher than the batch timeout. Duplicated batch-trace entries are a known issue in 5.5 when the two are combined badly.

function_deployments[].resources

There's no field for GPUs in 5.5.

The server rejects a deployment where:

• autoscaling is set together with either deprecated field.
• max_concurrent_requests_limit is set (and not 0) but is lower than the concurrency autoscaling target, or lower than num_workers × batch_max_size when batching is on.

resources.autoscaling

metric is required. Provide the one configuration object that matches it.

concurrency and rps take the same fields; concurrency adds hard_limit.

resource:

If a deployment has no autoscaling (and neither deprecated field), the server adds a concurrency block with target set to num_workers × batch_max_size before it stores the manifest. A deployment with no resources at all gets a complete block with the defaults above. You'll see them when you read the manifest back.

manifest.links

Links say who invokes whom. Each link connects one or more sources to one or more destinations. The API's rule: each source in source invokes every function deployment in destination, if all of the link's conditional trigger labels match.

• Every name in source and destination must be defined in triggers or function_deployments.
• You can't connect a trigger to a trigger, and you can't make a loop.

Topics on links

• A topic VRN is vast:dataengine:topics:<broker>/<topic>, where <broker> is the broker name a trigger uses (broker.name; for a VAST Event Broker, the bucket name of the broker's view).
• The dead-letter topic can't be used on a link.
• For topics themselves (partitions, retention), see topics in the VAST Event Broker section.

links[].config

The API gives no defaults for these. If you leave retries out, a server default applies, and a failing event can be retried for a long time before it reaches the dead-letter topic. Set retries on every link. See retries and the dead-letter topic.

The VAST Web UI shows Retries and Method of Delivery on the function deployment. In the file they belong to the link. See coming from the Web UI.

Conditional routing

conditional_trigger_labels makes a function-to-function link fire only for events that carry all of the listed labels with the listed values. The source function sets the labels on the events it returns; see the Runtime SDK reference.

• Labels are ANDed: an event must match every label on the link.
• Only links whose sources are function deployments can have labels. A link from a trigger with labels is rejected.
• Label names must be unique within a link.
• Underscores aren't allowed in names or values.

  links:
    - source: [thumbnailer]
      destination: [indexer]
      conditional_trigger_labels:
        - triggerLabelName: kind
          triggerLabelValue: photo

What the server fills in

You don't write these fields. vastde pipelines get <name> -o yaml shows them.

In the stored manifest, the server also:

• Fills in resources, with the defaults and an autoscaling block, where they're missing (see above).

When you read the manifest back, unpinned function deployments show revision: null, and an empty config section isn't shown.

At deploy time, without changing the stored manifest, it:

• Resolves each unpinned revision to the function's default or latest revision.
• Computes max_concurrent_requests_limit where it's not set.

Fields you can't change

To move a pipeline to another cluster or namespace, create a new pipeline from the same file with a new name.

What a deploy requires

vastde pipelines deploy <name|GUID> takes no file. It deploys the pipeline's latest revision. vastde pipelines create --config <file> --deploy creates and deploys in one step.

A deploy fails unless:

• manifest.triggers has at least one trigger.
• manifest.links has at least one link, and at least one link has a trigger as a source.
• Every secret the manifest names exists in the namespace.
• Every function's container registry is linked to the pipeline's compute cluster and namespace. Otherwise the deploy fails with Access to Container registry … is denied for the kubernetes cluster …. That's a platform-team fix; see setup prerequisites.

Create and update check the rest when you submit the file: that the cluster VRN is there and the namespace belongs to that cluster, that the triggers and functions exist, that link names resolve, that no environment variable has a reserved name, that no secret is named at both levels, and the batching, autoscaling, topic and label rules on this page.

A failed deploy check doesn't undo the create: vastde pipelines create --config <file> --deploy that fails at the deploy step leaves the pipeline in Draft. Fix the file, then run vastde pipelines update and vastde pipelines deploy, or delete the pipeline. The messages for each check are in the error reference.

Updates and revisions

vastde pipelines update <name|GUID> --config <file> replaces the pipeline's definition with the file. Each manifest is stored as a revision.

Send whole sections. The four manifest sections (config, triggers, function_deployments, links) are each replaced as a whole when they're in the file. A section you leave out is kept from the latest revision; inside config, the same applies to secrets and environment_variables separately.

The top-level description isn't kept. Unlike a manifest section, a description you leave out of the file is cleared on update, not kept.

• To remove everything in a section, send an empty list (links: []). Leaving it out doesn't remove it.
• To remove one link or one function deployment, send the section without it.
• Always include manifest, and keep the file complete. A complete file in Git gives the same result whatever the cluster holds.

Revisions.

• pipelines create makes revision 1.
• Updating doesn't redeploy. Run vastde pipelines deploy to put the new revision live.
• An update adds a revision even when the file is identical to the latest one, if that one is deployed.
• Deploy always takes the latest revision. There's no command to deploy an older revision. To roll back, check out the earlier file from Git, run update, and deploy.
• vastde pipelines get <name> --manifest shows the latest revision's manifest; add --deployed to see the deployed one.

No version check. Updates carry no version or ETag, so the last writer wins, section by section. An edit saved in the VAST Web UI and an update from an old copy of the file overwrite each other without a warning. Pick one place to edit each pipeline. For pipelines you manage from Git, treat the Web UI as read-only. See coming from the Web UI.

Complete example

One element trigger, uploads-created, watches the media-uploads bucket. Its events go to thumbnailer, which passes events on to indexer. indexer takes events in batches. The trigger and the two functions must exist before you create the pipeline; their files are in function and trigger files.

The example file as the graph it describes, with the section of the file that defines each part.

# media-pipeline.yaml
name: media-pipeline
description: Thumbnails and index entries for new images
tags: [media, dev]
kubernetes_cluster_vrn: vast:dataengine:kubernetes-clusters:compute-1
namespace: dataengine-ns-<tenant-guid>   # your tenant's DataEngine namespace on compute-1

manifest:
  config:                                 # every function deployment gets these
    environment_variables:
      - name: OUTPUT_BUCKET
        value: media-derived
    secrets:
      - media-s3-keys                     # Kubernetes Secret in the namespace; create it with --secret-file

  triggers:
    - name: uploads                       # local name, used in links below
      vrn: vast:dataengine:triggers:uploads-created

  function_deployments:
    - name: thumbnailer                   # local name, at most 40 characters
      function_vrn: vast:dataengine:functions:thumbnailer
      revision: 3                         # pinned; omit to follow the default or latest revision
      config:
        log_level: INFO
        environment_variables:
          - name: THUMBNAIL_WIDTH
            value: "320"                  # values are strings
      resources:
        min_concurrency: 1
        max_concurrency: 8
        min_cpu: 250m
        max_cpu: "1"
        min_memory: 256Mi
        max_memory: 512Mi
        timeout: 120                      # seconds
        autoscaling:
          metric: concurrency             # scale on in-flight requests per instance
          concurrency:
            target: 20
            target_utilization_percentage: 70

    - name: indexer
      function_vrn: vast:dataengine:functions:indexer
      revision: 1
      config:
        log_level: INFO
        event_batching:                   # the handler must accept a batch
          batch_max_size: 50
          batch_timeout_ms: 2000          # must be > 0 when batch_max_size > 1
      resources:
        min_concurrency: 1
        max_concurrency: 4
        num_workers: 4                    # up to 4 x 50 events in flight per instance
        timeout: 60                       # longer than the batch timeout
                                          # no autoscaling block: the server adds a concurrency default

  links:
    - source: [uploads]                   # trigger -> function: uses the trigger's topic
      destination: [thumbnailer]
      config:
        retries: 3
        events_order: unordered

    - source: [thumbnailer]               # function -> function: no topic, so the tenant default topic
      destination: [indexer]
      config:
        retries: 3
        events_order: unordered

Create it and deploy it:

vastde pipelines create --config media-pipeline.yaml --secret-file media-s3-keys.yaml --deploy

After you edit the file:

vastde pipelines update media-pipeline --config media-pipeline.yaml
vastde pipelines deploy media-pipeline

Exporting an existing pipeline

Use this to bring a pipeline that was built in the VAST Web UI, or by someone else, under Git.

From the CLI. vastde pipelines get <name> --manifest -o yaml prints the latest revision's manifest, and only the manifest: no name, kubernetes_cluster_vrn or namespace. It isn't a file pipelines update accepts as it stands. Wrap it:

1. Run vastde pipelines get <name> -o yaml and copy name, kubernetes_cluster_vrn and namespace (and description and tags if you use them). Leave out the read-only fields (id, guid, vrn, status, reason, last_revision_number, timestamps, owner).
2. Put the output of vastde pipelines get <name> --manifest -o yaml under a top-level manifest: key, indented one level.

vastde pipelines get media-pipeline -o yaml
vastde pipelines get media-pipeline --manifest -o yaml

Add --deployed to the second command to export the deployed revision rather than the latest one. Without -o yaml, --manifest prints JSON.

From the Web UI. The Visual Builder's YAML view has a Download button that saves pipeline-manifest.yaml, with name, description, tags, namespace, kubernetes_cluster_vrn and manifest: the full file.

Either way, the exported manifest isn't exactly the file that created the pipeline. It includes the resources the server filled in, with their autoscaling blocks, and revision: null on unpinned deployments, and the CLI export leaves out an empty config. You can keep the filled-in values or delete them; the server adds them again where they're missing. Delete revision: null or replace it with a revision number to pin the deployment. The wrapped CLI export is accepted by pipelines update as it is: re-applying it unchanged left the exported manifest identical. Add description (and tags) to the wrapper if the pipeline has them, or the update clears them. Check the export into Git, run vastde pipelines update <name> --config <file> once to confirm it's accepted, and from then on change the pipeline only through the file.

See also

• Pipelines as code: the workflow end to end.
• Function and trigger files
• Coming from the Web UI
• CLI reference
• Runtime SDK reference: batch handlers and trigger labels.
• Building and Deploying a Pipeline on VAST DataEngine on the KB
• Create a new Pipeline in the VAST DataEngine API reference