Function and trigger files

Applies to VAST Cluster 5.5.

Functions and triggers are objects you create before the pipeline that uses them. You can create each one from flags or from a YAML file. Keep the files in Git next to the pipeline file, so that the whole pipeline can be rebuilt from the repository.

-f is the short form of --from-file for both commands. Files can be YAML or JSON. When you give both a file and flags, the flags win.

This page uses one running example: an element trigger, uploads-created, on the source bucket media-uploads, feeding the functions thumbnailer and indexer. The pipeline that connects them is the complete example on the pipeline YAML page.

Function file

# functions/thumbnailer.yaml
name: thumbnailer
description: Writes a thumbnail for each new image
tags: [media]
container_registry_vrn: team-registry            # registry name or VRN
artifact_source: media/thumbnailer               # path of the image in the registry
image_tag: "1.3.0"
architecture: x86
is_published: true
revision_alias: thumbs-1-3
revision_description: Smaller JPEG output

• The CLI requires name, the registry, artifact_source and image_tag on create.
• The server checks that it can reach the image when the revision is created. The API's skip_image_validation turns this off for registries whose credentials only the compute cluster holds; the CLI has no flag or key for it.
• The function file has no handler or entry point. The handler is chosen when you build the image (vastde functions build --handlers). See build and test locally.
• The API's runtime field (Python version) has no CLI flag or file key. The Python version is fixed when you build the image.
• Keys the CLI doesn't know are ignored without a warning, so check spelling.

Revisions

A function has numbered revisions. Each revision records one image (artifact_source and image_tag) plus its registry, architecture and publish state.

• functions create makes revision 1.
• functions update makes a new revision when you change the image, registry, architecture, artifact type, publish state, tags or alias. Changing only the description updates the function without a new revision.
• A new revision doesn't change a running pipeline. A function deployment that pins a revision keeps it. One that doesn't pin picks up the function's default revision, or its latest revision if no default is set, the next time the pipeline is deployed. See pinning a revision.
• --publish publishes the new revision. It doesn't make it the default revision; set --default-revision-number for that.
• vastde functions get <name> -r lists the revisions. functions get without -r doesn't show artifact_source or image_tag, because they belong to revisions, so its output isn't a function file you can re-apply.

Publish every revision a pipeline will run.

To release new code: build and push a new image tag, change image_tag in the file, run vastde functions update thumbnailer -f functions/thumbnailer.yaml, then change revision in the pipeline file if the deployment is pinned, and update and deploy the pipeline.

Element trigger file

An element trigger turns changes to objects or files in a source view into events.

# triggers/uploads-created.yaml
name: uploads-created
description: New JPEGs in media-uploads
tags: [media]
source_types: [S3]
config:
  events: ["ObjectCreated:*"]
  source_bucket_name: media-uploads
  name_filters:
    prefixes: ["incoming/"]
    suffixes: [".jpg"]
broker:
  type: Internal
  name: media-broker               # the VAST Event Broker view's bucket name
topic_name: media-uploads-events
custom_extensions:
  team: media

The file has the shape of the API's element trigger.

Each flag on the command line replaces the matching key from the file. A filter flag sets a one-item list.

Events

Quote the names in YAML ("ObjectCreated:*"); in a shell, quote them too. The events a trigger delivers use the Web UI names: with ObjectCreated:*, the event's type is vastdata.com:Element.ElementCreated and event.subtype is ElementCreated. See events.

Source types

The source view must be set up for the source type you choose, which is a platform-team task. For what NFS triggers do and don't report, see triggers.

Tag filters

Tag filters work only on a trigger whose only event is ObjectTagging:Put. Any other combination is rejected with Tag filters can only be used with ObjectTagging:Put by itself.

Broker and topic

• Give broker and topic_name together, or neither. The CLI rejects one without the other.
• With neither, the trigger publishes to your tenant's default topic on the tenant's default broker.
• If the topic doesn't exist on the broker, it's created. The API describes topic_name as "used with broker field for automatic topic creation".
• The dead-letter topic can't be a trigger's topic.
• The trigger's topic is the topic a pipeline link from this trigger uses. Its VRN is vast:dataengine:topics:<broker>/<topic>, here vast:dataengine:topics:media-broker/media-uploads-events. vastde triggers create prints it as Topic:. See topics on links.

For creating and sizing topics, see topics in the VAST Event Broker section.

Schedule trigger file

A schedule trigger produces an event on a schedule.

# triggers/nightly-reindex.yaml
name: nightly-reindex
description: Rebuild the media index every night
config:
  cron_schedule: "0 2 * * *"       # 02:00 every day
broker:
  type: Internal
  name: media-broker
topic_name: media-schedule-events
custom_extensions:
  job: reindex

cron_schedule accepts two syntaxes. The server tries the first, then the second:

A 6-field expression doesn't need Quartz's ?: 0 */5 * * * * fires every 5 minutes, on the minute. The Web UI's Advanced schedule takes Quartz syntax. Use the 5-field form in files unless you need seconds or a year.

An expression that neither syntax accepts is rejected when you create the trigger. The message reports the 5-field parser's error, so it says expected 5 even though 6 and 7 fields are accepted:

400 Bad Request: Invalid cron schedule: Wrong number of fields; got 3, expected 5

400 Bad Request: Invalid cron schedule: Error validating expression '61': the last value (61) is higher than the maximum value (59)

The first is from --cron-schedule "every 5 minutes", the second from --cron-schedule "61 * * * *". See the error reference.

Schedule triggers don't fire while the cluster's management service (VMS) is down, for example during an upgrade or failover. Element triggers aren't affected.

The generic triggers create form

vastde triggers create --type Element|Schedule --from-file <file> also exists, and the CLI's shipped sample files are written for it. It reads a flatter file (type, topic_name, source_bucket_name at the top level) and its flags differ (--topic instead of --topic-name). In 5.5 it reads source_types from the file but doesn't send it. Use triggers create element and triggers create schedule with the file shape on this page.

Updating a trigger

vastde triggers update <name|GUID> changes a trigger in place. It takes the same flags as create, except --name (you can't rename a trigger) and --topic in place of --topic-name. It reads the trigger, applies your changes and writes the whole trigger back.

triggers update --from-file expects a flatter file than create (topic_name, source_bucket_name, events, name_prefix and so on, at the top level), and doesn't read custom_extensions from it. Until that's resolved, use flags for updates, or delete and recreate the trigger from its file.

There's no version check on triggers or functions: the last update wins.

See also

• Pipeline YAML
• Publish functions
• Triggers
• Events
• Coming from the Web UI
• CLI reference
• Creating a Trigger and Creating a DataEngine Function on the KB
• Create a new Function, Create a new Element Trigger and Create a new Schedule Trigger in the VAST DataEngine API reference