End-to-end patterns

Applies to VAST Cluster 5.5.

Common ways to combine the services. Each pattern is an outline: every step links to the page that shows how to do it, with runnable code and its expected output. Get access to each service the pattern uses first; see get access.

These are step-by-step recipes for combining two or three services. If you're building toward a business goal, such as an AI factory or a cyber lakehouse, start from solutions: each solution lists the patterns that build it.

Process new objects into a table

New objects in a bucket are turned into rows of a VAST DataBase table as they arrive. Services: VAST DataStore, VAST DataEngine, VAST Event Broker, VAST DataBase.

Full guide: VAST DataBase and Vector Search from a function, in VAST DataEngine.

Complete example: VSS blueprint, video search and analysis (reference application on GitHub, not verified against these docs).

1. Objects land in a bucket on the cluster, written with any S3 client (DataStore quickstart, Python clients).
2. An element trigger watches the bucket. For each object created under a prefix, it publishes a CloudEvent to a Kafka topic, by default your tenant's default topic, usually on VAST Event Broker (triggers, brokers and topics).
3. A function writes the row. Its handler uses the VAST DataBase Python SDK, with S3 keys passed as a secret, and writes a row per event in its own transaction (VAST DataBase and Vector Search from a function).
4. A pipeline file wires the trigger to the function, and vastde creates and deploys it (pipelines as code).
5. Query the table with the SDK, Trino, Spark or the VAST Query Engine (read the table back, querying).
6. Watch for failures. Events that still fail after their retries go to the tenant's dead-letter topic, which you read with a Kafka consumer (dead-letter topic, observability).

Without DataEngine: if your own application should react, ask for a bucket notification instead. The cluster publishes an event to a VAST Event Broker topic each time an object is created, tagged or deleted; you consume it with a Kafka consumer, or query the topic as a table (S3 bucket events, query topics as tables).

Land files and query them with SQL

Parquet files stored on the cluster become a table you query with SQL, without the data passing through your client. Services: VAST DataStore, VAST DataBase.

Full guide: Import Parquet files, in VAST DataBase.

1. Stage the files in a bucket on the same cluster with any S3 client (stage the files). For large files, use multipart uploads (large objects and multipart uploads). Files written over NFS to a view that also has S3 are objects in its bucket (multiprotocol access).
2. Import them. import_files() in the Python SDK has the cluster read the files into an existing table, or into a new table created from the files (import Parquet files). From SQL, the VAST Query Engine does the same with INSERT (import Parquet files with INSERT).
3. Check what's been imported (see which files have been imported).
4. Query the table with Trino, Spark or the VAST Query Engine (query with Trino, Spark or the VAST Query Engine).

For JSON and CSV files, read them with PyArrow and insert the rows instead (load JSON, CSV and Parquet files).

Stream JSON events into a table

Kafka producers write JSON events to a topic, and each event arrives as a typed row of a VAST DataBase table, with no consumer or stream-processing job in between. Services: VAST Event Broker, VAST DataBase.

Full guide: Expand JSON events into columns, in VAST Event Broker.

1. Get a Kafka view and a topic. Your platform team creates the view; your user also needs the blob expansion permissions (setup, topics).
2. Create the target table, with a column for each JSON field you'll query, in a schema you can create tables in. It must exist before the expansion (before you start, set up).
3. Create the blob expansion on the topic with the Python SDK. Only events produced from then on are expanded (create the expansion).
4. Produce JSON events with any Kafka client. Each event and its row commit in one transaction, so the table is never behind the topic (produce events, producing).
5. Query the target table with the SDK, the VAST Query Engine, Trino or Spark (query the target table).
6. Watch for bad input. An event that doesn't parse still writes a row, with blob_expansion_error_output set to 1 and its fields null (what happens with bad input). Expand only the fields you query: parsing runs inside every produce request (throughput and replication).

Without blob expansion: the topic itself is a table. Read its raw key and value columns with the same clients, and parse them in your query (query topics as tables).

Retrieval for RAG, with permissions

Document chunks and their embeddings go into a vector-indexed table, and each question retrieves the nearest chunks the user may read. Services: VAST DataStore, VAST Vector Search (on VAST DataBase), and optionally VAST DataEngine.

Full guide: Build RAG retrieval with permissions, in VAST Vector Search.

Complete example: Research assistant agent, a RAG chat assistant (reference application on GitHub, not verified against these docs).

1. Chunk and embed your documents with Spark, a DataEngine pipeline or your own code, and write Parquet files with one row per chunk (the pipeline in production, from a function).
2. Land the files in an S3 bucket on the cluster.
3. Create the table with a vector index on the embedding column, and the column that tags each chunk with its document's S3 path if you'll filter by permissions. Both are decided at creation (design a vector table, create an index).
4. Import the files with the Python SDK (ingest vectors).
5. Retrieve through the VAST Query Engine with ADBC: a distance in ORDER BY, your filters in WHERE, and LIMIT k (query design, filters).
6. Return only chunks the user may read, with the row filter by S3 path, and the end user's name when one service account searches for many users (worked example, security).

Find and audit data

Reports on what's stored where, what changed and who touched it, from two system tables. Services: VAST Catalog (with the audit log), VAST DataBase clients.

Full guide: the VAST Catalog section.

1. Get the tables enabled. The Catalog and saving the audit log to VAST DataBase are both off by default (access).
2. Find files and objects by age, size, owner, extension or path (find files), or by S3 tag and metadata (tags and metadata).
3. Report capacity by owner, extension or directory (capacity reports).
4. Detect changes between two Catalog snapshots: deleted, added and moved files (snapshots and change detection).
5. Find out who did what to a path, from which client and whether it succeeded (query the audit log).
6. Build dashboards and scheduled reports through Trino, or join with your own tables in Spark (Trino and Spark, Grafana).

Event sourcing and CQRS

Many writers append events to a log table; one materializer folds them into a current-state table that readers query. Services: VAST DataBase, and optionally VAST Event Broker.

Full guide: Event sourcing and CQRS, in VAST DataBase.

1. Decide the two tables: an append-only event table sorted on the aggregate key, and a current-state table with one row per aggregate (the shape, sorted tables).
2. Bring the events in from your services, or from a VAST Event Broker topic with a consumer that commits its offsets after each append (Event Broker ingress, consume in a group).
3. Append only. New rows don't conflict, so many writers can append at once; give each event an ID so duplicates can be dropped later (append events from two writers, when two transactions conflict).
4. Run one materializer that folds new events, replaces current-state rows and advances its watermark in one transaction (run a materializer pass, transactions in practice).
5. Read the current state or the history, and rebuild by replaying the log (query the current state and the history, replay).

Provision and deploy from CI

Infrastructure in Git: CI creates the database, its access and its topics, then deploys a DataEngine pipeline. Services: all, through the automation tools.

Full guide: Automation tools and each section's CI/CD and automation page, starting with VAST DataBase.

1. Pick the tool for each object. Terraform where it has resources, vastpy or the REST API where it doesn't, and vastde for DataEngine (automation tools).
2. Provision the database and its access: the view, view policy, identity policy, user and S3 key, with Terraform or a re-runnable vastpy script (Terraform, vastpy).
3. Create schemas and tables from a pipeline step (schemas and tables from a pipeline step). Vector-indexed tables are created by the application with the SDK, or over REST (Vector Search CI/CD).
4. Create topics and blob expansions, and keep their definitions in Git (Event Broker CI/CD).
5. Build and deploy the pipeline with a vastde deploy script in GitHub Actions or GitLab CI (DataEngine CI/CD).
6. Check from the data plane that the objects work for a client (check from the data plane, smoke-test the topic).