---
title: "End-to-end patterns"
slug: "dev-start-end-to-end-patterns"
updated: 2026-10-30T18:18:46Z
published: 2026-10-30T18:18:46Z
canonical: "docs.vastdata.com/dev-start-end-to-end-patterns"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# End-to-end patterns

*Applies to VAST Cluster 5.5.*

Common ways to combine the services. Each pattern is an outline: every step links to the page that shows how to do it, with runnable code and its expected output. Get access to each service the pattern uses first; see [get access](/develop/docs/dev-start-get-access).

These are step-by-step recipes for combining two or three services. If you're building toward a business goal, such as an AI factory or a cyber lakehouse, start from [solutions](/develop/docs/dev-start-solutions): each solution lists the patterns that build it.

## Process new objects into a table

New objects in a bucket are turned into rows of a VAST DataBase table as they arrive. Services: VAST DataStore, VAST DataEngine, VAST Event Broker, VAST DataBase.

**Full guide:** [VAST DataBase and Vector Search from a function](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search), in VAST DataEngine.

**Complete example:** [VSS blueprint](https://github.com/vast-data/vss-blueprint), video search and analysis (reference application on GitHub, not verified against these docs).

![Objects land in a bucket on VAST DataStore. A VAST DataEngine element trigger watches the bucket and publishes a CloudEvent for each new object to a VAST Event Broker topic. A DataEngine function, wired to the trigger by a pipeline file deployed with vastde, writes one row per event to a VAST DataBase table with the Python SDK, in its own transaction. You query the table with the SDK, Trino, Spark or the VAST Query Engine. Events that keep failing go to the tenant's dead-letter topic.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-objects-to-table.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Objects land in a bucket** on the cluster, written with any S3 client ([DataStore quickstart](/develop/docs/dev-datastore-quickstart), [Python clients](/develop/docs/dev-datastore-integrations-python)).
2. **An element trigger watches the bucket.** For each object created under a prefix, it publishes a CloudEvent to a Kafka topic, by default your tenant's default topic, usually on VAST Event Broker ([triggers](/develop/docs/dev-dataengine-guides-triggers#element-triggers), [brokers and topics](/develop/docs/dev-dataengine-guides-triggers#brokers-and-topics)).
3. **A function writes the row.** Its handler uses the VAST DataBase Python SDK, with S3 keys passed as a secret, and writes a row per event in its own transaction ([VAST DataBase and Vector Search from a function](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search)).
4. **A pipeline file wires the trigger to the function,** and `vastde` creates and deploys it ([pipelines as code](/develop/docs/dev-dataengine-guides-pipelines-as-code)).
5. **Query the table** with the SDK, Trino, Spark or the VAST Query Engine ([read the table back](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search#read-the-table-back), [querying](/develop/docs/dev-database-guides-querying)).
6. **Watch for failures.** Events that still fail after their retries go to the tenant's dead-letter topic, which you read with a Kafka consumer ([dead-letter topic](/develop/docs/dev-dataengine-integrations-dead-letter-topic), [observability](/develop/docs/dev-dataengine-guides-observability)).

**Without DataEngine:** if your own application should react, ask for a bucket notification instead. The cluster publishes an event to a VAST Event Broker topic each time an object is created, tagged or deleted; you consume it with a Kafka consumer, or query the topic as a table ([S3 bucket events](/develop/docs/dev-event-broker-guides-s3-bucket-events), [query topics as tables](/develop/docs/dev-event-broker-guides-querying-topics)).

## Land files and query them with SQL

Parquet files stored on the cluster become a table you query with SQL, without the data passing through your client. Services: VAST DataStore, VAST DataBase.

**Full guide:** [Import Parquet files](/develop/docs/dev-database-guides-ingest#import-parquet-files), in VAST DataBase.

![Stage Parquet files in a bucket on VAST DataStore with any S3 client, using multipart uploads for large files. Import them into a VAST DataBase table with import_files() in the Python SDK, which has the cluster read the files. Check which files have been imported, then query the table with Trino, Spark or the VAST Query Engine.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-files-to-sql.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Stage the files in a bucket** on the same cluster with any S3 client ([stage the files](/develop/docs/dev-database-guides-ingest#stage-the-files)). For large files, use multipart uploads ([large objects and multipart uploads](/develop/docs/dev-datastore-guides-large-objects-and-multipart)). Files written over NFS to a view that also has S3 are objects in its bucket ([multiprotocol access](/develop/docs/dev-datastore-guides-multiprotocol)).
2. **Import them.** `import_files()` in the Python SDK has the cluster read the files into an existing table, or into a new table created from the files ([import Parquet files](/develop/docs/dev-database-guides-ingest#import-parquet-files)). From SQL, the VAST Query Engine does the same with `INSERT` ([import Parquet files with INSERT](/develop/docs/dev-database-reference-sql-import-parquet)).
3. **Check what's been imported** ([see which files have been imported](/develop/docs/dev-database-guides-ingest#see-which-files-have-been-imported)).
4. **Query the table** with Trino, Spark or the VAST Query Engine ([query with Trino, Spark or the VAST Query Engine](/develop/docs/dev-database-guides-querying#query-with-trino-spark-or-the-vast-query-engine)).

For JSON and CSV files, read them with PyArrow and insert the rows instead ([load JSON, CSV and Parquet files](/develop/docs/dev-database-guides-load-files)).

## Stream JSON events into a table

Kafka producers write JSON events to a topic, and each event arrives as a typed row of a VAST DataBase table, with no consumer or stream-processing job in between. Services: VAST Event Broker, VAST DataBase.

**Full guide:** [Expand JSON events into columns](/develop/docs/dev-event-broker-guides-blob-expansion), in VAST Event Broker.

![Kafka producers write JSON events to a VAST Event Broker topic; the topic is also a VAST DataBase table. A blob expansion on the topic parses each event's JSON value as it's produced and writes the fields you chose as typed columns of a VAST DataBase target table, in the same transaction as the event, with metadata columns about the event and the parse. You query the target table with the Python SDK, the VAST Query Engine, Trino or Spark. Events already in the topic before the expansion was created aren't expanded, and an event that doesn't parse still writes a row, flagged in blob_expansion_error_output.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-events-to-table.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Get a Kafka view and a topic.** Your platform team creates the view; your user also needs the blob expansion permissions ([setup](/develop/docs/dev-event-broker-operations-setup), [topics](/develop/docs/dev-event-broker-guides-topics)).
2. **Create the target table**, with a column for each JSON field you'll query, in a schema you can create tables in. It must exist before the expansion ([before you start](/develop/docs/dev-event-broker-guides-blob-expansion#before-you-start), [set up](/develop/docs/dev-event-broker-guides-blob-expansion#set-up)).
3. **Create the blob expansion** on the topic with the Python SDK. Only events produced from then on are expanded ([create the expansion](/develop/docs/dev-event-broker-guides-blob-expansion#create-the-expansion)).
4. **Produce JSON events** with any Kafka client. Each event and its row commit in one transaction, so the table is never behind the topic ([produce events](/develop/docs/dev-event-broker-guides-blob-expansion#produce-events), [producing](/develop/docs/dev-event-broker-guides-producing)).
5. **Query the target table** with the SDK, the VAST Query Engine, Trino or Spark ([query the target table](/develop/docs/dev-event-broker-guides-blob-expansion#query-the-target-table)).
6. **Watch for bad input.** An event that doesn't parse still writes a row, with `blob_expansion_error_output` set to 1 and its fields null ([what happens with bad input](/develop/docs/dev-event-broker-guides-blob-expansion#what-happens-with-bad-input)). Expand only the fields you query: parsing runs inside every produce request ([throughput and replication](/develop/docs/dev-event-broker-guides-blob-expansion#throughput-and-replication)).

**Without blob expansion:** the topic itself is a table. Read its raw `key` and `value` columns with the same clients, and parse them in your query ([query topics as tables](/develop/docs/dev-event-broker-guides-querying-topics)).

## Retrieval for RAG, with permissions

Document chunks and their embeddings go into a vector-indexed table, and each question retrieves the nearest chunks the user may read. Services: VAST DataStore, VAST Vector Search (on VAST DataBase), and optionally VAST DataEngine.

**Full guide:** [Build RAG retrieval with permissions](/develop/docs/dev-vector-search-guides-rag-retrieval), in VAST Vector Search.

**Complete example:** [Research assistant agent](https://github.com/vast-data/vast-research-assistant-agent), a RAG chat assistant (reference application on GitHub, not verified against these docs).

![Chunk and embed documents with Spark, a DataEngine pipeline or your own code, and write Parquet files with one row per chunk. Land them in an S3 bucket on VAST DataStore. Create a VAST DataBase table with a vector index on the embedding column and a column for each chunk's document path, then import the files. Retrieve through the VAST Query Engine with ADBC: a distance in ORDER BY, filters in WHERE and LIMIT k. Return only the chunks the user may read, with a row filter on the document path and the end user's name.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-rag.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Chunk and embed** your documents with Spark, a DataEngine pipeline or your own code, and write Parquet files with one row per chunk ([the pipeline in production](/develop/docs/dev-vector-search-guides-rag-retrieval#the-pipeline-in-production), [from a function](/develop/docs/dev-dataengine-integrations-vast-database-and-vector-search#write-embeddings)).
2. **Land the files** in an S3 bucket on the cluster.
3. **Create the table** with a vector index on the embedding column, and the column that tags each chunk with its document's S3 path if you'll filter by permissions. Both are decided at creation ([design a vector table](/develop/docs/dev-vector-search-guides-table-design), [create an index](/develop/docs/dev-vector-search-guides-create-an-index)).
4. **Import the files** with the Python SDK ([ingest vectors](/develop/docs/dev-vector-search-guides-ingest#import-parquet-files)).
5. **Retrieve** through the VAST Query Engine with ADBC: a distance in `ORDER BY`, your filters in `WHERE`, and `LIMIT k` ([query design](/develop/docs/dev-vector-search-guides-query-design), [filters](/develop/docs/dev-vector-search-guides-filters)).
6. **Return only chunks the user may read,** with the row filter by S3 path, and the end user's name when one service account searches for many users ([worked example](/develop/docs/dev-vector-search-guides-rag-retrieval#worked-example-keep-restricted-documents-out-of-retrieval), [security](/develop/docs/dev-vector-search-operations-security)).

## Find and audit data

Reports on what's stored where, what changed and who touched it, from two system tables. Services: VAST Catalog (with the audit log), VAST DataBase clients.

**Full guide:** the [VAST Catalog](/develop/docs/dev-catalog-overview) section.

![Your platform team enables the VAST Catalog and saving the audit log to VAST DataBase; both are off by default. Then find files and objects by age, size, owner, path, tag or metadata, report capacity, detect deleted, added and moved files between two Catalog snapshots, and find out who did what from the audit log. Build dashboards and scheduled reports through Trino, or join with your own tables in Spark.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-find-and-audit.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Get the tables enabled.** The Catalog and saving the audit log to VAST DataBase are both off by default ([access](/develop/docs/dev-catalog-operations-access)).
2. **Find files and objects** by age, size, owner, extension or path ([find files](/develop/docs/dev-catalog-guides-find-files)), or by S3 tag and metadata ([tags and metadata](/develop/docs/dev-catalog-guides-tags-and-metadata)).
3. **Report capacity** by owner, extension or directory ([capacity reports](/develop/docs/dev-catalog-guides-capacity-reports)).
4. **Detect changes** between two Catalog snapshots: deleted, added and moved files ([snapshots and change detection](/develop/docs/dev-catalog-guides-snapshots-and-change-detection)).
5. **Find out who did what** to a path, from which client and whether it succeeded ([query the audit log](/develop/docs/dev-catalog-guides-audit-log)).
6. **Build dashboards and scheduled reports** through Trino, or join with your own tables in Spark ([Trino and Spark](/develop/docs/dev-catalog-integrations-trino-and-spark), [Grafana](/develop/docs/dev-catalog-integrations-grafana)).

## Event sourcing and CQRS

Many writers append events to a log table; one materializer folds them into a current-state table that readers query. Services: VAST DataBase, and optionally VAST Event Broker.

**Full guide:** [Event sourcing and CQRS](/develop/docs/dev-database-guides-event-sourcing-and-cqrs), in VAST DataBase.

![Commands and events enter, optionally through a VAST Event Broker topic. Writers only append to the events table: event_id, aggregate key, event type, event time, append time and payload, sorted on the aggregate key. One materializer reads the events appended after its watermark, folds them into the current-state table and advances the watermark, all in one transaction. Readers query the current-state table for the latest state, or the events table for one aggregate's history. Replaying the whole log rebuilds the current state.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/event-sourcing-cqrs.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Decide the two tables:** an append-only event table sorted on the aggregate key, and a current-state table with one row per aggregate ([the shape](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#the-shape), [sorted tables](/develop/docs/dev-database-guides-data-modeling#sorted-tables)).
2. **Bring the events in** from your services, or from a VAST Event Broker topic with a consumer that commits its offsets after each append ([Event Broker ingress](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#optional-event-broker-ingress), [consume in a group](/develop/docs/dev-event-broker-guides-consuming#consume-in-a-group)).
3. **Append only.** New rows don't conflict, so many writers can append at once; give each event an ID so duplicates can be dropped later ([append events from two writers](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#append-events-from-two-writers), [when two transactions conflict](/develop/docs/dev-database-concepts-architecture-transactions#when-two-transactions-conflict)).
4. **Run one materializer** that folds new events, replaces current-state rows and advances its watermark in one transaction ([run a materializer pass](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#run-a-materializer-pass), [transactions in practice](/develop/docs/dev-database-guides-transactions-in-practice)).
5. **Read the current state or the history,** and rebuild by replaying the log ([query the current state and the history](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#query-the-current-state-and-the-history), [replay](/develop/docs/dev-database-guides-event-sourcing-and-cqrs#replay-to-rebuild-the-current-state)).

## Provision and deploy from CI

Infrastructure in Git: CI creates the database, its access and its topics, then deploys a DataEngine pipeline. Services: all, through the automation tools.

**Full guide:** [Automation tools](/develop/docs/dev-database-reference-automation-tools) and each section's CI/CD and automation page, starting with [VAST DataBase](/develop/docs/dev-database-operations-ci-cd).

![Pick a tool per object: Terraform where it has resources, vastpy or the REST API where it doesn't, and vastde for DataEngine. Provision the view, view policy, identity policy, user and S3 key; create schemas and tables from a pipeline step; create Event Broker topics and keep their definitions in Git; build and deploy the DataEngine pipeline with vastde from GitHub Actions or GitLab CI; then check from the data plane that a client can use the objects.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/pat-ci.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A23Z&se=2026-11-08T09%3A55%3A23Z&sr=c&sp=r&sig=o%2B4ikpnJDeDU3tlQ%2FEeyKoyyZbr5%2FsjG2DscIl3Pa6c%3D)

1. **Pick the tool for each object.** Terraform where it has resources, `vastpy` or the REST API where it doesn't, and `vastde` for DataEngine ([automation tools](/develop/docs/dev-database-reference-automation-tools#which-tool-fits)).
2. **Provision the database and its access:** the view, view policy, identity policy, user and S3 key, with Terraform or a re-runnable `vastpy` script ([Terraform](/develop/docs/dev-database-operations-ci-cd#terraform), [vastpy](/develop/docs/dev-database-operations-ci-cd#vastpy)).
3. **Create schemas and tables** from a pipeline step ([schemas and tables from a pipeline step](/develop/docs/dev-database-operations-ci-cd#schemas-and-tables-from-a-pipeline-step)). Vector-indexed tables are created by the application with the SDK, or over REST ([Vector Search CI/CD](/develop/docs/dev-vector-search-operations-ci-cd)).
4. **Create topics** and blob expansions, and keep their definitions in Git ([Event Broker CI/CD](/develop/docs/dev-event-broker-operations-ci-cd)).
5. **Build and deploy the pipeline** with a `vastde` deploy script in GitHub Actions or GitLab CI ([DataEngine CI/CD](/develop/docs/dev-dataengine-operations-ci-cd)).
6. **Check from the data plane** that the objects work for a client ([check from the data plane](/develop/docs/dev-database-operations-ci-cd#check-from-the-data-plane), [smoke-test the topic](/develop/docs/dev-event-broker-operations-ci-cd#smoke-test-the-topic)).