A data engineer builds and runs the pipelines that move, transform, and serve data on the VAST AI OS. They model tabular data in the VAST DataBase, load data from Parquet and other sources, and connect query engines such as Trino and Spark to run analytics directly on the cluster. They use the VAST Catalog to find and query file and object metadata, stream data through the VAST Event Broker, and automate event-driven processing with VAST DataEngine functions, triggers, and pipelines.
Common Data Engineer Tasks
Configure the cluster for database access: Prepare the views, users, and permissions needed to work with the VAST DataBase.
Manage tables: Create, modify, and delete tables in VAST DataBase schemas.
Import data to the VAST DataBase: Load Parquet files and other data into VAST tables.
Manage database permissions: Control which users and applications can read and write tabular data.
Install the VAST connector for Trino: Query VAST tables with Trino SQL.
Install the VAST connector for Spark: Read and write VAST tables from Spark jobs.
Query Engine (ADBC): Run SQL queries on VAST DataBase.
Publish events to the VAST Event Broker: Stream data with Kafka-compatible topics on the cluster.
Implement a VAST DataEngine pipeline: Chain functions together to process data automatically.
Create a DataEngine trigger: Start pipelines on a schedule or when new data arrives.
Publish S3 bucket events to the VAST Event Broker: Trigger application logic when objects are created or deleted.