Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

The VAST DataEngine

Prev Next

VAST DataEngine is the programmable compute fabric for real-time operations. With serverless triggers, functions, and in-place processing, it automates pipelines and delivers continuous insights at scale.

VAST DataEngine

  • Serverless Functions

    • Run lightweight Python functions directly on VAST Native Compute directly in VAST CNodes. Automatically triggered by events, they build enrichment, transformation, and routing workflows, eliminating ETL and external schedulers while enabling complex data and AI pipelines in real time.

  • Event Triggers

    • The VAST Event Broker, a Kafka-compatible engine, detects file and object events in real time. These events trigger downstream workflows instantly, turning raw data changes into automated, intelligent actions across your pipelines.

  • Managed Trino & Spark

    • VAST runs Trino and Spark natively as managed services on compute nodes (CNodes) within the platform. With direct NVMe access to all data, they deliver faster, simpler performance—no external clusters, data movement, or duplication required.

  • VAST Native SQL + Vector Query Engine

    • Run analytics directly where data lives. The VAST Native Query Engine executes aggregations and statistical functions in-place, eliminating data movement and external pipelines. This simplifies architecture and delivers faster insights on live data.

  • In-Place Processing

    • The VAST DataEngine brings serverless compute to your data. It enables real-time processing and event-driven automation—enriching, transforming, and routing data instantly, without replication. This accelerates your pipelines while guaranteeing data freshness and simplicity.

  • Confidential AI with DataEnclave

    • Run foundation AI on sensitive data without exposing either. VAST DataEnclave uses hardware-isolated execution and cryptographic attestation to protect enterprise data and model IP while AI actively processes it.

  • Event-Driven Workflows

    • The VAST Event Broker detects data changes the moment they occur. Triggers fire instantly, launching workflows that automate ingest, enrichment, and retrieval—keeping pipelines and intelligence real-time.

  • All Workflows, One Platform

    • Automate workflows across streaming, batch, SQL, and vector pipelines. DataEngine integrates them natively, so workflows stay synchronized and consistent without external orchestration.

  • Unified Control Plane

    • Manage pipelines, triggers, and functions from a single system. Monitor job execution, track metrics, and enforce policies across every workflow with built-in observability.

  • Low Code or Code-First Design

    • Build AI pipelines fast with our low-code, drag-and-drop interface. Developers can use custom Python for full control. VAST supports both code-first and low-code workflows, offering total flexibility and complete pipeline tracking and visibility in the DataEngine interface.

  • Disaggregated Shared Everything

    • VAST’s DASE architecture decouples compute from storage while maintaining high-speed NVMe-oF access to all data. This ensures elastic scaling, massive parallelism, and predictable high performance.

  • Exabyte Scale

    • DataEngine scales seamlessly as data grows. Built on the VAST DASE Architecture, its linearly scaling, shared-everything design eliminates rebalancing and delivers consistent low-latency performance at exabyte scale.

  • Near-Zero Latency

    • Built on VAST’s DASE architecture, DataEngine gives every compute node (CNode) direct NVMe-oF access to all data nodes (DNodes) in the global dataset. The result is near-zero latency and the responsiveness modern data pipelines require.

  • Fault Tolerance

    • Running on the resilient DASE architecture, VAST delivers  zero data loss and continuous availability. Pipelines are uninterrupted, and workloads remain fully trusted.

  • Confidential Compute

    • Run AI workloads in hardware-isolated environments that protect sensitive data and proprietary models across CPU and accelerated GPU execution, extending VAST data protection from data at rest and in flight to data and models in action.

  • Verify Before Decrypt

    • Cryptographically verify hardware, software, and workload policy before granting access to protected assets. If the environment fails attestation, keys and secrets remain unavailable and confidential workloads cannot run.

  • Independent Key Control

    • Keep enterprise data and proprietary model IP under their respective owners' control. DataEnclave integrates with external key management services so each party determines when its protected assets can become accessible.

  • Auditable by Design

    • Maintain a tamper-proof, queryable record of attestation, key releases, policy decisions, and workload activity, giving security and compliance teams evidence of what ran and under which approved policy without exposing protected assets.

  • Deploy Anywhere

    • Bring confidential AI to connected, private, sovereign, edge, and fully air-gapped environments. Support remote or on-premises attestation and key brokering while maintaining the same verify-before-decrypt security model.

Related articles