Applies to VAST Cluster 5.5.
Four clients work on the same VAST DataBase tables: the VAST Query Engine (SQL over ADBC, inside the cluster), Trino and Spark (with the VAST connectors), and the Python SDK. This page compares what each can do in VAST Cluster 5.5. Each cell links to the page that documents it. "Not covered" means no page in these docs covers it.
| Capability | VAST Query Engine | Trino | Spark | Python SDK |
|---|---|---|---|---|
| Interface | SQL over ADBC, run in the cluster (ADBC) | SQL, through the VAST connector (Trino) | Spark SQL and DataFrames, through the VAST connector (Spark) | Python API with Arrow results (Python SDK) |
| Filters pushed down to VAST DataBase | Comparisons with constants, ranges, IS NULL, OR on one column, consecutive IN lists. Functions, wildcard LIKE and OR across columns run in the engine (SELECT) |
Per-column filters, IN, LIKE '%substring%'; OR across columns with a session property (Trino) |
Per-column filters, IN, IS NULL, OR on one column; no LIKE (Spark) |
An AND of one-column OR groups; anything else raises NotImplementedError (pushdowns) |
| Aggregation | Without GROUP BY only (SELECT) |
With GROUP BY (ingest) |
With GROUP BY (ingest) |
None: aggregate the Arrow result (querying) |
ORDER BY |
Only with LIMIT (SELECT) |
Yes (Trino) | Yes (Spark) | Not covered |
| Joins | No (not supported) | Yes, with dynamic filtering into VAST DataBase (pushdowns) | Not covered | No join API (SDK reference) |
UPDATE, DELETE |
Yes (UPDATE, DELETE) | Yes, and MERGE (updates and deletes) |
Yes (updates and deletes) | update() and delete() by row ID (updates and deletes) |
| Transactions | Per statement, or autocommit=False with commit() and rollback(); SQL BEGIN refused (SQL reference) |
START TRANSACTION (transactions) |
ndb.create_tx() (transactions) |
session.transaction() (transactions) |
| Sorted tables | Create (SORT KEY), add a sort key, use (CREATE TABLE) |
Make a table sorted (sorted_by), use (Trino) |
Whether Spark can enable sorting is unconfirmed (Spark) | add_sorting_key() (SDK reference) |
| Partitioned tables | No: every statement is refused (not supported) | Yes (Trino) | Yes (Spark) | Yes (SDK reference) |
| Vector index search | Yes (vector search SQL) | No (Trino) | No (Spark) | vector_search(), which runs through the Query Engine (SDK reference) |
| Creating a vector index | VECTOR CLUSTER BY (vector search SQL) |
No vector type (data types) | No vector type (data types) | VectorIndexSpec (SDK reference) |
| Parquet import | INSERT into "<table> vast.import_data" (import Parquet files) |
INSERT into "<table> vast.import_data()" (ingest) |
INSERT into `<table> vast.import_data()` (ingest) |
import_files() (ingest) |
CREATE TABLE ... AS SELECT |
No; CREATE TABLE, then INSERT ... SELECT (CREATE TABLE) |
Yes (ingest) | Yes (ingest) | util.create_table_from_files() creates a table from Parquet files (ingest) |
| Views | No (not supported) | Yes, with limits (Trino) | Yes (Spark) | Not covered |
| Reading snapshots | "<bucket>/.snapshot/<snapshot>/<schema>".<table> (SELECT) |
Not covered | Not covered | bucket.snapshot() (snapshots) |
| Event Broker topic tables | Read (query topics) | Read (query topics) | Read (query topics) | Not covered |
| Row filters and column masks | The table is refused (vector search security) | Applied (Trino) | Not confirmed for 5.5 (security) | select() is refused with 403 (vector search security) |
| Column allow and deny | Applied (vector search security) | Applied (Trino) | Not covered | Not covered |
| End-user impersonation | vast.db.end_user (ADBC) |
Yes, with the catalog's security flags (Trino) | Not supported (security) | end_user=, for ADBC calls only (ADBC) |
Choosing
- The Query Engine suits SQL from your own application, aggregates over one table, top-N queries and vector search, with no engine to run. It reads one table per query.
- Trino or Spark suit joins, grouping, views, partitioned tables and large transformations.
- The Python SDK suits bulk reads and writes as Arrow data, and table structures the SQL engines can't create, such as semi-sorted projections.
Ingest, querying and updates and deletes show the same task in each.