Applies to VAST Cluster 5.5.
This page covers the ideas the rest of the docs rely on: how data is organised, how transactions work, how data moves between your client and the cluster, and the ways you can lay out a table. It describes what you can see and use as a client. For how the database works internally, see the architecture pages.
Databases, schemas and tables
A database holds schemas, schemas hold tables, and some structures are attached to a table rather than listed in the schema.
Database. A database is an S3 bucket on a view that has the Database protocol enabled. Your platform team creates it. You refer to it by its bucket name, which follows S3 bucket naming rules. In the Python SDK, a database is a Bucket (tx.bucket("name")). See setup prerequisites.
Schema. A schema holds tables, and schemas can be nested. Schema names follow S3 object naming rules. You can create schemas yourself if your permissions allow it.
Table. A table has a fixed set of typed columns, and you can add, drop or rename them later. Tables are created by your client or engine; you don't need a separate metastore. For column types, see data types.
Naming a table
How you name a table depends on the client:
| Client | Form | Example |
|---|---|---|
| Python SDK | Walk down from the transaction | tx.bucket("sales").schema("raw").table("orders") |
| Trino | <catalog>."<database>/<schema>".<table> |
vast."sales/raw".orders |
| VAST Query Engine | "<database>/<schema>".<table> |
"sales/raw".orders |
If schemas are nested, the schema path includes every level: "sales/raw/eu".orders.
Tables are stored like files. A table on a Database-enabled view looks like a file in the view's filesystem. Who can see or change it therefore also depends on the view's access policy, not only on database permissions. This is something your platform team controls; see security.
Transactions
Every operation runs in a transaction, and transactions are ACID. That includes creating a schema, inserting a batch of rows and running a query. In the Python SDK, the transaction is explicit:
with session.transaction() as tx: # begins a transaction
table = tx.bucket("sales").schema("raw").table("orders")
table.insert(batch)
table.delete(old_rows)
# leaving the block without an exception commits; an exception rolls back
What follows from this:
- One transaction can span several operations. Either they all commit or none do.
- Objects belong to their transaction. A table handle, or a query result that's still streaming, can't be used after its transaction ends. Read results fully inside the block. To avoid looking up the same table in every transaction, see Reusing metadata across transactions.
- One exception: deleting a whole partition isn't transactional. The Python SDK makes you opt in with
delete_partitions(..., allow_non_acid=True). See Partitions.
What other transactions can see before you commit, and what happens when two writers conflict, is covered in transactions. To build retries, see transactions in practice.
Data in and out: Arrow and pushdown
Arrow in, Arrow out. The Python SDK describes tables with Apache Arrow schemas. You write data as Arrow record batches or tables, and queries return a stream of Arrow record batches. That means you can hand data to pandas, Polars, DuckDB or anything else that reads Arrow without converting it. See Arrow, pandas and Polars.
Pushdown. When you query, you choose the columns, and you can add a filter. The client or engine sends both to the cluster, and the CNodes do the column selection and filtering next to the data. Only matching rows and the columns you asked for come back. Trino and Spark do the same with the parts of a query plan the database can handle. The more of your filter that can be pushed down, the less data has to move. See querying and pushdowns.
Where the work runs. Your client connects to an endpoint, which is a virtual IP pool on the cluster's CNodes. A large query or import can be spread over several endpoints at once. See performance.
Rows and row IDs
Every row has a row ID. You don't normally see it, but you use it to update or delete specific rows. In the Python SDK you ask for it with select(..., internal_row_id=True), which adds a $row_id column, and then pass those rows to update() or delete(). See updates and deletes.
The database allocates row IDs by default. You can instead control them yourself with user-defined row IDs: the table gets a vastdb_rowid column of type int64, values can't exceed 2^47, and you supply them on insert. Once a table uses user-defined row IDs for one insert, every insert into it must supply them. They're useful when you need to address known ranges of rows, and they rule out some other table features (see below).
Table structures
A regular table needs no layout decisions. For better query performance you can give a table one or more of these structures:
| Structure | What it does | Decided |
|---|---|---|
| Sorted table | Sorts rows by one or more sort columns, in the background, which speeds up queries that filter on those columns. | At creation, or added later. Can't be changed or removed afterwards. |
| Partitioned table | Splits rows into partitions by the values of up to 4 partition keys, so queries that filter on them skip whole partitions. You can also drop a whole partition at once. | At creation only |
| Semi-sorted projection | An extra copy of chosen columns, sorted differently, which queries use automatically when it helps. | Can be added to an existing table |
| User-defined row IDs | You control row ID allocation (see above). | At creation, or by the first insert that supplies vastdb_rowid |
| Vector index | Enables similarity search on a vector column. See the VAST Vector Search. | At creation |
These structures can't be combined freely, and some combinations also rule out replication. For example, a sorted table can't have semi-sorted projections or user-defined row IDs, and a partitioned table can't have a vector index. Limits has the full combinability matrix. Data modeling helps you choose, and table structures explains what each one costs.
Snapshots and replication
Tables are elements in the cluster's Element Store, next to files and objects, so cluster snapshots include them. You can read a table as it was at a snapshot with the same calls you use on the live table. In the Python SDK, that's tx.bucket("sales").snapshot("<name>"). Your platform team creates and schedules snapshots.
A database on a path protected by asynchronous replication is also copied to other clusters. The copies are built from periodic snapshots, so a replica is a read-only image of the database as of the last replicated snapshot. It holds only committed transactions, and it lags the source by up to the replication interval. If the platform team fails over to a replica, it becomes writable. Sorted and partitioned tables can't be replicated, and semi-sorted projections aren't copied (Overview of VAST Replication). See snapshots and replication for reading snapshots and replicas, and the cluster architecture for where replication fits.
How it works inside
These pages explain the mechanisms behind the behaviour described above:
- The cluster underneath (DASE): the Element Store that tables share with files and objects, CNodes, storage enclosures and shared storage, data reduction, and why tables never need re-sharding.
- Element store layout: how table data is physically organised.
- Write path: what happens after you commit an insert, update or delete.
- Read path and pruning: why data order and filters decide how much a query reads.
- Transactions: isolation, visibility and conflicts.
- Table structures: what sorted, partitioned, projected and vector-indexed layouts buy and cost.