---
title: "Concepts"
slug: "dev-database-concepts-index"
description: "Explore how databases, schemas, and tables are organized, along with transaction mechanics and data movement in your client-cluster interactions."
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-database-concepts-index"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Concepts

*Applies to VAST Cluster 5.5.*

This page covers the ideas the rest of the docs rely on: how data is organised, how transactions work, how data moves between your client and the cluster, and the ways you can lay out a table. It describes what you can see and use as a client. For how the database works internally, see the [architecture](/develop/docs/dev-database-concepts-index#how-it-works-inside) pages.

## Databases, schemas and tables

![A database is a bucket on a Database-enabled view. It holds schemas, which can be nested, and schemas hold tables. A table has typed columns. Some structures are attached to a table rather than listed in the schema: semi-sorted projections, the imports table, and a partitioned table's partitions.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/object-model.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A54%3A54Z&se=2026-11-08T09%3A54%3A54Z&sr=c&sp=r&sig=BM1nl%2Bd8mQEXC7ttdw4EbwbWloOtj1CrWp8mEF%2B7clk%3D)

*A database holds schemas, schemas hold tables, and some structures are attached to a table rather than listed in the schema.*

**Database.** A database is an S3 bucket on a view that has the Database protocol enabled. Your platform team creates it. You refer to it by its bucket name, which follows S3 bucket naming rules. In the Python SDK, a database is a `Bucket` (`tx.bucket("name")`). See [setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites).

**Schema.** A schema holds tables, and schemas can be nested. Schema names follow S3 object naming rules. You can create schemas yourself if your permissions allow it.

**Table.** A table has a fixed set of typed columns, and you can add, drop or rename them later. Tables are created by your client or engine; you don't need a separate metastore. For column types, see [data types](/develop/docs/dev-database-reference-data-types).

### Naming a table

How you name a table depends on the client:

| Client | Form | Example |
| --- | --- | --- |
| Python SDK | Walk down from the transaction | `tx.bucket("sales").schema("raw").table("orders")` |
| Trino | `<catalog>."<database>/<schema>".<table>` | `vast."sales/raw".orders` |
| VAST Query Engine | `"<database>/<schema>".<table>` | `"sales/raw".orders` |

If schemas are nested, the schema path includes every level: `"sales/raw/eu".orders`.

**Tables are stored like files.** A table on a Database-enabled view looks like a file in the view's filesystem. Who can see or change it therefore also depends on the view's access policy, not only on database permissions. This is something your platform team controls; see [security](/develop/docs/dev-database-operations-security).

## Transactions

**Every operation runs in a transaction**, and transactions are ACID. That includes creating a schema, inserting a batch of rows and running a query. In the Python SDK, the transaction is explicit:

```
with session.transaction() as tx:      # begins a transaction
    table = tx.bucket("sales").schema("raw").table("orders")
    table.insert(batch)
    table.delete(old_rows)
# leaving the block without an exception commits; an exception rolls back
```

What follows from this:

- **One transaction can span several operations.** Either they all commit or none do.
- **Objects belong to their transaction.** A table handle, or a query result that's still streaming, can't be used after its transaction ends. Read results fully inside the block. To avoid looking up the same table in every transaction, see [Reusing metadata across transactions](/develop/docs/dev-database-reference-sdk#reusing-metadata-across-transactions).
- **One exception: deleting a whole partition isn't transactional.** The Python SDK makes you opt in with `delete_partitions(..., allow_non_acid=True)`. See [Partitions](/develop/docs/dev-database-reference-sdk#partitions).

What other transactions can see before you commit, and what happens when two writers conflict, is covered in [transactions](/develop/docs/dev-database-concepts-architecture-transactions). To build retries, see [transactions in practice](/develop/docs/dev-database-guides-transactions-in-practice).

## Data in and out: Arrow and pushdown

**Arrow in, Arrow out.** The Python SDK describes tables with [Apache Arrow](https://arrow.apache.org) schemas. You write data as Arrow record batches or tables, and queries return a stream of Arrow record batches. That means you can hand data to pandas, Polars, DuckDB or anything else that reads Arrow without converting it. See [Arrow, pandas and Polars](/develop/docs/dev-database-integrations-arrow-pandas-polars).

**Pushdown.** When you query, you choose the columns, and you can add a filter. The client or engine sends both to the cluster, and the CNodes do the column selection and filtering next to the data. Only matching rows and the columns you asked for come back. Trino and Spark do the same with the parts of a query plan the database can handle. The more of your filter that can be pushed down, the less data has to move. See [querying](/develop/docs/dev-database-guides-querying) and [pushdowns](/develop/docs/dev-database-reference-pushdowns).

**Where the work runs.** Your client connects to an endpoint, which is a virtual IP pool on the cluster's CNodes. A large query or import can be spread over several endpoints at once. See [performance](/develop/docs/dev-database-operations-performance).

## Rows and row IDs

Every row has a **row ID**. You don't normally see it, but you use it to update or delete specific rows. In the Python SDK you ask for it with `select(..., internal_row_id=True)`, which adds a `$row_id` column, and then pass those rows to `update()` or `delete()`. See [updates and deletes](/develop/docs/dev-database-guides-updates-and-deletes).

The database allocates row IDs by default. You can instead control them yourself with **user-defined row IDs**: the table gets a `vastdb_rowid` column of type `int64`, values can't exceed 2^47, and you supply them on insert. Once a table uses user-defined row IDs for one insert, every insert into it must supply them. They're useful when you need to address known ranges of rows, and they rule out some other table features (see below).

## Table structures

A regular table needs no layout decisions. For better query performance you can give a table one or more of these structures:

| Structure | What it does | Decided |
| --- | --- | --- |
| **Sorted table** | Sorts rows by one or more sort columns, in the background, which speeds up queries that filter on those columns. | At creation, or added later. Can't be changed or removed afterwards. |
| **Partitioned table** | Splits rows into partitions by the values of up to 4 partition keys, so queries that filter on them skip whole partitions. You can also drop a whole partition at once. | At creation only |
| **Semi-sorted projection** | An extra copy of chosen columns, sorted differently, which queries use automatically when it helps. | Can be added to an existing table |
| **User-defined row IDs** | You control row ID allocation (see above). | At creation, or by the first insert that supplies `vastdb_rowid` |
| **Vector index** | Enables similarity search on a vector column. See the [VAST Vector Search](/develop/docs/dev-vector-search-overview). | At creation |

These structures **can't be combined freely**, and some combinations also rule out replication. For example, a sorted table can't have semi-sorted projections or user-defined row IDs, and a partitioned table can't have a vector index. [Limits](/develop/docs/dev-database-reference-limits) has the full combinability matrix. [Data modeling](/develop/docs/dev-database-guides-data-modeling) helps you choose, and [table structures](/develop/docs/dev-database-concepts-architecture-table-structures) explains what each one costs.

## Snapshots and replication

Tables are elements in the cluster's [Element Store](/develop/docs/dev-database-concepts-architecture-cluster-architecture#one-element-store), next to files and objects, so cluster snapshots include them. You can read a table as it was at a snapshot with the same calls you use on the live table. In the Python SDK, that's `tx.bucket("sales").snapshot("<name>")`. Your platform team creates and schedules snapshots.

A database on a path protected by asynchronous replication is also copied to other clusters. The copies are built from periodic snapshots, so a replica is a read-only image of the database as of the last replicated snapshot. It holds only committed transactions, and it lags the source by up to the replication interval. If the platform team fails over to a replica, it becomes writable. Sorted and partitioned tables can't be replicated, and semi-sorted projections aren't copied ([Overview of VAST Replication](/docs-5-5/docs/overview-of-vast-replication#asynchronous-replication-of-vast-databases)). See [snapshots and replication](/develop/docs/dev-database-operations-snapshots-and-replication) for reading snapshots and replicas, and the [cluster architecture](/develop/docs/dev-database-concepts-architecture-cluster-architecture#copies-on-other-clusters) for where replication fits.

## How it works inside

These pages explain the mechanisms behind the behaviour described above:

- [The cluster underneath (DASE)](/develop/docs/dev-database-concepts-architecture-cluster-architecture): the Element Store that tables share with files and objects, CNodes, storage enclosures and shared storage, data reduction, and why tables never need re-sharding.
- [Element store layout](/develop/docs/dev-database-concepts-architecture-element-store-layout): how table data is physically organised.
- [Write path](/develop/docs/dev-database-concepts-architecture-write-path): what happens after you commit an insert, update or delete.
- [Read path and pruning](/develop/docs/dev-database-concepts-architecture-read-path-and-pruning): why data order and filters decide how much a query reads.
- [Transactions](/develop/docs/dev-database-concepts-architecture-transactions): isolation, visibility and conflicts.
- [Table structures](/develop/docs/dev-database-concepts-architecture-table-structures): what sorted, partitioned, projected and vector-indexed layouts buy and cost.