Python SDK reference

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

This page is the API reference for the vastdb Python SDK, version 2.1.0. Every signature, default and exception on it comes from the 2.1.0 source. If you're new to the SDK, start with the quickstart. For idioms and patterns, see Python SDK.

Install

pip install vastdb==2.1.0

Before you can connect, you need an endpoint, an access key pair and an identity policy that grants the Tabular actions. See setup prerequisites.

Object model

Every database operation runs inside a transaction. You reach objects by walking down from the transaction:

vastdb.connect(...)  →  Session
session.transaction()  →  Transaction        (context manager: commit on success, roll back on exception)
tx.bucket(name)  →  Bucket
bucket.schema(name)  →  Schema               (schemas can nest)
schema.table(name)  →  Table

import pyarrow as pa
import vastdb

session = vastdb.connect(endpoint="http://<vip-or-dns>", access="<access-key>", secret="<secret-key>")

with session.transaction() as tx:
    schema = tx.bucket("<bucket>").create_schema("<schema>", fail_if_exists=False)
    table = schema.create_table("<table>", pa.schema([("id", pa.int64()), ("name", pa.utf8())]),
                                fail_if_exists=False)
    table.insert(pa.table({"id": [1, 2], "name": ["a", "b"]}))
    result = table.select(columns=["name"], predicate=table["id"] > 1).read_all()

Objects you get from a transaction are bound to it. Once the with block exits they can't be used any more, and that includes the RecordBatchReader that select() returns. Read it all before the block ends. To reuse table metadata across transactions, use TableMetadata.

Module vastdb

SessionConfig isn't re-exported from the top level. Import it with from vastdb.config import SessionConfig.

Session

vastdb.session.Session(
    access: str | None = None,
    secret: str | None = None,
    endpoint: str | None = None,
    *,
    max_entities_per_page: int = 10_000,
    ssl_verify: bool = True,
    timeout=None,
    backoff_config: BackoffConfig | None = None,
    adbc_driver_path: str | None = None,
    end_user: str | None = None,
    config: SessionConfig = SessionConfig(),
)

Constructing a Session contacts the endpoint straight away to detect the cluster version, which decides which features you can use (see Version-gated features). If the endpoint isn't a VAST DataBase endpoint, the constructor raises an error. repr(session) never shows the secret key.

Transaction

with session.transaction() as tx:
    ...

When you enter the block, a transaction starts. If the block exits normally, the transaction commits. If it exits because of an exception, the transaction rolls back and the exception propagates.

Bucket

You can read a snapshot with the same schema() / table() / select() calls you use on the live bucket. For what snapshots contain, see snapshots and replication.

Schema

Schema.create_table

create_table(
    table_name: str,
    columns: pa.Schema,
    fail_if_exists: bool = True,
    use_external_row_ids_allocation: bool = False,
    sorting_key: list[str] | list[int] = [],
    vector_index: VectorIndexSpec | None = None,
    partition_spec: PartitionSpec | None = None,
) -> Table

Sorting, partitioning, vector indexing and user-defined row IDs can't all be combined freely. See limits for which combinations are allowed. For how to choose between them, see data modeling.

Table

schema.table() and schema.create_table() return a Table. tx.table_from_metadata() returns a TableInTransaction, which has the same query and write methods but lacks the DDL methods marked Table only below.

Properties and metadata

TableStats fields are num_rows, size_in_bytes, sorting_score, write_amplification, acummulative_row_inserition_count (the spelling is the SDK's), vips, is_external_rowid_alloc, sorting_key_enabled, sorting_done, endpoints, vector_index and partitioning_info.

vips lists the virtual IPs of the pool that the session's endpoint belongs to, so a session opened on a Query Engine endpoint reports Query Engine addresses. On a vector-indexed table, num_rows can be higher than the real row count while the cluster indexes new rows in the background; use a count query when you need the exact number.

select

select(
    columns: list[str] | None = None,
    predicate: ibis.expr.types.BooleanColumn | ibis.common.deferred.Deferred | None = None,
    config: QueryConfig | None = None,
    *,
    internal_row_id: bool = False,
    limit_rows: int | None = None,
) -> pa.RecordBatchReader

The return value is a pyarrow.RecordBatchReader that streams results while you iterate. Call .read_all() to get a pa.Table, or .read_pandas(). Consume it before the transaction ends. If the transaction closes while the reader is still in use, you get errors.MissingTransaction.

How a query runs:

• The query is split into disjoint row ranges called splits, and the splits are spread across the data endpoints. Unless you set QueryConfig.num_splits, the number of splits is estimated from the table's row count, as num_rows // QueryConfig.rows_per_split with a minimum of 1. For a sorted table, the SDK also asks the server for an estimate that takes the predicate into account, and uses the smaller of the two.
• By default only the endpoints the session discovers are used. To spread a query over more CNodes, set QueryConfig.data_endpoints. See performance.
• On a partitioned table, the SDK first selects the matching partitions, using only the parts of the predicate that reference partition source columns. It then queries each partition. If the predicate doesn't constrain partition columns, every partition is scanned.

select_splits

select_splits(columns=None, predicate=None, config=None, *, internal_row_id=False, limit_rows=None)
    -> list[pa.RecordBatchReader]

Takes the same arguments as select(), but returns one reader per split instead of merging them. Use it when you want to parallelise consumption yourself, for example one split per worker process. Partitioned tables raise NotImplementedError("Method select_splits is not implemented for partitioned tables").

Predicates

You build predicates with Ibis expressions, either from the table (t["c"]) or from the deferred _ (from ibis import _). The SDK serialises a predicate and the cluster evaluates it. A predicate the SDK can't serialise raises NotImplementedError before anything is sent.

Shape rules:

• The predicate must be an AND of ORs. Inside each OR group, every term must reference the same column. For example, ((t["a"] > 1) | (t["a"].isnull())) & (t["b"] == 2) works, but (t["a"] > 1) | (t["b"] == 2) raises NotImplementedError.
• Comparisons must be against a literal. Column-to-column comparisons aren't supported.
• You can filter on a struct sub-field (t["s"]["x"]) only when it's a leaf. Fixed-size lists with a flat element type are supported. Other nested columns aren't.

For which operators push down per engine, see pushdowns.

insert

insert(rows: pa.RecordBatch | pa.Table, by_columns: bool = True) -> pa.ChunkedArray

Inserts the rows and returns the row IDs that were assigned to them. A pa.Table is combined into a single batch before it's sent. by_columns is ignored and only kept for backward compatibility.

Each request is limited to 5 MiB, and the SDK keeps within that for you:

1. If the batch fits in one request, it's sent as a single insert.
2. If it doesn't, the SDK first inserts as many columns as fit, splitting the rows across several requests if needed. That always includes the columns you can't update later: sort keys, partition columns and the vector column. This step returns the row IDs.
3. The remaining columns are then written with updates against those row IDs.

All of these requests run in your transaction, so they commit or roll back together. An oversized batch does cost several requests, though, so batch size still affects throughput; see ingest. If a single row can't fit in one request, you get errors.TooWideRow("One of the rows in the given batch is too big for an rpc").

update and delete

update(rows: pa.RecordBatch | pa.Table, columns: list[str] | None = None) -> None
delete(rows: pa.RecordBatch | pa.Table) -> None

Both identify rows by a $row_id column, which must be present in rows. Without it you get errors.MissingRowIdColumn. update() writes the named columns, or every column except $row_id if you don't name any.

The usual way to get row IDs is to select them first:

with session.transaction() as tx:
    t = tx.bucket("<bucket>").schema("<schema>").table("<table>")
    hits = t.select(columns=["id"], predicate=t["id"] == 42, internal_row_id=True).read_all()
    t.delete(hits)

The $row_id type depends on the table. It's uint64 on regular tables, and decimal128(38, 0) on sorted, partitioned and vector-indexed tables. If you take $row_id straight from select(..., internal_row_id=True), it has the right type for that table.

You can't update sort-key columns, partition columns or the vector-index column.

Import Parquet files

import_files(files_to_import: Iterable[str], config: ImportConfig | None = None) -> None
import_partitioned_files(files_and_partitions: dict[str, pa.RecordBatch], config: ImportConfig | None = None) -> None

These have the cluster import Parquet files that are already in a VAST S3 bucket, using the session's credentials. The data doesn't pass through the client. Paths have the form "/<bucket>/<path/to/file.parquet>". A path without the leading / raises errors.InvalidArgument.

The work is spread over the discovered data endpoints, or over ImportConfig.data_endpoints if you set it. By default the concurrency is twice the number of CNodes. Files are sent in batches of up to 10 per request. import_partitioned_files maps each file to a one-row RecordBatch that holds its partition values.

To infer a schema from the files, create the table and import in one call, use vastdb.util.create_table_from_files(schema, table_name, parquet_files, schema_merge_func=None, config=None). schema_merge_func must be one of:

See ingest for guidance.

Schema evolution — Table only

Sorting

Partitions

These methods only work on partitioned tables. On any other table they raise AssertionError("partitions can only be used on a partitioned table").

For loading data into partitioned tables, see Buffered inserts.

Semi-sorted projections

Projection has the attributes name and stats, where stats.num_rows and stats.size_in_bytes are populated. Its methods are columns() -> pa.Schema, rename(new_name) and drop().

Queries use projections automatically. QueryConfig.use_semi_sorted_projections turns that off, and QueryConfig.semi_sorted_projection_name forces a particular projection (cluster 5.1 or later).

Imports table

Each table can have an imports table that records which files have been imported into it. It needs cluster 5.2 or later.

Vector index

Vector-indexed tables are created with create_table(..., vector_index=VectorIndexSpec(...)) and searched with vector_search(). Both need cluster 5.5 or later; on an older cluster the SDK raises Vector tables require 5.5+ VAST release. You can only add the index when you create the table, and a table with a vector index can't also be sorted or partitioned, or use user-defined row IDs; see limits. This section covers the SDK surface only. For search semantics, index behaviour and tuning, see VAST Vector Search, and for a worked example its Python SDK page.

VectorIndexSpec and DefaultQuantization are exported from the vastdb package (from vastdb import VectorIndexSpec, DefaultQuantization). VectorIndex is returned by the SDK; you don't build it yourself.

VectorIndexSpec is the index you ask for when you create a table:

@dataclass
class VectorIndexSpec:
    column: str
    distance_metric: str
    cluster_quantization: str | DefaultQuantization = DefaultQuantization()
    search_quantization: str | DefaultQuantization = DefaultQuantization()

If you set search_quantization="none", set cluster_quantization="none" too: the cluster rejects "scalar_8bit" clustering with "none" search with errors.BadRequest.

VectorIndex is the index a table has, as the cluster reports it:

@dataclass
class VectorIndex:
    column: str
    distance_metric: str
    sql_distance_function: str
    cluster_quantization: str
    search_quantization: str

table.vector_index and table.stats.vector_index hold the table's VectorIndex, or None if it has no index. If table.vector_index is None for a table you know is indexed, load the table again in a new transaction. For a table created with VectorIndexSpec(column="embedding", distance_metric="l2sq"), its value is:

VectorIndex(column='embedding', distance_metric='l2sq', sql_distance_function='array_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')

The same definition is in the table's Arrow schema metadata, table.arrow_schema.metadata, under the key b"VAST:vector_index_spec", as JSON with the column, metric, SQL function and both quantization settings.

Vector search

On tables indexed with the ip metric, don't use vector_search() in SDK 2.1.0: it returns the least similar rows first. Search those tables with SQL ordered by array_negative_inner_product; see inner-product indexes.

vector_search(vec: list[float], columns: list[str], limit: int, predicate=None) -> pa.RecordBatchReader

The result has the columns you asked for, plus vast_pysdk_vector_dist, the distance to vec, sorted from smallest to largest. Consume the reader before the transaction ends.

vector_search() builds one SQL query from table.vector_index and runs it on the transaction's ADBC connection, so it needs the ADBC driver and s3:QueryEngine*. It has no option for the vector search session settings, so it uses the cluster's defaults (search session settings).

Errors:

• A query vector of the wrong length fails with Cannot cast array of size <n> to array of size <m>.
• If the ADBC driver isn't installed, vector_search() raises vastdb.transaction.NoAdbcConnectionError: Adbc Driver may not have been supplied. See ADBC.

Blob expansion

create_blob_expansion(), blob_expansion() and the BlobExpansion / BlobExpansionConfig / ExpansionFormat types manage blob expansion. They need cluster 5.5 or later. For how to set it up and what it does, see blob expansion in the VAST Event Broker docs.

TableMetadata and TableRef

TableRef(bucket: str, schema: str, table: str) names a table.

TableMetadata(ref, arrow_schema=None, table_type=None, vector_index=None, partitions_metadata=None) holds a table's schema, type and stats, and you can keep it between transactions.

TableType values are Regular, Elysium (a sorted table), TableImports, Partitioned, and SortedPartitions (a table that is both partitioned and sorted). A vector-indexed table reports Regular.

Reusing metadata across transactions

schema.table() makes several round trips each time you call it: it lists the table, loads the schema and loads the stats. For repeated work, load the metadata once and rebuild the table handle in each transaction:

from vastdb import TableRef
from vastdb.table_metadata import TableMetadata

md = TableMetadata(TableRef("<bucket>", "<schema>", "<table>"))
with session.transaction() as tx:
    md.load(tx)

with session.transaction() as tx:
    t = tx.table_from_metadata(md)
    t.insert(batch)

If you already know the schema, pass arrow_schema= to skip loading it. That's enough for insert(), but select() also needs the table type, which only load(tx) sets. Without it, select() raises ValueError: TableType was not loaded.... Operations that don't need the schema, such as import_files(), work on a TableMetadata that has only a TableRef.

Buffered inserts (partitioned tables)

from concurrent.futures import ThreadPoolExecutor
from vastdb import BufferedInsertConfig, TableRef

with ThreadPoolExecutor(max_workers=8) as pool:
    with session.buffered_inserter(TableRef("<bucket>", "<schema>", "<table>"), pool) as ins:
        for batch in batches:
            ins.add_batch(batch)
    # leaving the inner block calls finish(): flushes what's left and waits for all writes

The inserter groups rows by partition and flushes each group through executor. Its methods are add_batch(batch) and finish(), and it can be used as a context manager. Calling add_batch() after finish() raises AssertionError. If the table isn't partitioned, you get ValueError("Table must be partitioned to use a buffered inserter").

Transactions: by default, each flush commits in its own transaction, so a failure part-way through leaves the earlier flushes committed. To make the whole load one transaction, pass BufferedInsertConfig(tx=...). See BufferedInsertConfig.

PartitionSpec and PartitionKey

from pyiceberg.transforms import BucketTransform, DayTransform, IdentityTransform
from vastdb import PartitionKey, PartitionSpec

spec = PartitionSpec([
    PartitionKey("event_time", DayTransform()),
    PartitionKey("event_type", IdentityTransform()),
])

PartitionKey(column: str, transform) takes a PyIceberg transform. PartitionSpec(partition_keys: list[PartitionKey]) allows at most 4 keys. More than that raises ValueError("A partitioned table can be partitioned on no more than 4 keys"). Partitioned tables need cluster 5.5 or later.

Configuration

All the configuration classes are in vastdb.config.

QueryConfig

ImportConfig

BufferedInsertConfig

A frozen dataclass.

Invalid values raise ValueError. That covers non-positive sizes or timeouts, a tx that isn't active, and tx combined with timeout.

BackoffConfig

SessionConfig

A frozen dataclass with one field, data_endpoints: list[str] | None = None. It sets the session-wide default data endpoints for queries and imports. An empty list raises ValueError.

Retries

The SDK retries a request automatically, following BackoffConfig, only in these cases:

• errors.Slowdown: the cluster is throttling requests.
• errors.ConnectionError with may_retry=True: a connection failure on a read-only request, or partway through reading a query response. Queries resume from where they stopped.

Everything else goes straight to your code, including connection failures on write requests. The transaction context manager then rolls back. Whether to retry the whole transaction is your decision; see transactions in practice.

Version-gated features

The SDK checks the connected cluster's version before it uses these features. If the cluster is too old, it raises errors.NotSupportedVersion and no request is sent.

For the full SDK × cluster × connector matrix, see versions.

Exceptions

All SDK exceptions are in vastdb.errors. These are the main families:

Error reference lists every error message with its cause and fix.

Deprecated in 2.1.0

These will be removed in vastdb 3.0.0: