Python SDK

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

The vastdb Python SDK is the native client for VAST DataBase. It talks to the cluster directly, and it uses Apache Arrow for everything: table schemas are PyArrow schemas, inserts take Arrow tables, and queries return Arrow record batches. This page explains how the SDK's objects fit together and the idioms that keep your code short and correct. For every class, parameter and default, see the Python SDK reference.

Run the code blocks in order in one Python session. Below each block is the output you should see.

Install

pip install "vastdb==2.1.0"

The SDK needs Python 3.10 or later. For which SDK versions work with which cluster versions, see versions. Before you can connect, your platform team has to give you an endpoint, keys and permissions; see setup prerequisites.

The object lifecycle

A database in VAST is an S3 bucket on a Database-enabled view, so the SDK calls it a bucket. For the concepts behind these objects, see concepts.

Connect, using the environment variables from the quickstart:

import os
import pyarrow as pa
import vastdb
from vastdb import errors

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = "integ_python_sdk"
print("Cluster version:", ".".join(str(n) for n in session.features.vast_version))

Cluster version: 5.5.0.1

vastdb.connect() takes the same arguments as Session: you can pass endpoint, access and secret explicitly instead of using the environment. It also takes timeout (per HTTP request), ssl_verify and backoff_config (see Retries).

Create-or-get, and get-or-none

Most setup code wants "create it unless it exists". Pass fail_if_exists=False to create_schema() and create_table() to get the existing object back instead of an error. Code that reads wants the opposite: pass fail_if_missing=False to schema() and table() to get None instead of an exception.

columns = pa.schema([("id", pa.int64()), ("name", pa.utf8())])

with session.transaction() as tx:
    bucket = tx.bucket(BUCKET)
    schema = bucket.create_schema(SCHEMA, fail_if_exists=False)
    table = schema.create_table("people", columns, fail_if_exists=False)
    print("table:", table.name)
    print("missing table:", schema.table("no_such_table", fail_if_missing=False))
    print("tables:", schema.tablenames())

table: people
missing table: None
tables: ['people']

With fail_if_exists=False, an existing table is returned as it is. Its columns aren't compared with the ones you passed, so check table.arrow_schema if that matters to you.

To list tables, prefer schema.tablenames() to schema.tables(). tables() opens every table, and that loads each one's columns and statistics.

Commit and roll back

A transaction commits only if its block ends without an exception. If anything raises inside the block, everything the transaction did is rolled back:

class Abort(Exception):
    pass

try:
    with session.transaction() as tx:
        tx.bucket(BUCKET).schema(SCHEMA).table("people").insert(
            pa.table({"id": [1], "name": ["Ada"]}, schema=columns))
        raise Abort("changed my mind")
except Abort:
    pass

with session.transaction() as tx:
    print("rows after rollback:", tx.bucket(BUCKET).schema(SCHEMA).table("people").select().read_all().num_rows)

rows after rollback: 0

When a transaction rolls back, the SDK logs rolling back txid=... due to: and the traceback at ERROR level, through the vastdb.transaction logger. If your program hasn't configured logging, Python prints that message to stderr, even when you catch the exception. Configure logging in your application to control where it goes.

Catch exceptions outside the with block when you want the transaction's work undone. If you catch them inside, the block ends normally and the transaction commits whatever was already written. For retry and concurrency patterns, see transactions in practice.

Reuse table metadata

schema.table(name) makes several requests each time: it checks the table exists, then loads its columns and statistics. In a loop that opens the same table in many transactions, load a TableMetadata once and rebuild the table handle from it:

from vastdb import TableRef
from vastdb.table_metadata import TableMetadata

md = TableMetadata(TableRef(BUCKET, SCHEMA, "people"))
with session.transaction() as tx:
    md.load(tx)

for start in range(0, 30, 10):
    with session.transaction() as tx:
        people = tx.table_from_metadata(md)
        people.insert(pa.table({
            "id": list(range(start, start + 10)),
            "name": [f"user-{i}" for i in range(start, start + 10)],
        }, schema=columns))

with session.transaction() as tx:
    people = tx.table_from_metadata(md)
    print(people.select(columns=["id"]).read_all().num_rows, "rows")

30 rows

load(tx) loads the columns, the table type and the statistics. select() needs the table type, so a TableMetadata you only built with arrow_schema= is enough for insert() but not for select(). tx.table_from_metadata() returns a table without the DDL methods (drop(), add_column() and so on). After you change the table's columns, load the metadata again. See TableMetadata and TableRef.

Configuration objects

The SDK groups settings into small dataclasses in vastdb.config. You pass them where they're used:

QueryConfig, ImportConfig, BufferedInsertConfig and BackoffConfig are also importable straight from vastdb. SessionConfig is only in vastdb.config. Invalid values, such as an empty data_endpoints list, raise ValueError when you create the object.

Retries and errors

The SDK retries a request by itself only when the cluster asks it to slow down (errors.Slowdown), or after a connection failure on a request that's safe to repeat, such as a read. BackoffConfig controls those retries; by default the SDK makes up to 10 attempts within 60 seconds. Everything else is raised to your code.

All SDK exceptions are in vastdb.errors. Catch the family you can handle:

from vastdb.config import QueryConfig

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema(SCHEMA)
    try:
        schema.table("no_such_table")
    except errors.MissingTable as e:
        print(type(e).__name__, "->", e.schema, e.table)

    people = schema.table("people")
    try:
        people.select(config=QueryConfig(queue_priority=-1)).read_all()
    except errors.HttpError as e:
        print(type(e).__name__, "->", e.status, e.code)

MissingTable -> integ_python_sdk no_such_table
BadRequest -> 400 InvalidArgument

Notice that the second error was caught inside the transaction, and the transaction then committed. That's fine here because the failed query didn't change anything. The error reference lists every message with its cause and fix. Never print e.url or e.headers into shared logs without checking them: they contain your endpoint's address. The SDK also logs every failed request at WARNING level, as RPC failed: {...} with the same fields, through the vastdb.errors logger.

To see what the SDK is doing, turn on its logger:

import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger("vastdb").setLevel(logging.DEBUG)

Check what the cluster supports

session.features describes the connected cluster. The SDK checks it before it uses a version-dependent feature, and raises errors.NotSupportedVersion without sending anything if the cluster is too old. You can check first yourself:

try:
    session.features.check_partitioned()
    print("partitioned tables: supported")
except errors.NotSupportedVersion:
    print("partitioned tables: not supported by this cluster")

partitioned tables: supported

Version-gated features lists the checks and the cluster version each one needs.

Threads and processes

The SDK uses one session from several threads itself: a query runs one worker thread per data endpoint, and the buffered inserter flushes from your thread pool, opening transactions on the same session.

Tests against a real cluster

The SDK's own test suite shows a pattern that works well: give each test run its own schema, create it with fail_if_exists=False, and drop every table in it, then the schema, when the run ends.

import pytest
import vastdb

@pytest.fixture
def schema_name():
    session = vastdb.connect()
    name = "test_myapp"
    yield name
    with session.transaction() as tx:
        schema = tx.bucket(BUCKET).schema(name, fail_if_missing=False)
        if schema is not None:
            for table_name in schema.tablenames():
                schema.table(table_name).drop()
            schema.drop()

Clean up

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema(SCHEMA)
    schema.table("people").drop()
    schema.drop()
    print("Dropped", SCHEMA)

Dropped integ_python_sdk

Next steps

• Ingest and querying: the SDK's data paths in depth.
• Arrow, pandas and Polars: moving data between the SDK and DataFrame libraries.
• Python SDK reference: the full API.