--- title: "Python SDK reference" slug: "dev-database-reference-sdk" description: "Explore the vastdb Python SDK 2.1.0 API reference, covering transactions, sessions, and database operations for efficient data management." updated: 2026-10-30T18:18:46Z published: 2026-10-30T18:18:46Z canonical: "docs.vastdata.com/dev-database-reference-sdk" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Python SDK reference *Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.* This page is the API reference for the `vastdb` Python SDK, **version 2.1.0**. Every signature, default and exception on it comes from the 2.1.0 source. If you're new to the SDK, start with the [quickstart](/develop/docs/dev-database-quickstart). For idioms and patterns, see [Python SDK](/develop/docs/dev-database-integrations-python-sdk). ## Install ``` pip install vastdb==2.1.0 ``` | Requirement | Value in 2.1.0 | | --- | --- | | Python | 3.10 or later (`python_requires='>=3.10.0'`) | | PyArrow | `>=18.0,<=23.0.1` | | Ibis | `ibis-framework~=10.1` (used to build predicates) | | Optional extra | `pip install 'vastdb[adbc]'` installs `adbc-driver-vastdb~=0.0.26`. Vector search needs it. | Before you can connect, you need an endpoint, an access key pair and an identity policy that grants the Tabular actions. See [setup prerequisites](/develop/docs/dev-database-operations-setup-prerequisites). ## Object model Every database operation runs inside a transaction. You reach objects by walking down from the transaction: ``` vastdb.connect(...) → Session session.transaction() → Transaction (context manager: commit on success, roll back on exception) tx.bucket(name) → Bucket bucket.schema(name) → Schema (schemas can nest) schema.table(name) → Table ``` ``` import pyarrow as pa import vastdb session = vastdb.connect(endpoint="http://", access="", secret="") with session.transaction() as tx: schema = tx.bucket("").create_schema("", fail_if_exists=False) table = schema.create_table("", pa.schema([("id", pa.int64()), ("name", pa.utf8())]), fail_if_exists=False) table.insert(pa.table({"id": [1, 2], "name": ["a", "b"]})) result = table.select(columns=["name"], predicate=table["id"] > 1).read_all() ``` Objects you get from a transaction are bound to it. Once the `with` block exits they can't be used any more, and that includes the `RecordBatchReader` that `select()` returns. Read it all before the block ends. To reuse table metadata across transactions, use [TableMetadata](/develop/docs/dev-database-reference-sdk#tablemetadata-and-tableref). ## Module `vastdb` | Name | Description | | --- | --- | | `connect(*args, **kwargs) -> Session` | A shorthand for `Session(...)` that takes the same arguments. | | `version() -> str` | Returns the installed SDK version, e.g. `"2.1.0"`. | | `PartitionSpec`, `PartitionKey` | [Partitioning](/develop/docs/dev-database-reference-sdk#partitionspec-and-partitionkey) | | `VectorIndexSpec`, `DefaultQuantization` | [Vector index](/develop/docs/dev-database-reference-sdk#vector-index) | | `QueryConfig`, `ImportConfig`, `BufferedInsertConfig`, `BackoffConfig` | [Configuration](/develop/docs/dev-database-reference-sdk#configuration) | | `BlobExpansionConfig`, `ExpansionFormat` | [Blob expansion](/develop/docs/dev-database-reference-sdk#blob-expansion) | | `TableRef`, `TableType` | [Table metadata](/develop/docs/dev-database-reference-sdk#tablemetadata-and-tableref) | `SessionConfig` isn't re-exported from the top level. Import it with `from vastdb.config import SessionConfig`. ## Session ``` vastdb.session.Session( access: str | None = None, secret: str | None = None, endpoint: str | None = None, *, max_entities_per_page: int = 10_000, ssl_verify: bool = True, timeout=None, backoff_config: BackoffConfig | None = None, adbc_driver_path: str | None = None, end_user: str | None = None, config: SessionConfig = SessionConfig(), ) ``` | Parameter | Description | | --- | --- | | `access`, `secret` | S3 access and secret key. If you omit them, they're read from `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. If the variable isn't set, you get a `KeyError`. | | `endpoint` | URL of the VAST endpoint. If omitted, it's read from `AWS_S3_ENDPOINT_URL`. If neither is set, you get `ValueError: One of endpoint and AWS_S3_ENDPOINT_URL env is required`. | | `max_entities_per_page` | Page size for listing schemas, tables, columns and so on. | | `ssl_verify` | Passed to the HTTP client. Set `False` to skip TLS certificate verification. | | `timeout` | Per-request timeout, passed to the HTTP client (`requests`). `None` means no timeout. | | `backoff_config` | Retry policy. See [BackoffConfig](/develop/docs/dev-database-reference-sdk#backoffconfig) and [Retries](/develop/docs/dev-database-reference-sdk#retries). | | `adbc_driver_path` | Path to the VAST ADBC driver library. If omitted, the SDK uses the driver from `vastdb[adbc]` when it's installed. You only need ADBC for [vector search](/develop/docs/dev-database-reference-sdk#vector-search). | | `end_user` | Runs ADBC (VAST Query Engine) operations, such as `vector_search()`, as this end user instead of as the key's owner. The SDK passes it as the `vast.db.end_user` connection option. The key's owner needs the `s3:EndUserImpersonation` permission. Row and column security then applies as that end user; see [security](/develop/docs/dev-database-operations-security). | | `config` | Session defaults. See [SessionConfig](/develop/docs/dev-database-reference-sdk#sessionconfig). | Constructing a `Session` contacts the endpoint straight away to detect the cluster version, which decides which features you can use (see [Version-gated features](/develop/docs/dev-database-reference-sdk#version-gated-features)). If the endpoint isn't a VAST DataBase endpoint, the constructor raises an error. `repr(session)` never shows the secret key. | Member | Description | | --- | --- | | `transaction() -> Transaction` | Returns a new transaction that hasn't started yet. Use it as a context manager. | | `buffered_inserter(metadata: TableRef \\| TableMetadata, executor: ThreadPoolExecutor, flush_config: BufferedInsertConfig \\| None = None)` | Returns a buffered inserter for streaming record batches into a **partitioned** table. See [Buffered inserts](/develop/docs/dev-database-reference-sdk#buffered-inserts-partitioned-tables). Needs cluster 5.5 or later. | | `features` | A `Features` object describing what the connected cluster version supports. | ## Transaction ``` with session.transaction() as tx: ... ``` When you enter the block, a transaction starts. If the block exits normally, the transaction **commits**. If it exits because of an exception, the transaction **rolls back** and the exception propagates. | Member | Description | | --- | --- | | `bucket(name: str) -> Bucket` | Returns the bucket. Raises `errors.MissingBucket` if it doesn't exist. | | `table_from_metadata(metadata: TableMetadata) -> ITable` | Returns a table handle built from metadata you already have, without listing bucket, schema and table. See [Reusing metadata across transactions](/develop/docs/dev-database-reference-sdk#reusing-metadata-across-transactions). | | `catalog(snapshot: Bucket \\| None = None, fail_if_missing=True) -> Table \\| None` | Returns the VAST Catalog table. See [VAST Catalog](/develop/docs/dev-catalog-overview). | | `catalog_snapshots() -> list[Bucket]` | Lists the VAST Catalog snapshots. See [snapshots and change detection](/develop/docs/dev-catalog-guides-snapshots-and-change-detection). | | `audit_log(fail_if_missing=True) -> Table \\| None` | Returns the audit log table. See [query the audit log](/develop/docs/dev-catalog-guides-audit-log). | | `is_active -> bool` | Whether the transaction is open. | | `active_txid -> int` | The transaction ID. Raises `TransactionNotActiveError` if the transaction isn't open. | | `adbc_conn` | The ADBC connection for this transaction. Raises `NoAdbcConnectionError` if no ADBC driver is available. | ## Bucket | Member | Description | | --- | --- | | `name: str` | Bucket name. For a snapshot, this is `"<bucket>/.snapshot/<snapshot>"`. | | `create_schema(name: str, fail_if_exists=True) -> Schema` | Creates a top-level schema. If the schema already exists, raises `errors.SchemaExists`, or returns the existing one when `fail_if_exists=False`. | | `schema(name: str, fail_if_missing=True) -> Schema \\| None` | Returns a top-level schema. Raises `errors.MissingSchema`, or returns `None` when `fail_if_missing=False`. | | `schemas(batch_size: int \\| None = None) -> list[Schema]` | Lists the top-level schemas. | | `snapshot(name: str, fail_if_missing=True) -> Bucket \\| None` | Returns a snapshot of this bucket as a `Bucket`. Raises `errors.MissingSnapshot`, or returns `None` when `fail_if_missing=False`. | | `snapshots() -> list[Bucket]` | Lists this bucket's snapshots, each as a `Bucket`. | You can read a snapshot with the same `schema()` / `table()` / `select()` calls you use on the live bucket. For what snapshots contain, see [snapshots and replication](/develop/docs/dev-database-operations-snapshots-and-replication). ## Schema | Member | Description | | --- | --- | | `name: str` | Full schema path within the bucket. Nested schemas are `/`-separated. | | `create_schema(name, fail_if_exists=True) -> Schema` | Creates a child schema. | | `schema(name, fail_if_missing=True) -> Schema \\| None` | Returns a child schema. | | `schemas(batch_size=None) -> list[Schema]` | Lists the child schemas. | | `create_table(...) -> Table` | See [below](/develop/docs/dev-database-reference-sdk#schemacreatetable). | | `table(name: str, fail_if_missing=True) -> Table \\| None` | Returns a table and loads its schema and stats. Raises `errors.MissingTable`, or returns `None`. | | `tables(table_name: str = "", page_size=1000) -> list[Table]` | Lists the tables. If you pass `table_name`, it returns only that table, if it exists. **Each returned `Table` loads its own metadata**, so on a schema with many tables use `tablenames()` instead. | | `tablenames(page_size=1000) -> list[str]` | Lists the table names only. | | `drop() -> None` | Drops this schema. | | `rename(new_name) -> None` | Renames this schema. | ### `Schema.create_table` ``` create_table( table_name: str, columns: pa.Schema, fail_if_exists: bool = True, use_external_row_ids_allocation: bool = False, sorting_key: list[str] | list[int] = [], vector_index: VectorIndexSpec | None = None, partition_spec: PartitionSpec | None = None, ) -> Table ``` | Parameter | Description | | --- | --- | | `table_name` | Name of the new table. | | `columns` | The table's columns, as a PyArrow schema. Types that Ibis can't represent are rejected with `errors.NotSupportedSchema`. For supported types, see [data types](/develop/docs/dev-database-reference-data-types). | | `fail_if_exists` | If `False` and the table exists, the existing table is returned unchanged. The other arguments aren't compared against it. | | `use_external_row_ids_allocation` | Creates the table with user-defined row IDs. Needs cluster 5.1 or later. | | `sorting_key` | Columns to sort by, as names or column indexes. A non-empty key creates a sorted table. | | `vector_index` | Creates a vector-indexed table. See [Vector index](/develop/docs/dev-database-reference-sdk#vector-index). | | `partition_spec` | Creates a partitioned table. See [PartitionSpec](/develop/docs/dev-database-reference-sdk#partitionspec-and-partitionkey). | Sorting, partitioning, vector indexing and user-defined row IDs can't all be combined freely. See [limits](/develop/docs/dev-database-reference-limits) for which combinations are allowed. For how to choose between them, see [data modeling](/develop/docs/dev-database-guides-data-modeling). ## Table `schema.table()` and `schema.create_table()` return a `Table`. `tx.table_from_metadata()` returns a `TableInTransaction`, which has the same query and write methods but lacks the DDL methods marked **Table only** below. ### Properties and metadata | Member | Description | | --- | --- | | `name -> str` | Table name. | | `ref -> TableRef` | Bucket, schema and table name. | | `path -> str` | `"<bucket>/<schema>/<table>"` | | `arrow_schema -> pa.Schema` | The cached column schema. | | `columns() -> pa.Schema` | **Table only.** Reloads the column schema from the server and returns it. | | `retrieve_column_names() -> list[str]` | Fetches only the column names. On cluster 5.5 this is a cheaper call than loading the full schema. | | `reload_schema() -> None` | Reloads the cached schema. | | `stats` | `TableStats`. On a `Table` this **reloads stats from the server each time you read it**. On a `TableInTransaction` it returns the cached value, which may be `None`. | | `get_stats() -> TableStats` | **Table only.** The same as `stats`. | | `reload_stats() -> None` | Reloads the cached stats. | | `sorted_columns() -> list[pa.Field]` | The table's sort-key columns. In 2.1.0, `Table.sorted_columns()` goes through the deprecated reload path internally, so it emits a `DeprecationWarning` even though `sorted_columns()` itself isn't deprecated. `TableInTransaction.sorted_columns()` returns the cached value and doesn't warn. | | `sorted_table -> bool` | **Table only.** Whether the table is sorted. | | `vector_index` | The table's `VectorIndex`, or `None`. | | `handle -> int` | **Table only.** The server-side table handle. | | `t["col"]` | Returns an Ibis column expression for building [predicates](/develop/docs/dev-database-reference-sdk#predicates). | `TableStats` fields are `num_rows`, `size_in_bytes`, `sorting_score`, `write_amplification`, `acummulative_row_inserition_count` (the spelling is the SDK's), `vips`, `is_external_rowid_alloc`, `sorting_key_enabled`, `sorting_done`, `endpoints`, `vector_index` and `partitioning_info`. `vips` lists the virtual IPs of the pool that the session's endpoint belongs to, so a session opened on a Query Engine endpoint reports Query Engine addresses. On a vector-indexed table, `num_rows` can be higher than the real row count while the cluster indexes new rows in the background; use a count query when you need the exact number. ### `select` ``` select( columns: list[str] | None = None, predicate: ibis.expr.types.BooleanColumn | ibis.common.deferred.Deferred | None = None, config: QueryConfig | None = None, *, internal_row_id: bool = False, limit_rows: int | None = None, ) -> pa.RecordBatchReader ``` | Parameter | Description | | --- | --- | | `columns` | The columns to return. `None` returns all of them. Only these columns are read (projection pushdown). | | `predicate` | A row filter that's evaluated on the cluster. See [Predicates](/develop/docs/dev-database-reference-sdk#predicates). `True` is treated as no filter, and `False` returns an empty result without running a query. | | `config` | Query execution settings. See [QueryConfig](/develop/docs/dev-database-reference-sdk#queryconfig). | | `internal_row_id` | Adds a `$row_id` column to the result. You need it for `update()` and `delete()`. | | `limit_rows` | Stops once this many rows have been returned. It must be non-negative. | The return value is a `pyarrow.RecordBatchReader` that streams results while you iterate. Call `.read_all()` to get a `pa.Table`, or `.read_pandas()`. **Consume it before the transaction ends.** If the transaction closes while the reader is still in use, you get `errors.MissingTransaction`. How a query runs: - The query is split into disjoint row ranges called *splits*, and the splits are spread across the data endpoints. Unless you set `QueryConfig.num_splits`, the number of splits is estimated from the table's row count, as `num_rows // QueryConfig.rows_per_split` with a minimum of 1. For a sorted table, the SDK also asks the server for an estimate that takes the predicate into account, and uses the smaller of the two. - By default only the endpoints the session discovers are used. To spread a query over more CNodes, set `QueryConfig.data_endpoints`. See [performance](/develop/docs/dev-database-operations-performance). - On a **partitioned** table, the SDK first selects the matching partitions, using only the parts of the predicate that reference partition source columns. It then queries each partition. If the predicate doesn't constrain partition columns, every partition is scanned. ### `select_splits` ``` select_splits(columns=None, predicate=None, config=None, *, internal_row_id=False, limit_rows=None) -> list[pa.RecordBatchReader] ``` Takes the same arguments as `select()`, but returns one reader per split instead of merging them. Use it when you want to parallelise consumption yourself, for example one split per worker process. Partitioned tables raise `NotImplementedError("Method select_splits is not implemented for partitioned tables")`. ### Predicates You build predicates with [Ibis](https://ibis-project.org) expressions, either from the table (`t["c"]`) or from the deferred `_` (`from ibis import _`). The SDK serialises a predicate and the cluster evaluates it. A predicate the SDK can't serialise raises `NotImplementedError` **before anything is sent**. | Ibis expression | Meaning | | --- | --- | | `t["c"] > v`, `>=`, `<`, `<=`, `==`, `!=` | Comparison with a literal | | `t["c"].isnull()` | `IS NULL` | | `~t["c"].isnull()` | `IS NOT NULL`. `~` is only supported directly around `isnull()`. | | `t["c"].isin([v1, v2])` | `IN` (sent as OR'ed equalities) | | `t["c"].between(lo, hi)` | `BETWEEN` | | `t["c"].contains("s")` | Substring match (`LIKE '%s%'`) | | `t["c"].startswith("s")` | Prefix match. `startswith("")` is treated as `IS NOT NULL`. | | `a & b`, `a \\| b` | AND, OR | Shape rules: - The predicate must be an **AND of ORs**. Inside each OR group, every term must reference the **same column**. For example, `((t["a"] > 1) | (t["a"].isnull())) & (t["b"] == 2)` works, but `(t["a"] > 1) | (t["b"] == 2)` raises `NotImplementedError`. - Comparisons must be against a literal. Column-to-column comparisons aren't supported. - You can filter on a struct sub-field (`t["s"]["x"]`) only when it's a leaf. Fixed-size lists with a flat element type are supported. Other nested columns aren't. For which operators push down per engine, see [pushdowns](/develop/docs/dev-database-reference-pushdowns). ### `insert` ``` insert(rows: pa.RecordBatch | pa.Table, by_columns: bool = True) -> pa.ChunkedArray ``` Inserts the rows and returns the row IDs that were assigned to them. A `pa.Table` is combined into a single batch before it's sent. `by_columns` is ignored and only kept for backward compatibility. Each request is limited to 5 MiB, and the SDK keeps within that for you: 1. If the batch fits in one request, it's sent as a single insert. 2. If it doesn't, the SDK first inserts as many columns as fit, splitting the rows across several requests if needed. That always includes the columns you can't update later: sort keys, partition columns and the vector column. This step returns the row IDs. 3. The remaining columns are then written with **updates** against those row IDs. All of these requests run in your transaction, so they commit or roll back together. An oversized batch does cost several requests, though, so batch size still affects throughput; see [ingest](/develop/docs/dev-database-guides-ingest). If a single row can't fit in one request, you get `errors.TooWideRow("One of the rows in the given batch is too big for an rpc")`. ### `update` and `delete` ``` update(rows: pa.RecordBatch | pa.Table, columns: list[str] | None = None) -> None delete(rows: pa.RecordBatch | pa.Table) -> None ``` Both identify rows by a `$row_id` column, which must be present in `rows`. Without it you get `errors.MissingRowIdColumn`. `update()` writes the named `columns`, or every column except `$row_id` if you don't name any. The usual way to get row IDs is to select them first: ``` with session.transaction() as tx: t = tx.bucket("").schema("").table("
") hits = t.select(columns=["id"], predicate=t["id"] == 42, internal_row_id=True).read_all() t.delete(hits) ``` The `$row_id` type depends on the table. It's `uint64` on regular tables, and `decimal128(38, 0)` on sorted, partitioned and vector-indexed tables. If you take `$row_id` straight from `select(..., internal_row_id=True)`, it has the right type for that table. You can't update sort-key columns, partition columns or the vector-index column. ### Import Parquet files ``` import_files(files_to_import: Iterable[str], config: ImportConfig | None = None) -> None import_partitioned_files(files_and_partitions: dict[str, pa.RecordBatch], config: ImportConfig | None = None) -> None ``` These have the cluster import Parquet files that are already in a VAST S3 bucket, using the session's credentials. The data doesn't pass through the client. Paths have the form `"/<bucket>/<path/to/file.parquet>"`. A path without the leading `/` raises `errors.InvalidArgument`. The work is spread over the discovered data endpoints, or over `ImportConfig.data_endpoints` if you set it. By default the concurrency is twice the number of CNodes. Files are sent in batches of up to 10 per request. `import_partitioned_files` maps each file to a one-row `RecordBatch` that holds its partition values. To infer a schema from the files, create the table and import in one call, use `vastdb.util.create_table_from_files(schema, table_name, parquet_files, schema_merge_func=None, config=None)`. `schema_merge_func` must be one of: | Function | Behaviour | | --- | --- | | `util.default_schema_merge` (default) | Each file's schema must be a subset of the largest. Otherwise it raises `InvalidArgument("Found mismatch in parquet files schemas.")`. | | `util.strict_schema_merge` | All files must have identical schemas. | | `util.union_schema_merge` | Unifies all the schemas (`pa.unify_schemas`). | See [ingest](/develop/docs/dev-database-guides-ingest) for guidance. ### Schema evolution — **Table only** | Method | Description | | --- | --- | | `add_column(new_column: pa.Schema)` | Adds the columns in `new_column`. | | `drop_column(column_to_drop: pa.Schema)` | Drops the columns in `column_to_drop`. | | `rename_column(current_column_name: str, new_column_name: str)` | Renames a column. | | `rename(new_name: str)` | Renames the table. | | `drop()` | Drops the table. | ### Sorting | Method | Description | | --- | --- | | `add_sorting_key(sorting_key: list[str] \\| list[int])` | **Table only.** Adds a sort key to a table that doesn't already have one. It raises `ValueError` if the key is empty. Needs cluster 5.3 or later. | | `sorting_done() -> bool` | Whether the background sort has caught up. It's always `False` for unsorted tables. | | `sorting_score() -> int` | Sorting progress score. It's always `0` for unsorted tables. | ### Partitions These methods only work on partitioned tables. On any other table they raise `AssertionError("partitions can only be used on a partitioned table")`. | Method | Description | | --- | --- | | `partitions() -> ReadOnlyTable` | Returns a queryable view of the table's partitions. It supports `select(...)`, `arrow_schema` and `p["col"]`. Besides the partition columns, each row has estimate columns `est_row_count` and `est_byte_size`. | | `delete_partitions(partitions: pa.RecordBatch, *, allow_non_acid: bool = False)` | Deletes whole partitions. Each row of `partitions` identifies one partition, using the schema returned by `partitions()`. **This operation is not transactional.** It refuses to run with `AssertionError` unless you pass `allow_non_acid=True`. | For loading data into partitioned tables, see [Buffered inserts](/develop/docs/dev-database-reference-sdk#buffered-inserts-partitioned-tables). ### Semi-sorted projections | Method | Description | | --- | --- | | `create_projection(projection_name: str, sorted_columns: list[str], unsorted_columns: list[str]) -> Projection` | **Table only.** Creates a semi-sorted projection. | | `projection(name: str, include_stats: bool = True) -> Projection` | Returns a projection. Raises `errors.MissingProjection` if it doesn't exist. | | `projections(projection_name: str = "", include_stats: bool = True) -> list[Projection]` | Lists the projections. | `Projection` has the attributes `name` and `stats`, where `stats.num_rows` and `stats.size_in_bytes` are populated. Its methods are `columns() -> pa.Schema`, `rename(new_name)` and `drop()`. Queries use projections automatically. `QueryConfig.use_semi_sorted_projections` turns that off, and `QueryConfig.semi_sorted_projection_name` forces a particular projection (cluster 5.1 or later). ### Imports table Each table can have an *imports table* that records which files have been imported into it. It needs cluster 5.2 or later. | Method | Description | | --- | --- | | `create_imports_table(fail_if_exists=True) -> ITable` | **Table only.** Creates the imports table. On a 5.5 cluster a new table already has one, so this raises `InternalServerError` (`failed creating import table (res=EXIST)`). `fail_if_exists` is ignored in 2.1.0. Use `imports_table()` instead. | | `imports_table() -> ITable` | Returns the imports table, which you can query with `select()`. | ### Vector index Vector-indexed tables are created with `create_table(..., vector_index=VectorIndexSpec(...))` and searched with [vector_search()](/develop/docs/dev-database-reference-sdk#vector-search). Both need cluster 5.5 or later; on an older cluster the SDK raises `Vector tables require 5.5+ VAST release`. You can only add the index when you create the table, and a table with a vector index can't also be sorted or partitioned, or use user-defined row IDs; see [limits](/develop/docs/dev-database-reference-limits#combining-table-structures). This section covers the SDK surface only. For search semantics, index behaviour and tuning, see [VAST Vector Search](/develop/docs/dev-vector-search-overview), and for a worked example its [Python SDK](/develop/docs/dev-vector-search-integrations-python-sdk) page. `VectorIndexSpec` and `DefaultQuantization` are exported from the `vastdb` package (`from vastdb import VectorIndexSpec, DefaultQuantization`). `VectorIndex` is returned by the SDK; you don't build it yourself. `VectorIndexSpec` is the index you ask for when you create a table: ``` @dataclass class VectorIndexSpec: column: str distance_metric: str cluster_quantization: str | DefaultQuantization = DefaultQuantization() search_quantization: str | DefaultQuantization = DefaultQuantization() ``` | Field | Description | | --- | --- | | `column` | The name of the vector column. In the Arrow schema, it's a fixed-size list of non-nullable `float32` items. | | `distance_metric` | `"l2sq"`, `"cosine"` or `"ip"`. The 2.1.0 docstring lists only `"l2sq"` and `"ip"`, but the cluster also accepts `"cosine"`. | | `cluster_quantization` | `"scalar_8bit"` or `"none"`. Leave it as `DefaultQuantization()` to use the cluster's default (`scalar_8bit`). | | `search_quantization` | `"scalar_8bit"` or `"none"`. Leave it as `DefaultQuantization()` to use the cluster's default (`scalar_8bit`). | If you set `search_quantization="none"`, set `cluster_quantization="none"` too: the cluster rejects `"scalar_8bit"` clustering with `"none"` search with `errors.BadRequest`. `VectorIndex` is the index a table has, as the cluster reports it: ``` @dataclass class VectorIndex: column: str distance_metric: str sql_distance_function: str cluster_quantization: str search_quantization: str ``` | Member | Description | | --- | --- | | `sql_distance_function` | The SQL function the cluster stores for the table's metric: `array_distance` for `l2sq`, `array_cosine_distance` for `cosine`, and `array_inner_product` for `ip`. `vector_search()` orders by it. | | `to_vector_index_spec()` | Turns it back into a `VectorIndexSpec`. | `table.vector_index` and `table.stats.vector_index` hold the table's `VectorIndex`, or `None` if it has no index. If `table.vector_index` is `None` for a table you know is indexed, load the table again in a new transaction. For a table created with `VectorIndexSpec(column="embedding", distance_metric="l2sq")`, its value is: `VectorIndex(column='embedding', distance_metric='l2sq', sql_distance_function='array_distance', cluster_quantization='scalar_8bit', search_quantization='scalar_8bit')` The same definition is in the table's Arrow schema metadata, `table.arrow_schema.metadata`, under the key `b"VAST:vector_index_spec"`, as JSON with the column, metric, SQL function and both quantization settings. ### Vector search **On tables indexed with the `ip` metric, don't use `vector_search()` in SDK 2.1.0:** it returns the least similar rows first. Search those tables with SQL ordered by `array_negative_inner_product`; see [inner-product indexes](/develop/docs/dev-vector-search-integrations-python-sdk#inner-product-indexes-search-with-sql). ``` vector_search(vec: list[float], columns: list[str], limit: int, predicate=None) -> pa.RecordBatchReader ``` | Parameter | Description | | --- | --- | | `vec` | The query vector. Its length must match the column's dimension. | | `columns` | The columns to return. | | `limit` | How many nearest rows to return. Required; there's no default. `limit=100000` returned 100,000 rows. | | `predicate` | An optional filter, written like a `select()` [predicate](/develop/docs/dev-database-reference-sdk#predicates). | The result has the columns you asked for, plus `vast_pysdk_vector_dist`, the distance to `vec`, sorted from smallest to largest. Consume the reader before the transaction ends. `vector_search()` builds one SQL query from `table.vector_index` and runs it on the transaction's ADBC connection, so it needs the [ADBC driver](/develop/docs/dev-database-reference-sdk#install) and `s3:QueryEngine*`. It has no option for the vector search session settings, so it uses the cluster's defaults ([search session settings](/develop/docs/dev-vector-search-reference-configuration)). Errors: - A query vector of the wrong length fails with `Cannot cast array of size <n> to array of size <m>`. - If the ADBC driver isn't installed, `vector_search()` raises `vastdb.transaction.NoAdbcConnectionError: Adbc Driver may not have been supplied`. See [ADBC](/develop/docs/dev-database-integrations-adbc#if-the-driver-isnt-there). ### Blob expansion `create_blob_expansion()`, `blob_expansion()` and the `BlobExpansion` / `BlobExpansionConfig` / `ExpansionFormat` types manage blob expansion. They need cluster 5.5 or later. For how to set it up and what it does, see [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion) in the VAST Event Broker docs. ## TableMetadata and TableRef `TableRef(bucket: str, schema: str, table: str)` names a table. | Member | Value | | --- | --- | | `full_path` / `str(ref)` | `"<bucket>/<schema>/<table>"` | | `query_engine_full_path` | `'"<bucket>/<schema>".<table>'`, the form used in VAST Query Engine SQL | `TableMetadata(ref, arrow_schema=None, table_type=None, vector_index=None, partitions_metadata=None)` holds a table's schema, type and stats, and you can keep it between transactions. | Member | Description | | --- | --- | | `load(tx)` | Loads the schema, table type and stats. | | `load_schema(tx)` / `load_stats(tx)` | Loads only the schema or only the stats. | | `arrow_schema`, `stats`, `ref`, `table_type`, `partitioning`, `sorted_columns` | Cached values. `table_type` and `sorted_columns` raise `ValueError` if they haven't been loaded. | `TableType` values are `Regular`, `Elysium` (a sorted table), `TableImports`, `Partitioned`, and `SortedPartitions` (a table that is both partitioned and sorted). A vector-indexed table reports `Regular`. ### Reusing metadata across transactions `schema.table()` makes several round trips each time you call it: it lists the table, loads the schema and loads the stats. For repeated work, load the metadata once and rebuild the table handle in each transaction: ``` from vastdb import TableRef from vastdb.table_metadata import TableMetadata md = TableMetadata(TableRef("", "", "
")) with session.transaction() as tx: md.load(tx) with session.transaction() as tx: t = tx.table_from_metadata(md) t.insert(batch) ``` If you already know the schema, pass `arrow_schema=` to skip loading it. That's enough for `insert()`, but `select()` also needs the table type, which only `load(tx)` sets. Without it, `select()` raises `ValueError: TableType was not loaded...`. Operations that don't need the schema, such as `import_files()`, work on a `TableMetadata` that has only a `TableRef`. ## Buffered inserts (partitioned tables) ``` from concurrent.futures import ThreadPoolExecutor from vastdb import BufferedInsertConfig, TableRef with ThreadPoolExecutor(max_workers=8) as pool: with session.buffered_inserter(TableRef("", "", "
"), pool) as ins: for batch in batches: ins.add_batch(batch) # leaving the inner block calls finish(): flushes what's left and waits for all writes ``` The inserter groups rows by partition and flushes each group through `executor`. Its methods are `add_batch(batch)` and `finish()`, and it can be used as a context manager. Calling `add_batch()` after `finish()` raises `AssertionError`. If the table isn't partitioned, you get `ValueError("Table must be partitioned to use a buffered inserter")`. **Transactions:** by default, **each flush commits in its own transaction**, so a failure part-way through leaves the earlier flushes committed. To make the whole load one transaction, pass `BufferedInsertConfig(tx=...)`. See [BufferedInsertConfig](/develop/docs/dev-database-reference-sdk#bufferedinsertconfig). ## PartitionSpec and PartitionKey ``` from pyiceberg.transforms import BucketTransform, DayTransform, IdentityTransform from vastdb import PartitionKey, PartitionSpec spec = PartitionSpec([ PartitionKey("event_time", DayTransform()), PartitionKey("event_type", IdentityTransform()), ]) ``` `PartitionKey(column: str, transform)` takes a [PyIceberg transform](https://py.iceberg.apache.org/reference/pyiceberg/transforms/). `PartitionSpec(partition_keys: list[PartitionKey])` allows **at most 4 keys**. More than that raises `ValueError("A partitioned table can be partitioned on no more than 4 keys")`. Partitioned tables need cluster 5.5 or later. ## Configuration All the configuration classes are in `vastdb.config`. ### QueryConfig | Field | Default | Description | | --- | --- | --- | | `num_splits` | `None` (estimated) | Number of disjoint row ranges, queried concurrently with separate requests. | | `num_sub_splits` | `4` | Concurrent reads within a single request, run on the server. | | `rows_per_split` | `4_000_000` | Used to estimate `num_splits` from the row count. | | `limit_rows_per_sub_split` | `131072` | Rows returned per sub-split per response page. | | `num_row_groups_per_sub_split` | `8` | Row groups each sub-split reads before moving on. The SDK notes that semi-sorted projections need the value `8`. | | `data_endpoints` | `None` | Endpoints to spread the query across, one worker thread each. You may list an endpoint more than once. An empty list raises `ValueError`. | | `use_semi_sorted_projections` | `True` | Whether queries may use semi-sorted projections. | | `semi_sorted_projection_name` | `None` | Forces a particular projection. Needs cluster 5.1 or later. A forced projection returns only the rows it has already sorted, so recent rows can be missing; don't force projections in production queries. | | `queue_priority` | `None` | A non-negative integer. Lower values are served first. If unset, the request joins the end of the queue. | | `query_id` | `""` | Used in worker thread names, which helps with debugging. | ### ImportConfig | Field | Default | Description | | --- | --- | --- | | `import_concurrency` | `None` (2 × number of CNodes) | Number of parallel import workers. | | `data_endpoints` | `None` | Endpoints to spread the import across. An empty list raises `ValueError`. | | `key_names` | `None` | Import key column names. If set, the import matches the file's rows to existing table rows on these columns and writes the file's other columns into them. Needs cluster 5.3.1 or later. See [ingest](/develop/docs/dev-database-guides-ingest#add-columns-to-existing-rows). | ### BufferedInsertConfig A frozen dataclass. | Field | Default | Description | | --- | --- | --- | | `capacity_in_bytes` | `10485760` (10 MiB) | Maximum amount of data held in the buffer. | | `target_num_rows_per_partition` | `65536` | Rows a partition must collect before it's flushed. When the buffer is full, or the timer fires, flushes happen anyway. | | `timeout` | `None` | A `timedelta`. When it elapses, everything is flushed. It can't be combined with `tx`. | | `tx` | `None` | One transaction for the whole load. It must already be active. If unset, each flush uses its own transaction. | | `data_endpoints` | `[]` | Endpoints to send flushes to. If empty, the session's endpoints are used. | Invalid values raise `ValueError`. That covers non-positive sizes or timeouts, a `tx` that isn't active, and `tx` combined with `timeout`. ### BackoffConfig | Field | Default | Description | | --- | --- | --- | | `wait_gen` | `backoff.expo` | Wait generator from the [backoff](https://pypi.org/project/backoff/) library. | | `max_value` | `None` | Upper limit on a single wait, in seconds. | | `max_tries` | `10` | Maximum number of attempts. | | `max_time` | `60.0` | Maximum total retry time, in seconds. | | `backoff_log_level` | `logging.DEBUG` | Log level for retry messages. | ### SessionConfig A frozen dataclass with one field, `data_endpoints: list[str] | None = None`. It sets the session-wide default data endpoints for queries and imports. An empty list raises `ValueError`. ## Retries The SDK retries a request automatically, following `BackoffConfig`, only in these cases: - `errors.Slowdown`: the cluster is throttling requests. - `errors.ConnectionError` with `may_retry=True`: a connection failure on a read-only request, or partway through reading a query response. Queries resume from where they stopped. Everything else goes straight to your code, including connection failures on write requests. The transaction context manager then rolls back. Whether to retry the whole transaction is your decision; see [transactions in practice](/develop/docs/dev-database-guides-transactions-in-practice). ## Version-gated features The SDK checks the connected cluster's version before it uses these features. If the cluster is too old, it raises `errors.NotSupportedVersion` and no request is sent. | Feature | SDK entry point | Minimum cluster | | --- | --- | --- | | User-defined row IDs | `create_table(use_external_row_ids_allocation=True)` | 5.1 | | Forcing a semi-sorted projection | `QueryConfig.semi_sorted_projection_name` | 5.1 | | Imports table | `create_imports_table()`, `imports_table()` | 5.2 | | Adding a sort key | `add_sorting_key()` | 5.3 | | Import key columns | `ImportConfig.key_names` | 5.3.1 | | Partitioned tables | `Session.buffered_inserter()` | 5.5 | | Blob expansion | `create_blob_expansion()`, `blob_expansion()` | 5.5 | | Vector-indexed tables | loading a table that has a vector index | 5.5 | For the full SDK × cluster × connector matrix, see [versions](/develop/docs/dev-database-reference-versions). ## Exceptions All SDK exceptions are in `vastdb.errors`. These are the main families: | Family | Raised when | | --- | --- | | `HttpError` subclasses (`BadRequest`, `Forbidden`, `NotFound`, `Conflict`, `ServiceUnavailable`/`Slowdown`, `InternalServerError`, …) | The cluster returned an error. Each one carries `code`, `message`, `method`, `url`, `status` and `headers`. | | `Missing*` (`MissingBucket`, `MissingSchema`, `MissingTable`, `MissingProjection`, `MissingSnapshot`, `MissingTransaction`, `MissingRowIdColumn`) | An object or required input doesn't exist. | | `Exists` (`SchemaExists`, `TableExists`) | A create call used `fail_if_exists=True` and the object already exists. | | `InvalidArgument` (`TooLargeRequest`, `TooWideRow`) | The SDK rejected the input. `TooLargeRequest` is raised before anything is sent. `TooWideRow` can be raised partway through an insert, after some of the row's columns have been written; see [ingest](/develop/docs/dev-database-guides-ingest#rows-that-are-too-big-for-one-request). | | `NotSupported*` (`NotSupportedVersion`, `NotSupportedSchema`, `NotSupportedCommand`) | The cluster version, a column type or the table kind doesn't support the operation. | | `ConnectionError` | A network failure. `may_retry` says whether it was safe to retry. | [Error reference](/develop/docs/dev-database-troubleshooting-errors) lists every error message with its cause and fix. ## Deprecated in 2.1.0 These will be removed in vastdb 3.0.0: | Deprecated | Use instead | | --- | --- | | `reload_sorted_columns()` | Nothing. Sorted columns now load with the schema. | | `TableInTransaction.insert_in_column_batches(rows)` | `insert(rows)` |