Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.
This page is a one-page summary of the Python SDK calls you'll use most, the main limits, and where to look next. Every call here is shown running, with its output, on the page it links to.
Connect and run a transaction
import os
import pyarrow as pa
import vastdb
session = vastdb.connect() # reads AWS_S3_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
BUCKET = os.environ["VASTDB_BUCKET"]
with session.transaction() as tx: # commits at the end of the block, rolls back on an exception
schema = tx.bucket(BUCKET).create_schema("my_schema", fail_if_exists=False)
table = schema.create_table("my_table", columns, fail_if_exists=False)
Quickstart · Python SDK · Transactions in practice
Read and write
| Task | Call | More |
|---|---|---|
| Insert rows | table.insert(pa_table) |
Ingest |
| Import Parquet files already in a bucket | table.import_files(["/<bucket>/<key>", ...]) |
Ingest |
| Load a partitioned table | session.buffered_inserter(TableRef(...), pool, BufferedInsertConfig(tx=tx)) |
Ingest |
| Query | table.select(columns=[...], predicate=...).read_all() (inside the transaction) |
Querying |
| Sample rows | table.select(..., limit_rows=10) |
Querying |
| Get row IDs | table.select(..., internal_row_id=True) adds $row_id |
Updates and deletes |
| Update rows | table.update(rows_with_row_id, columns=[...]) |
Updates and deletes |
| Delete rows | table.delete(rows_with_row_id) |
Updates and deletes |
| Table statistics | table.get_stats() |
Monitoring |
Filters that push down
t["id"] == 5
t["id"].between(100, 199)
t["sensor"].isin(["s1", "s2"])
t["note"].startswith("check-1")
t["note"].isnull()
~t["note"].isnull()
(t["id"] > 10) & (t["sensor"] == "s1") # AND across columns: fine
(t["id"] < 10) | (t["id"] > 90) # OR: every term on the same column
Each comparison is between a column and a literal value. An OR across different columns, a comparison between two columns, and NOT around anything except isnull() don't push down. Querying · Pushdowns · SDK predicates
Create other table types
from pyiceberg.transforms import DayTransform, IdentityTransform
from vastdb import PartitionKey, PartitionSpec
schema.create_table("sorted_t", columns, sorting_key=["sensor_id", "ts"])
schema.create_table("partitioned_t", columns, partition_spec=PartitionSpec([
PartitionKey("ts", DayTransform()),
PartitionKey("region", IdentityTransform()),
]))
schema.create_table("row_id_t", columns_with_vastdb_rowid, use_external_row_ids_allocation=True)
table.create_projection("by_sensor", sorted_columns=["sensor_id"], unsorted_columns=["ts", "value"])
Most of these choices can't be changed later. Data modeling · Limits
Limits to remember
| Limit | Value |
|---|---|
| Cell size | 5 MB (64 KB for fixed-size binary) |
| Sort columns | Up to 4 |
| Partition keys | Up to 4 |
| User-defined row ID | At most 2^47 |
| Can't be combined | Sorted with projections, user-defined row IDs or vector index. Partitioned with projections, user-defined row IDs or vector index. |
Details and sources: Limits.
Permissions to ask for
s3:ListBucket on the database, plus the s3:Tabular* actions for what you do. Add s3:QueryEngine* for the VAST Query Engine. Setup prerequisites