Quickstart

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

In this quickstart you'll use the Python SDK to connect to VAST DataBase, create a table, insert a few rows and query them, with and without a filter. Finally you'll remove what you created. It takes about 10 minutes.

Run each code block in order in the same Python session. That can be a python3 prompt, IPython or a notebook. Below each block is the output you should see.

Before you start

You need four things from your platform team:

Setup prerequisites has a checklist you can send to your platform team.

You also need Python 3.10 or later on a Linux client that can reach the endpoint.

What your platform team sets up, in order, before your client connects. See setup prerequisites for each item.

1. Install the SDK

pip install "vastdb==2.1.0"

Check the installed version:

import vastdb
print(vastdb.version())

2.1.0

2. Connect

Put your connection details in environment variables, so they stay out of your code:

export AWS_S3_ENDPOINT_URL="http://vastdb.example.com"   # your endpoint
export AWS_ACCESS_KEY_ID="<access key>"
export AWS_SECRET_ACCESS_KEY="<secret key>"
export VASTDB_BUCKET="<database name>"

vastdb.connect() reads the three AWS_* variables automatically. These are the standard AWS variable names, so if you also use AWS from the same shell, set them in a separate one. VASTDB_BUCKET is only used by this quickstart.

Start Python in that shell and connect:

import os
import pyarrow as pa
import vastdb

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]
print("Connected. Cluster version:", ".".join(str(n) for n in session.features.vast_version))

Connected. Cluster version: 5.5.0.1

The version you see depends on your cluster. connect() contacts the endpoint straight away. If it fails, see Connecting.

3. Create a schema and a table

All work happens inside a transaction. When the with block finishes, the transaction commits. If an exception is raised inside the block, the transaction rolls back.

Table columns are defined as a PyArrow schema:

columns = pa.schema([
    ("id", pa.int64()),
    ("city", pa.utf8()),
    ("temperature", pa.float64()),
])

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).create_schema("quickstart", fail_if_exists=False)
    table = schema.create_table("readings", columns, fail_if_exists=False)
    print(table.columns())

id: int64
  -- field metadata --
  VAST:column_id: '64'
city: string
  -- field metadata --
  VAST:column_id: '65'
temperature: double
  -- field metadata --
  VAST:column_id: '66'

The cluster adds VAST:column_id metadata to each column. You don't need it for this quickstart.

fail_if_exists=False returns the existing schema or table instead of raising an error, so you can safely run this block again.

4. Insert rows

insert() takes a PyArrow table or record batch whose columns match the table's:

rows = pa.table({
    "id": [1, 2, 3, 4, 5],
    "city": ["Lisbon", "Oslo", "Cairo", "Lima", "Perth"],
    "temperature": [21.5, 4.0, 30.2, 18.9, 25.1],
}, schema=columns)

with session.transaction() as tx:
    table = tx.bucket(BUCKET).schema("quickstart").table("readings")
    row_ids = table.insert(rows)
    print(f"Inserted {len(row_ids)} rows")

Inserted 5 rows

insert() returns the row IDs the database assigned to the new rows. You'll need row IDs if you later update or delete specific rows; see updates and deletes.

5. Query the table

select() returns a stream of record batches. read_all() collects them into a PyArrow table, and you need to call it before the transaction ends:

with session.transaction() as tx:
    table = tx.bucket(BUCKET).schema("quickstart").table("readings")
    result = table.select().read_all()

print(result.sort_by("id"))

pyarrow.Table
id: int64
city: string
temperature: double
----
id: [[1,2,3,4,5]]
city: [["Lisbon","Oslo","Cairo","Lima","Perth"]]
temperature: [[21.5,4,30.2,18.9,25.1]]

Rows don't come back in any guaranteed order, so the example sorts them on the client before printing.

6. Filter and choose columns

Pass the columns you want, and a filter written as an Ibis expression. Both are applied on the cluster, so only matching rows and the requested columns are sent back to you:

with session.transaction() as tx:
    table = tx.bucket(BUCKET).schema("quickstart").table("readings")
    warm = table.select(
        columns=["city", "temperature"],
        predicate=table["temperature"] > 20,
    ).read_all()

print(warm.sort_by("city"))

pyarrow.Table
city: string
temperature: double
----
city: [["Cairo","Lisbon","Perth"]]
temperature: [[30.2,21.5,25.1]]

You can combine conditions with & (and) and | (or). Every condition inside an | has to be on the same column. Predicates lists what you can filter on, and querying explains how to write filters that run efficiently.

7. Clean up

Drop the table, then the schema:

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("quickstart")
    schema.table("readings").drop()
    schema.drop()
    print("Dropped quickstart.readings and quickstart")

Dropped quickstart.readings and quickstart

Next steps

• Concepts: how databases, schemas, tables and transactions fit together.
• Ingest: loading data at volume, including importing Parquet files.
• Data modeling: choosing between sorted, partitioned and regular tables.
• Trino and Spark: querying the same tables with SQL engines.
• Python SDK reference: the full API.