Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.
The Python SDK speaks Apache Arrow: insert() takes Arrow tables, and select() returns an Arrow RecordBatchReader. That makes it easy to move data between VAST DataBase and pandas, Polars, DuckDB or Parquet files, usually without copying. This page shows the patterns, and the type conversions to watch for on the way in.
Run the code blocks in order in one Python session. Below each block is the output you should see.
Install
pip install "vastdb==2.1.0" pandas polars duckdb
Examples use pandas 2.3, Polars 1.44 and DuckDB 1.5.
Set up
import datetime as dt
import os
import duckdb
import pandas as pd
import polars as pl
import pyarrow as pa
import pyarrow.compute as pc
import vastdb
from vastdb import errors
session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = "integ_arrow"
weather_schema = pa.schema([
("id", pa.int64()),
("city", pa.utf8()),
("temp", pa.float64()),
("ts", pa.timestamp("us")),
])
with session.transaction() as tx:
schema = tx.bucket(BUCKET).create_schema(SCHEMA, fail_if_exists=False)
schema.create_table("weather", weather_schema, fail_if_exists=False)
print("Ready:", SCHEMA)
Ready: integ_arrow
Insert from pandas
Convert the DataFrame with pa.Table.from_pandas(). Pass the table's own schema and preserve_index=False:
df = pd.DataFrame({
"id": [1, 2, 3],
"city": ["Oslo", "Lima", "Cairo"],
"temp": [4.0, None, 30.2],
"ts": pd.to_datetime(["2026-01-01", "2026-01-02", "2026-01-03"]),
})
print(pa.Table.from_pandas(df).schema.field("ts"))
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
rows = pa.Table.from_pandas(df, schema=weather.arrow_schema, preserve_index=False)
print(rows.schema.field("ts"))
print("Inserted", len(weather.insert(rows)), "rows")
pyarrow.Field<ts: timestamp[ns]>
pyarrow.Field<ts: timestamp[us]>
Inserted 3 rows
Why both arguments:
schema=converts pandas' types to the table's. pandas stores timestamps in nanoseconds, while the table here uses microseconds. And an integer column that has missing values becomesfloat64in pandas; the schema converts it back.preserve_index=Falsestops pandas from adding its index as an extra column (__index_level_0__) when the index isn't a plain range. The table doesn't have that column.
Insert from Polars
Polars' to_arrow() produces Arrow large_string columns for strings. The cluster doesn't accept large_string, so inserting the result as it is fails with errors.NotSupportedSchema:
pdf = pl.DataFrame({
"id": [4, 5],
"city": ["Perth", "Rome"],
"temp": [25.1, 19.0],
"ts": [dt.datetime(2026, 1, 4), dt.datetime(2026, 1, 5)],
})
print(pdf.to_arrow().schema.field("city"))
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
try:
weather.insert(pdf.to_arrow())
except errors.NotSupportedSchema as e:
print(type(e).__name__, "->", e.cause.code, "->", e.message)
pyarrow.Field<city: large_string>
NotSupportedSchema -> TabularUnsupportedColumnType -> Unsupported column type on file to import.
Cast to the table's schema first:
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
print("Inserted", len(weather.insert(pdf.to_arrow().cast(weather.arrow_schema))), "rows")
Inserted 2 rows
The same rule applies to any Arrow data you didn't build yourself: cast it to table.arrow_schema before you insert it. For which Arrow types a table can hold, see data types.
Read into pandas
A RecordBatchReader has read_pandas(), which reads every batch and converts the result in one step:
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
df = weather.select().read_pandas()
df = df.sort_values("id").reset_index(drop=True)
print(df.to_string(index=False))
print(df.dtypes.to_string())
id city temp ts
1 Oslo 4.0 2026-01-01
2 Lima NaN 2026-01-02
3 Cairo 30.2 2026-01-03
4 Perth 25.1 2026-01-04
5 Rome 19.0 2026-01-05
id int64
city object
temp float64
ts datetime64[us]
A null in a float64 column becomes NaN, and strings arrive as Python objects. If you'd rather keep Arrow's types and nulls, use read_all().to_pandas(types_mapper=pd.ArrowDtype).
Read into Polars
pl.from_arrow() accepts the reader directly:
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
pdf = pl.from_arrow(weather.select(columns=["id", "city", "temp"], predicate=weather["id"] >= 2))
print(pdf.sort("id"))
shape: (4, 3)
┌─────┬───────┬──────┐
│ id ┆ city ┆ temp │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ f64 │
╞═════╪═══════╪══════╡
│ 2 ┆ Lima ┆ null │
│ 3 ┆ Cairo ┆ 30.2 │
│ 4 ┆ Perth ┆ 25.1 │
│ 5 ┆ Rome ┆ 19.0 │
└─────┴───────┴──────┘
Here the predicate runs on the cluster, so only the matching rows reach Polars. Push filters down this way rather than filtering a full table in Polars; see querying.
Query with DuckDB
DuckDB can query an Arrow reader that's in a Python variable, by its name. Use it for what VAST DataBase doesn't do on its own, such as aggregations and joins, after the cluster has done the filtering:
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
warm = weather.select(columns=["city", "temp"], predicate=weather["temp"] > 10)
result = duckdb.sql("SELECT count(*) AS n, round(avg(temp), 2) AS avg_temp FROM warm").fetchall()
print(result)
[(3, 24.77)]
Two rules:
- Run the DuckDB query inside the transaction. DuckDB pulls batches from the reader while the query runs, and the reader stops working when the transaction ends.
- A reader can be read once. To run several DuckDB queries over the same data, call
select()again, orread_all()into apa.Tablefirst and query that.
Write query results to Parquet
Stream the reader into a ParquetWriter, one batch at a time, so the whole result never has to be in memory:
import tempfile
import pyarrow.parquet as pq
path = os.path.join(tempfile.mkdtemp(), "weather.parquet")
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
reader = weather.select(columns=["id", "city", "temp"])
with pq.ParquetWriter(path, reader.schema) as writer:
for batch in reader:
writer.write_batch(batch)
print(pq.read_table(path).sort_by("id").to_pydict())
{'id': [1, 2, 3, 4, 5], 'city': ['Oslo', 'Lima', 'Cairo', 'Perth', 'Rome'], 'temp': [4.0, None, 30.2, 25.1, 19.0]}
To go the other way, from Parquet files into a table, see ingest. If the files are already on the cluster, import_files() loads them without passing the data through your client.
Process batches with Arrow compute
For results too large for memory, compute over the batches as they arrive with pyarrow.compute:
with session.transaction() as tx:
weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather")
highest = None
for batch in weather.select(columns=["temp"], predicate=~weather["temp"].isnull()):
batch_max = pc.max(batch["temp"]).as_py()
highest = batch_max if highest is None else max(highest, batch_max)
print("Highest temperature:", highest)
Highest temperature: 30.2
Clean up
with session.transaction() as tx:
schema = tx.bucket(BUCKET).schema(SCHEMA)
schema.table("weather").drop()
schema.drop()
print("Dropped", SCHEMA)
Dropped integ_arrow
Next steps
- Querying: predicates and column choices that keep results small.
- Ingest: batch sizing for large DataFrames.
- Python SDK: transactions, metadata reuse and error handling.