--- title: "Arrow, pandas and Polars" slug: "dev-database-integrations-arrow-pandas-polars" description: "Learn how to efficiently move data between VAST DataBase and pandas, Polars, DuckDB, or Parquet files using the Python SDK with Arrow tables." updated: 2026-10-30T18:18:46Z published: 2026-10-30T18:18:46Z canonical: "docs.vastdata.com/dev-database-integrations-arrow-pandas-polars" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Arrow, pandas and Polars *Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.* The Python SDK speaks [Apache Arrow](https://arrow.apache.org/docs/python/): `insert()` takes Arrow tables, and `select()` returns an Arrow `RecordBatchReader`. That makes it easy to move data between VAST DataBase and pandas, Polars, DuckDB or Parquet files, usually without copying. This page shows the patterns, and the type conversions to watch for on the way in. Run the code blocks in order in one Python session. Below each block is the output you should see. ## Install ``` pip install "vastdb==2.1.0" pandas polars duckdb ``` Examples use pandas 2.3, Polars 1.44 and DuckDB 1.5. ## Set up ``` import datetime as dt import os import duckdb import pandas as pd import polars as pl import pyarrow as pa import pyarrow.compute as pc import vastdb from vastdb import errors session = vastdb.connect() BUCKET = os.environ["VASTDB_BUCKET"] SCHEMA = "integ_arrow" weather_schema = pa.schema([ ("id", pa.int64()), ("city", pa.utf8()), ("temp", pa.float64()), ("ts", pa.timestamp("us")), ]) with session.transaction() as tx: schema = tx.bucket(BUCKET).create_schema(SCHEMA, fail_if_exists=False) schema.create_table("weather", weather_schema, fail_if_exists=False) print("Ready:", SCHEMA) ``` ``` Ready: integ_arrow ``` ## Insert from pandas Convert the DataFrame with `pa.Table.from_pandas()`. Pass the **table's own schema** and `preserve_index=False`: ``` df = pd.DataFrame({ "id": [1, 2, 3], "city": ["Oslo", "Lima", "Cairo"], "temp": [4.0, None, 30.2], "ts": pd.to_datetime(["2026-01-01", "2026-01-02", "2026-01-03"]), }) print(pa.Table.from_pandas(df).schema.field("ts")) with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") rows = pa.Table.from_pandas(df, schema=weather.arrow_schema, preserve_index=False) print(rows.schema.field("ts")) print("Inserted", len(weather.insert(rows)), "rows") ``` ``` pyarrow.Field pyarrow.Field Inserted 3 rows ``` Why both arguments: - **`schema=`** converts pandas' types to the table's. pandas stores timestamps in nanoseconds, while the table here uses microseconds. And an integer column that has missing values becomes `float64` in pandas; the schema converts it back. - **`preserve_index=False`** stops pandas from adding its index as an extra column (`__index_level_0__`) when the index isn't a plain range. The table doesn't have that column. ## Insert from Polars Polars' `to_arrow()` produces Arrow `large_string` columns for strings. The cluster doesn't accept `large_string`, so inserting the result as it is fails with `errors.NotSupportedSchema`: ``` pdf = pl.DataFrame({ "id": [4, 5], "city": ["Perth", "Rome"], "temp": [25.1, 19.0], "ts": [dt.datetime(2026, 1, 4), dt.datetime(2026, 1, 5)], }) print(pdf.to_arrow().schema.field("city")) with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") try: weather.insert(pdf.to_arrow()) except errors.NotSupportedSchema as e: print(type(e).__name__, "->", e.cause.code, "->", e.message) ``` ``` pyarrow.Field NotSupportedSchema -> TabularUnsupportedColumnType -> Unsupported column type on file to import. ``` Cast to the table's schema first: ``` with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") print("Inserted", len(weather.insert(pdf.to_arrow().cast(weather.arrow_schema))), "rows") ``` ``` Inserted 2 rows ``` The same rule applies to any Arrow data you didn't build yourself: cast it to `table.arrow_schema` before you insert it. For which Arrow types a table can hold, see [data types](/develop/docs/dev-database-reference-data-types). ## Read into pandas A `RecordBatchReader` has `read_pandas()`, which reads every batch and converts the result in one step: ``` with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") df = weather.select().read_pandas() df = df.sort_values("id").reset_index(drop=True) print(df.to_string(index=False)) print(df.dtypes.to_string()) ``` ``` id city temp ts 1 Oslo 4.0 2026-01-01 2 Lima NaN 2026-01-02 3 Cairo 30.2 2026-01-03 4 Perth 25.1 2026-01-04 5 Rome 19.0 2026-01-05 id int64 city object temp float64 ts datetime64[us] ``` A null in a `float64` column becomes `NaN`, and strings arrive as Python objects. If you'd rather keep Arrow's types and nulls, use `read_all().to_pandas(types_mapper=pd.ArrowDtype)`. ## Read into Polars `pl.from_arrow()` accepts the reader directly: ``` with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") pdf = pl.from_arrow(weather.select(columns=["id", "city", "temp"], predicate=weather["id"] >= 2)) print(pdf.sort("id")) ``` ``` shape: (4, 3) ┌─────┬───────┬──────┐ │ id ┆ city ┆ temp │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ f64 │ ╞═════╪═══════╪══════╡ │ 2 ┆ Lima ┆ null │ │ 3 ┆ Cairo ┆ 30.2 │ │ 4 ┆ Perth ┆ 25.1 │ │ 5 ┆ Rome ┆ 19.0 │ └─────┴───────┴──────┘ ``` Here the predicate runs on the cluster, so only the matching rows reach Polars. Push filters down this way rather than filtering a full table in Polars; see [querying](/develop/docs/dev-database-guides-querying). ## Query with DuckDB DuckDB can query an Arrow reader that's in a Python variable, by its name. Use it for what VAST DataBase doesn't do on its own, such as aggregations and joins, after the cluster has done the filtering: ``` with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") warm = weather.select(columns=["city", "temp"], predicate=weather["temp"] > 10) result = duckdb.sql("SELECT count(*) AS n, round(avg(temp), 2) AS avg_temp FROM warm").fetchall() print(result) ``` ``` [(3, 24.77)] ``` Two rules: - **Run the DuckDB query inside the transaction.** DuckDB pulls batches from the reader while the query runs, and the reader stops working when the transaction ends. - **A reader can be read once.** To run several DuckDB queries over the same data, call `select()` again, or `read_all()` into a `pa.Table` first and query that. ## Write query results to Parquet Stream the reader into a `ParquetWriter`, one batch at a time, so the whole result never has to be in memory: ``` import tempfile import pyarrow.parquet as pq path = os.path.join(tempfile.mkdtemp(), "weather.parquet") with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") reader = weather.select(columns=["id", "city", "temp"]) with pq.ParquetWriter(path, reader.schema) as writer: for batch in reader: writer.write_batch(batch) print(pq.read_table(path).sort_by("id").to_pydict()) ``` ``` {'id': [1, 2, 3, 4, 5], 'city': ['Oslo', 'Lima', 'Cairo', 'Perth', 'Rome'], 'temp': [4.0, None, 30.2, 25.1, 19.0]} ``` To go the other way, from Parquet files into a table, see [ingest](/develop/docs/dev-database-guides-ingest#import-parquet-files). If the files are already on the cluster, `import_files()` loads them without passing the data through your client. ## Process batches with Arrow compute For results too large for memory, compute over the batches as they arrive with `pyarrow.compute`: ``` with session.transaction() as tx: weather = tx.bucket(BUCKET).schema(SCHEMA).table("weather") highest = None for batch in weather.select(columns=["temp"], predicate=~weather["temp"].isnull()): batch_max = pc.max(batch["temp"]).as_py() highest = batch_max if highest is None else max(highest, batch_max) print("Highest temperature:", highest) ``` ``` Highest temperature: 30.2 ``` ## Clean up ``` with session.transaction() as tx: schema = tx.bucket(BUCKET).schema(SCHEMA) schema.table("weather").drop() schema.drop() print("Dropped", SCHEMA) ``` ``` Dropped integ_arrow ``` ## Next steps - [Querying](/develop/docs/dev-database-guides-querying): predicates and column choices that keep results small. - [Ingest](/develop/docs/dev-database-guides-ingest): batch sizing for large DataFrames. - [Python SDK](/develop/docs/dev-database-integrations-python-sdk): transactions, metadata reuse and error handling.