Data types

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

This page shows which column types VAST DataBase supports in VAST Cluster 5.5, what each one is called in Arrow, Trino, Spark and the Python SDK, and where precision can change between them.

Type mapping

See Supported Data Types.

The SDK also supports float16 (pa.float16(), half precision) since SDK 2.0.6 (SDK changelog). The cluster accepts it.

Engine differences

Nested columns (list, map, struct) can't be partition columns or sort columns. For filtering on nested columns, see pushdowns.

Precision caveats

Timestamps. Check the precision of any type that has one. For example, a timestamp in a Parquet file can show as timestamp(3) in Trino, because of a Trino default, while VAST DataBase processes it as timestamp(6) (see Supported Data Types). From the SDK, the unit you give in pa.timestamp() is the unit the column keeps: a pa.timestamp("ns") column comes back as timestamp[ns]. A timezone is kept too: pa.timestamp("us", tz="Europe/Lisbon") comes back unchanged.

Spark has no unsigned or time types. A UINT or TIME column has no Spark type. If Spark has to read the table, use signed integers and timestamps.

Decimal filters in the Python SDK. SDK 2.1.0 turns a decimal literal in a predicate into a Python float before it scales it to an integer, and drops the remainder. So the value the cluster compares against can be one unit of the scale lower than the one you wrote. For example, on a decimal128(10, 2) column, t["price"] == 0.29 matches the row with 0.28. Don't rely on exact decimal comparisons in a pushed-down predicate. Push down a range that's wider by one unit of the scale on each side, then filter exactly in Arrow. Ibis also rejects a decimal.Decimal literal against a decimal column ("not comparable"), so pass a float.

Float literals are sent with the column's own type, so t["f32"] == 0.1 on a float32 column matches a stored 0.1.

Fixed-size lists and vectors

A vector column is a PyArrow fixed-size list whose element field is not nullable: pa.list_(pa.field("item", pa.float32(), nullable=False), dim). The SDK has supported fixed-size lists of numbers since 1.3.11 (SDK changelog).

To build a vector index on the column, see VectorIndexSpec. The size of a single cell is limited; see limits.

Types the SDK can't use

If a column type isn't supported, create_table() and add_column() raise vastdb.errors.NotSupportedSchema (error reference). These were all rejected:

"SDK" means the error is raised before any request is sent, because Ibis can't represent the type. The SDK needs Ibis to build predicates.

The cluster has a fixed-size binary type: limits gives its cell size. But SDK 2.1.0 can't create or open a table with one (SDK tests, test_unsupported_types).

Example: a table with several types

This example creates a table with a mix of types, prints the column types the cluster reports, then shows the decimal caveat and its workaround. It uses the connection environment variables from the quickstart.

import decimal
import os

import pyarrow as pa
import pyarrow.compute as pc
import vastdb

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]

columns = pa.schema([
    ("id", pa.int64()),
    ("reading", pa.float16()),
    ("price", pa.decimal128(10, 2)),
    ("day", pa.date32()),
    ("taken_at", pa.timestamp("us", tz="UTC")),
    ("tags", pa.list_(pa.utf8())),
    ("embedding", pa.list_(pa.field("item", pa.float32(), nullable=False), 3)),
    ("location", pa.struct([("lat", pa.float64()), ("lon", pa.float64())])),
    ("attrs", pa.map_(pa.utf8(), pa.utf8())),
])

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).create_schema("ref_data_types", fail_if_exists=False)
    table = schema.create_table("samples", columns, fail_if_exists=False)
    for field in table.columns():
        print(f"{field.name}: {field.type}")

id: int64
reading: halffloat
price: decimal128(10, 2)
day: date32[day]
taken_at: timestamp[us, tz=UTC]
tags: list<item: string>
embedding: fixed_size_list<item: float not null>[3]
location: struct<lat: double, lon: double>
attrs: map<string, string>

Insert two rows, then filter for a price of 0.29:

import datetime as dt

rows = pa.table({
    "id": [1, 2],
    "reading": pa.array([0.5, 1.5], pa.float16()),
    "price": [decimal.Decimal("0.28"), decimal.Decimal("0.29")],
    "day": [dt.date(2026, 1, 1), dt.date(2026, 1, 2)],
    "taken_at": [dt.datetime(2026, 1, 1, 12, tzinfo=dt.timezone.utc),
                 dt.datetime(2026, 1, 2, 12, tzinfo=dt.timezone.utc)],
    "tags": [["a"], []],
    "embedding": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
    "location": [{"lat": 38.7, "lon": -9.1}, {"lat": 59.9, "lon": 10.7}],
    "attrs": [[("colour", "red")], []],
}, schema=columns)

with session.transaction() as tx:
    table = tx.bucket(BUCKET).schema("ref_data_types").table("samples")
    table.insert(rows)
    result = table.select(columns=["id", "price"], predicate=table["price"] == 0.29).read_all()

print(result.sort_by("id").to_pylist())

[{'id': 1, 'price': Decimal('0.28')}]

The row that comes back has price 0.28, not 0.29. To filter on an exact decimal, push down a range one unit of the scale (here 0.01) wider on each side, then compare exactly in Arrow:

with session.transaction() as tx:
    table = tx.bucket(BUCKET).schema("ref_data_types").table("samples")
    wide = table.select(columns=["id", "price"],
                        predicate=table["price"].between(0.28, 0.30)).read_all()

wanted = pa.scalar(decimal.Decimal("0.29"), pa.decimal128(10, 2))
exact = wide.filter(pc.equal(wide["price"], wanted))
print(exact.to_pylist())

[{'id': 2, 'price': Decimal('0.29')}]

Clean up:

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("ref_data_types")
    schema.table("samples").drop()
    schema.drop()
print("Dropped")

Dropped