Applies to VAST Cluster 5.5.
In this quickstart you'll open the VAST Catalog with the VAST DataBase Python SDK, count what's in part of your namespace by element type, find the largest recently modified files, and read an earlier copy of the Catalog from a Catalog snapshot. You only read; nothing is created or changed. It takes about 10 minutes once your platform team has set up access.
Each code block is complete on its own: it imports what it needs and connects. Below each block is the output you should see.
Before you start
What your platform team sets up, in order, before you query.
You need:
- The Catalog enabled, and indexed at least once. It's disabled by default. Admin task: send to your platform team: Configuring VAST Catalog.
- An endpoint and an S3 access key pair whose identity policy allows
s3:TabularQueryDataon the Catalog. Access lists what to ask for. The endpoint and keys work the same way as for your own tables: see the VAST DataBase quickstart and setup prerequisites. - A path to look at that your policy lets you read, ending with
/, for example the path of one of your views. If you leave it out, the examples use/, your whole tenant. - Python 3.10 or later, and the SDK:
pip install vastdb
Put your connection details in environment variables:
export AWS_S3_ENDPOINT_URL="http://vastdb.example.com" # your endpoint
export AWS_ACCESS_KEY_ID="<access key>"
export AWS_SECRET_ACCESS_KEY="<secret key>"
export CATALOG_PATH="/projects/" # a path you can read
Open the Catalog
tx.catalog() opens the Catalog table. Check that it's there and see how many columns it has:
import vastdb
session = vastdb.connect()
with session.transaction() as tx:
catalog = tx.catalog()
names = [field.name for field in catalog.columns()]
print(len(names), "columns, including:", ", ".join(sorted(names)[:8]))
35 columns, including: abac_tags, atime, creation_time, ctime, element_type, extension, gid, group_owner_name
If this fails with a missing table or bucket error, the Catalog isn't enabled or hasn't been indexed yet. If it fails with Forbidden, your key's policy doesn't allow the Catalog. See troubleshooting. All the columns are described in the schema reference.
Count elements by type
Count the elements under your path by type. The search_path column restricts the query to that subtree on the cluster, and columns= asks only for the one column you need, so only that column comes back:
import os
import vastdb
from ibis import _
path = os.environ.get("CATALOG_PATH", "/")
session = vastdb.connect()
with session.transaction() as tx:
rows = tx.catalog().select(
columns=["element_type"],
predicate=_.search_path == path,
).read_all()
counts = rows.group_by("element_type").aggregate([("element_type", "count")])
for row in counts.sort_by("element_type").to_pylist():
print(f"{row['element_type']:<8} {row['element_type_count']:>10,}")
DIR 14
FILE 23
SYMLINK 1
The counting happens in your client, with PyArrow. Files are FILE and directories DIR; you may see other types too. See element types.
Find the largest recent files
Find the five largest files under your path that were modified in the last seven days. The filters on search_path, element_type and mtime all run on the cluster:
import os
from datetime import datetime, timedelta, timezone
import pyarrow.compute as pc
import vastdb
from ibis import _
path = os.environ.get("CATALOG_PATH", "/")
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=7)
session = vastdb.connect()
with session.transaction() as tx:
rows = tx.catalog().select(
columns=["parent_path", "name", "size", "mtime"],
predicate=(_.search_path == path) & (_.element_type == "FILE") & (_.mtime >= since),
).read_all()
print(rows.num_rows, "files modified since", since.date())
top = rows.sort_by([("size", "descending")]).slice(0, 5)
# Catalog times have nanosecond precision, finer than Python's datetime, so format them in Arrow.
when = pc.strftime(top["mtime"], format="%Y-%m-%d %H:%M").to_pylist()
for row, t in zip(top.drop_columns(["mtime"]).to_pylist(), when):
print(f"{row['size']:>14,} {t} {row['parent_path']}{row['name']}")
20 files modified since 2026-10-02
12,582,912 2026-10-09 16:41 /cat-data/projects/apollo/model.bin
4,194,304 2026-10-09 16:41 /cat-data/projects/apollo/part-003.parquet
3,145,728 2026-10-09 16:41 /cat-data/projects/apollo/part-007.parquet
3,145,728 2026-10-09 16:41 /cat-data/projects/apollo/part-001.parquet
2,097,152 2026-10-09 16:41 /cat-data/projects/apollo/part-002.parquet
parent_path ends with /, so parent_path + name is the element's full path. More filters are in find files.
Read a Catalog snapshot
The cluster keeps earlier copies of the Catalog table for a while. tx.catalog_snapshots() lists the snapshots of the Catalog's database; the copies of the Catalog table are the ones whose names start with bc_table_, followed by the date and time. Pass one to tx.catalog() to query the Catalog as it was then:
import os
import vastdb
from ibis import _
path = os.environ.get("CATALOG_PATH", "/")
session = vastdb.connect()
with session.transaction() as tx:
snapshots = sorted(
(s for s in tx.catalog_snapshots() if s.name.split("/.snapshot/")[-1].startswith("bc_table_")),
key=lambda s: s.name,
)
print(len(snapshots), "Catalog snapshots; oldest:", snapshots[0].name.split("/")[-1])
oldest = tx.catalog(snapshot=snapshots[0])
then = oldest.select(columns=["element_type"], predicate=_.search_path == path).read_all().num_rows
now = tx.catalog().select(columns=["element_type"], predicate=_.search_path == path).read_all().num_rows
print("elements under", path, "then:", then, "now:", now)
2 Catalog snapshots; oldest: bc_table_2026-10-09_16_37_09
elements under /cat-data/ then: 2 now: 38
How many snapshots you see, and how far back they go, depends on the Catalog's schedule. With the default schedule a copy is kept for 30 minutes. To compare two snapshots row by row, see snapshots and change detection.
Next steps
- Concepts: how the Catalog is built and how fresh it is.
- Find files and capacity reports.
- The audit log: the same idea for operations instead of files.
- Trino and Spark: the same queries in SQL.