Snapshots and change detection

Applies to VAST Cluster 5.5.

The cluster keeps earlier copies of the Catalog table as Catalog snapshots. Comparing two of them tells you what changed in between: which files were deleted, which were added, and, with care, which were moved. This guide lists the snapshots, reads one, and compares two.

Each code block is complete on its own. They read your connection settings from the environment, as in the quickstart, and use CATALOG_PATH as the subtree to compare.

List the Catalog snapshots

tx.catalog_snapshots() returns the snapshots of the Catalog's database, vast-big-catalog-bucket, as Bucket objects named vast-big-catalog-bucket/.snapshot/<name>. The copies of the Catalog table are the ones named bc_table_ followed by the date and time; the list may include other snapshots too. Sort them by name to put them in time order:

import vastdb

session = vastdb.connect()
with session.transaction() as tx:
    names = sorted(
        s.name.split("/.snapshot/")[-1]
        for s in tx.catalog_snapshots()
        if s.name.split("/.snapshot/")[-1].startswith("bc_table_")
    )

print(len(names), "Catalog snapshots")
for name in names:
    print(name)

2 Catalog snapshots
bc_table_2026-10-09_16_51_02
bc_table_2026-10-09_17_06_02

How many there are depends on the Catalog's schedule: one per interval, each kept for the "keep catalog copies" period. With the default schedule (every 30 minutes, kept for 30 minutes) you'll see only one or two. To compare across hours or days, ask your platform team for a longer retention; see snapshots and freshness.

Read a snapshot

Pass a snapshot to tx.catalog() to query the Catalog as it was then. Everything else works as on the current Catalog: columns, filters and search_path. Snapshots are read-only.

import os

import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    snapshots = sorted(
        (s for s in tx.catalog_snapshots() if s.name.split("/.snapshot/")[-1].startswith("bc_table_")),
        key=lambda s: s.name,
    )
    oldest = snapshots[0]
    rows = tx.catalog(snapshot=oldest).select(
        columns=["size"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    ).read_all()

print(oldest.name.split("/")[-1], rows.num_rows, "files,", sum(rows["size"].to_pylist()), "bytes")

bc_table_2026-10-09_16_51_02 23 files, 57794560 bytes

Find deleted and added files

To find what changed, read the same columns with the same filter from two snapshots, and compare the full paths. A path in the older snapshot but not the newer one was deleted (or moved away); a path only in the newer one was added (or moved in).

Two Catalog snapshots compared: the example output below, as a picture.

This example needs at least two Catalog snapshots, with a file deleted and a file added under your path between them:

import os

import pyarrow.compute as pc
import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

def full_paths(table):
    return pc.binary_join_element_wise(table["parent_path"], table["name"], "")

session = vastdb.connect()
with session.transaction() as tx:
    snapshots = sorted(
        (s for s in tx.catalog_snapshots() if s.name.split("/.snapshot/")[-1].startswith("bc_table_")),
        key=lambda s: s.name,
    )
    older, newer = snapshots[0], snapshots[-1]
    query = dict(
        columns=["parent_path", "name"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    )
    before = full_paths(tx.catalog(snapshot=older).select(**query).read_all())
    after = full_paths(tx.catalog(snapshot=newer).select(**query).read_all())

deleted = pc.filter(before, pc.invert(pc.is_in(before, value_set=after)))
added = pc.filter(after, pc.invert(pc.is_in(after, value_set=before)))

print("from", older.name.split("/")[-1], "to", newer.name.split("/")[-1])
print("deleted:", sorted(deleted.to_pylist())[:10])
print("added:", sorted(added.to_pylist())[:10])

from bc_table_2026-10-09_16_51_02 to bc_table_2026-10-09_17_06_02
deleted: ['/cat-data/archive/2025/notes.txt', '/cat-data/scratch/to-delete.tmp']
added: ['/cat-data/archive/2025/notes-2025.txt', '/cat-data/projects/gemini/sales-5.csv']

A few things to keep in mind:

Compare like with like. Use the same search_path and the same filters for both snapshots. A filter on a value that changes, such as mtime, makes unchanged files look added or deleted.
Changes between snapshots only. A file created and deleted between two snapshots appears in neither.
Sizes and times too. To find modified files, add mtime or size to the columns and compare them for paths that are in both.

Find moved files

A rename changes an element's path, so the comparison above shows a moved file as one deletion and one addition. The phandle column identifies the element itself, and a renamed file keeps it, so matching on it pairs each old path with its new one:

import os

import pyarrow.compute as pc
import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

def paths_by_handle(table):
    flat = table.flatten()
    return flat.select(["phandle.clone_id", "phandle.handle_id"]).append_column(
        "path", pc.binary_join_element_wise(flat["parent_path"], flat["name"], "")
    )

session = vastdb.connect()
with session.transaction() as tx:
    snapshots = sorted(
        (s for s in tx.catalog_snapshots() if s.name.split("/.snapshot/")[-1].startswith("bc_table_")),
        key=lambda s: s.name,
    )
    query = dict(
        columns=["phandle", "parent_path", "name"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    )
    before = paths_by_handle(tx.catalog(snapshot=snapshots[0]).select(**query).read_all())
    after = paths_by_handle(tx.catalog(snapshot=snapshots[-1]).select(**query).read_all())

keys = ["phandle.clone_id", "phandle.handle_id"]
joined = before.join(after, keys=keys, left_suffix="_before", right_suffix="_after")
moved = joined.filter(pc.not_equal(joined["path_before"], joined["path_after"]))
for row in moved.slice(0, 10).to_pylist():
    print(row["path_before"], "->", row["path_after"])

/cat-data/archive/2025/notes.txt -> /cat-data/archive/2025/notes-2025.txt

Moves into or out of your search_path show up only on one side, so run the comparison on a path that contains both the old and the new location.

Keep your own history

Catalog snapshots are deleted when their retention ends. If you need a longer history, for example a daily capacity trend, run your report on a schedule and insert the result into a table of your own in VAST DataBase. See ingest.