Capacity reports

Applies to VAST Cluster 5.5.

This guide builds typical Catalog reports: which owners use the most capacity in a path, which file extensions take the most space, and which directories are largest. The cluster filters; your client adds up.

Each code block is complete on its own. They read your connection settings from the environment, as in the quickstart, and use CATALOG_PATH as the subtree to report on.

Capacity by owner

Rank the owners under your path by the bytes their files use:

import os

import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["owner_name", "size", "used"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    ).read_all()

report = rows.group_by("owner_name").aggregate(
    [("size", "count"), ("size", "sum"), ("used", "sum")]
).sort_by([("used_sum", "descending")])

print(f"{'owner':<20} {'files':>10} {'size MiB':>10} {'used MiB':>10}")
for row in report.slice(0, 10).to_pylist():
    print(f"{str(row['owner_name']):<20} {row['size_count']:>10,} "
          f"{row['size_sum'] / 1024**2:>10.2f} {row['used_sum'] / 1024**2:>10.2f}")

owner                     files   size MiB   used MiB
cat-app                      23      55.12      55.12

size is the element's size, and used is the number of bytes it uses on disk. Report the one that answers your question: size for "how much data", used for "how much capacity". An owner with no user name shows as None; group by uid instead if your data is written over NFS by users without names.

Capacity by extension

Rank file extensions by size:

import os

import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["extension", "size"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    ).read_all()

report = rows.group_by("extension").aggregate(
    [("size", "count"), ("size", "sum")]
).sort_by([("size_sum", "descending")])

for row in report.slice(0, 10).to_pylist():
    print(f"{str(row['extension']):<10} {row['size_count']:>10,} files {row['size_sum'] / 1024**2:>10.2f} MiB")

bin                 2 files      37.00 MiB
parquet             8 files      16.00 MiB
csv                 5 files       1.40 MiB
log                 5 files       0.36 MiB
pdf                 1 files       0.29 MiB
tmp                 1 files       0.06 MiB
txt                 1 files       0.00 MiB

Capacity by directory

Sum file sizes by the directory directly under your path. parent_path holds the whole directory path, so take the next path component in your client:

import os

import pyarrow.compute as pc
import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["parent_path", "size"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    ).read_all()

relative = pc.utf8_slice_codeunits(rows["parent_path"], len(path))
top = pc.list_element(pc.split_pattern(relative, "/"), 0)
rows = rows.append_column("top", top)

report = rows.group_by("top").aggregate([("size", "count"), ("size", "sum")]).sort_by([("size_sum", "descending")])
for row in report.slice(0, 10).to_pylist():
    print(f"{path}{row['top'] or '.'}  {row['size_count']:>10,} files {row['size_sum'] / 1024**2:>10.2f} MiB")

/cat-data/projects          13 files      28.62 MiB
/cat-data/archive           3 files      25.79 MiB
/cat-data/logs           5 files       0.36 MiB
/cat-data/reports           1 files       0.29 MiB
/cat-data/scratch           1 files       0.06 MiB

Files directly in your path show as ..

Large namespaces

The examples above read every file row under the path into your client before they add up. For a large subtree, process the result in record batches instead, so only one batch is in memory at a time:

import os
from collections import Counter

import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

used_by_owner = Counter()
session = vastdb.connect()
with session.transaction() as tx:
    reader = tx.catalog().select(
        columns=["owner_name", "used"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    )
    for batch in reader:
        for owner, used in zip(batch.column("owner_name").to_pylist(), batch.column("used").to_pylist()):
            used_by_owner[owner] += used or 0

for owner, used in used_by_owner.most_common(5):
    print(f"{str(owner):<20} {used / 1024**2:>10.2f} MiB")

cat-app                   55.12 MiB

Read all the batches inside the transaction. For whole-tenant reports on a schedule, Trino or Spark can do the grouping for you; see Trino and Spark.

Capacity reports

Capacity by owner

Capacity by extension

Capacity by directory

Large namespaces

See also