Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Overview

Prev Next

Applies to VAST Cluster 5.5.

VAST Catalog is a table of metadata about the files, objects and directories on your cluster: names, paths, sizes, owners, timestamps, S3 tags and more, one row per element. The audit log is a table of the protocol operations clients performed on the cluster: who did what, to which path, over which protocol, and whether it succeeded. Both are system tables in VAST DataBase. You query them with the same clients you use for your own tables: the VAST DataBase Python SDK, Trino and Spark.

What it is

  • System tables in VAST DataBase. The Catalog is the table vast_big_catalog_table in the schema vast_big_catalog_schema of the database vast-big-catalog-bucket. The audit log is vast_audit_log_table in vast_audit_log_schema of vast-audit-log-bucket. In the Python SDK, tx.catalog() and tx.audit_log() open them for you.
  • Built from periodic snapshots. The cluster takes snapshots of its data on a schedule and indexes their metadata into the Catalog in the background. So the Catalog shows the namespace as it was at the last indexed snapshot, not this second. See snapshots and freshness.
  • Kept over time. The cluster keeps earlier copies of the Catalog table for a while, as Catalog snapshots, so you can compare the namespace between two points in time. See change detection.
  • Scoped to your tenant. On a cluster with several tenants, your queries return only your tenant's data. Your identity policy can narrow that further, to a path. See access.
  • Maintained by the cluster. You query these tables; the cluster writes them, and refuses inserts from clients.
  • Off until your platform team turns it on. The Catalog is disabled by default, and so is saving the audit log to VAST DataBase. Enabling either is an admin task.

Two system tables in VAST DataBase. Top row: the cluster's files, objects and directories, written over NFS, SMB and S3, are captured in a periodic snapshot (every 30 minutes by default, at most every 15 minutes), and the cluster indexes the snapshot's metadata into the Catalog table in the background. Earlier copies of the Catalog table are kept for a while as Catalog snapshots, named bc_table_ and the date and time. Bottom row: protocol operations in the audited categories are written as records to the audit log table, which keeps them for the retention period, one hour by default. Right: you query both tables with the Python SDK (tx.catalog(), tx.catalog_snapshots(), tx.audit_log()), Trino or Spark. Results are limited to your tenant, and a path-scoped key must filter on search_path inside its path. The cluster writes both tables; you only read them.

The cluster writes two system tables; you query them.

Use cases and considerations

Example use cases:

  • Find files across a namespace of millions of files without walking directories: for example, files older than 90 days and larger than 10 GB under /projects, or everything a user created since last week. See find files.
  • Capacity reports: rank owners by the capacity they use under a path, or rank capacity by file extension. See capacity reports.
  • Find S3 objects by tag or metadata: for example, every object whose processed tag is false. See tags and metadata.
  • Change detection: what was deleted or added between two Catalog snapshots. See snapshots and change detection.
  • Who did what: which user read, wrote or deleted a path, from which client address, and whether it succeeded. See the audit log.
  • Dashboards and scheduled reports through Trino. See Trino and Spark and Grafana.

Important to note:

  • Results lag the file system. The Catalog is as fresh as its last indexed snapshot. By default the cluster takes one every 30 minutes, and your platform team can make it as often as every 15 minutes. Don't use the Catalog for anything that needs to see a file the moment it's written.
  • The audit log keeps records for a limited time. The retention period is a cluster setting, one hour by default. Ask your platform team what yours is before you build on it.
  • Paths up to 1,024 characters. That's the longest path the Catalog supports in 5.5.
  • Not on every cluster. The Catalog can't be enabled on a cluster that uses encryption keys managed through an external key manager (EKM), including per-tenant and per-path keys.
  • Not replicated. The Catalog and the audit log aren't copied by VAST DataBase replication. Query them on the cluster that holds the data. With Global Access, the Catalog has no information about data cached from the remote cluster.
  • Not everything is audited. VAST Event Broker produce, consume and topic operations aren't covered by protocol auditing.

Which tools reach them

Tool Use it for Page
VAST DataBase Python SDK (vastdb) Scripts, notebooks, reports and change detection in Python. tx.catalog(), tx.catalog_snapshots(), tx.audit_log(). Quickstart
Trino SQL, dashboards and BI tools Trino and Spark
Spark Large reports and joins with your own data Trino and Spark
VAST Web UI Ad hoc lookups by an administrator, with CSV export KB: Querying VAST Catalog from the VAST Web UI

VAST Catalog also has a command-line tool, vast-db-cli. These docs don't cover it; see VAST Database CLI Quick Start Guide.

How to read these docs

Where to start

  1. Admin task: send to your platform team. Ask them to enable the Catalog (Configuring VAST Catalog), and, for the audit log, to save audit logs to VAST DataBase (Configuring Global Auditing Settings). Access lists everything to ask for.
  2. Run the quickstart.
  3. Read concepts before you build reports on the Catalog.

See also

Related articles