Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

VAST Catalog

Prev Next

VAST Catalog is a table of metadata about the files, objects and directories on your cluster: names, paths, sizes, owners, timestamps, S3 tags and more, one row per element. Start by asking your platform team to enable the Catalog and, for the audit log, to save audit logs to VAST DataBase, as Access describes. Then run the Quickstart, and read Concepts before you build reports. The guides cover finding files, capacity, change detection, tags and the audit log.

Overview and concepts

Where to begin: the overview and quickstart, then the concepts and architecture the rest of the section relies on, and the release notes.

  • Overview: VAST Catalog and the audit log are system tables of file and object metadata and client operations, queried like your own tables.
  • Quickstart: Open the Catalog with the Python SDK, count elements by type, find the largest recent files and read a Catalog snapshot.
  • Concepts: The source and freshness of Catalog and audit log rows, what a row means, and how they differ from your own tables.

Guides

Step-by-step queries for the questions the Catalog and the audit log answer. Start with Find files; use Capacity reports, Snapshots and change detection, and Tags and metadata for the Catalog, and Query the audit log for operations.

  • Find files: Find files by path, type, age, size, owner and extension, with filters pushed down to the cluster instead of walking directories.
  • Capacity reports: Report capacity by owner, file extension and directory from the Catalog with the Python SDK, and process large subtrees in record batches.
  • Snapshots and change detection: List and read Catalog snapshots, and compare two of them to find files deleted, added or moved between them.
  • Tags and metadata: Find objects by S3 tags and user metadata with the user_tags and user_metadata columns or with user-defined Catalog columns.
  • Query the audit log: Use the Python SDK to see recent operations, who accessed a bucket and which requests failed, from the audit log table.

Integrations

Querying the Catalog and the audit log from SQL engines and dashboards. Start with Trino and Spark for the table names and example queries, then Grafana to chart them.

  • Trino and Spark: Table names and example SQL for querying the Catalog and the audit log through the VAST connectors for Trino and Spark.
  • Grafana: Chart the Catalog and the audit log in Grafana through Trino, with two published community dashboards or panels you build.

Reference

The columns and types of both tables. Use Catalog schema for the Catalog and Audit log schema for the audit log.

  • Catalog schema: The 33 columns of vast_big_catalog_table with their Arrow types, plus user-defined columns and the table names.
  • Audit log schema: Every column of vast_audit_log_table with its type, for all protocols and for S3, NFS and SMB, plus the operation types.

Operations

What to ask your platform team for, and how to keep the Catalog's settings in code. Start with Access; use CI/CD and automation when you manage the snapshot schedule or user-defined columns.

  • Access: Request access to the Catalog and audit log from your platform team, including the identity policy, then check that it works.
  • CI/CD and automation: Keep the Catalog's snapshot schedule, retention and user-defined columns in code with the REST API or vastpy, and check freshness before reports.

Troubleshooting

Symptoms, causes and fixes for the Catalog and the audit log. Start with Troubleshooting VAST Catalog when a table is missing, access is denied or results look wrong.

  • Troubleshooting VAST Catalog: Causes and fixes for a missing Catalog table, denied access, results that look wrong and audit log problems, by symptom.