Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Concepts

Prev Next

Applies to VAST Cluster 5.5.

Read this before you build reports on the Catalog or the audit log. It explains where the rows come from, how fresh they are, what one row means, and how these tables behave like, and unlike, your own VAST DataBase tables.

Snapshots and freshness

Files, objects and tables are all elements in the cluster's Element Store, and the Catalog is an index of their metadata (background: the cluster underneath). The Catalog isn't updated as files change. Instead, when the Catalog is enabled:

  1. The cluster takes a snapshot of its data on a schedule.
  2. It indexes the metadata in that snapshot into the Catalog table, in the background.
  3. It keeps the indexed copy for a set time, as a Catalog snapshot you can still query.

So each query sees the namespace as it was at the snapshot behind the current Catalog table. A file written since then isn't there yet, and a file deleted since then is still there.

By default, the cluster takes a snapshot every 30 minutes and keeps each copy of the Catalog table for 30 minutes. Your platform team can change both, down to a 15-minute interval. They can also choose to keep the file-system snapshots the Catalog was built from. Changes to how long copies are kept apply to snapshots taken after the change. See Configuring VAST Catalog.

What that means for you:

  • Check what time your answer is from. For time-sensitive reports, list the Catalog snapshots and note the newest one's time with the result. See change detection.
  • History is as long as the retention. You can compare two points in time only while both copies are kept. With the defaults, that's a short window. If you need daily comparisons, ask your platform team for a longer "keep catalog copies" period, or save your own results to a table of your own.
  • New columns appear after the next snapshot. A user-defined column is filled in from the next snapshot onwards.

The first indexing after the Catalog is enabled runs as a background task. Until it completes, the Catalog isn't ready to query. Your platform team can watch its progress in the Web UI.

What a row is

Each row of the Catalog is one element: a file, an S3 object, a directory or another type of element, in one tenant's namespace.

The columns describe the element as all of its protocols see it:

Group Columns
Where it is name, extension, parent_path, symlink_path, and the virtual column search_path
What it is element_type, size, used, nlinks
When creation_time, mtime, atime, ctime
Who owns it uid, owner_name, owner_sid, login_name, gid, group_owner_name, group_owner_sid
Permissions and locks nfs_mode_bits, name_aces_exist, s3_locks_legal_hold, s3_locks_retention
Tags and metadata user_tags, user_tags_count, user_metadata, abac_tags, nfs4_xattrs, and any user-defined columns

The full list, with types, is in the schema reference.

parent_path is the directory the element is in, ending with /. Join it with name to get the full path.

Element types

element_type says what kind of element a row is: FILE (a file or S3 object), DIR (a directory), SYMLINK (a symbolic link), and SCHEMA and TABLE for VAST DataBase schemas and tables. Count the types on your cluster with the quickstart. Most reports filter on element_type == "FILE", so that directories don't count towards file counts and sizes. Directories the cluster creates for you are rows too: for example, the directories implied by S3 object keys, and the .vast directory in each S3 bucket, owned by UID 0.

search_path

search_path is a virtual column: it isn't a property of an element, and filtering on it restricts the query to a subtree. A query with search_path == "/some/path/" reads only the part of the Catalog under /some/path/. Your identity policy must let you read that subtree.

Use it in almost every query. It's how you tell the cluster which part of the namespace you care about, so it doesn't scan the rest. In the Python SDK:

predicate = (_.search_path == "/projects/") & (_.element_type == "FILE")

and in SQL, WHERE search_path = '/projects/'.

Tenants and paths

On a cluster with several tenants, each tenant's users see only that tenant's elements in the Catalog. Your identity policy can narrow what you see further, to one or more paths. Cluster administrators can query every tenant's data. See access.

Indexed and user-defined columns

The Catalog indexes a fixed set of attributes as columns: the ones above. All of an element's S3 tags are in user_tags, a map of key to value, and its S3 user metadata (x-amz-meta-*) is in user_metadata.

Your platform team can also add user-defined columns, one per tag key, metadata key or NFSv4.2 extended attribute that you use to classify data:

Type Column name
S3 tag <key> tag_<key>
S3 user metadata x-amz-meta-<key> metadata_<key>
NFSv4.2 extended attribute <key> nfs4_xattr_<key>

A user-defined column holds the value for that key as an ordinary string column, so you can filter on it on the cluster like any other column. A filter on a key inside the user_tags map can't be pushed down by the Python SDK: you fetch the map and filter in your client. That makes user-defined columns worth asking for when you filter on the same keys often. See tags and metadata.

Like your own VAST DataBase tables, and unlike them

The Catalog and the audit log are VAST DataBase tables, so most of what you know carries over:

  • Same clients and connection. Connect with the same endpoint and keys as for your own tables. See setup prerequisites.
  • Same query rules. Ask only for the columns you need, and write filters that push down to the cluster: comparisons, isin, between, startswith, contains and isnull, combined with & and |. See querying and pushdowns. Aggregation (counts, sums, group by) happens in your client or in the SQL engine, not in the cluster.
  • Same transactions. Open the tables inside session.transaction().

What's different:

  • You don't write to them. The cluster maintains both tables and refuses inserts into them, whatever your policy allows (TabularVirtualBucketNotAllowed).
  • Their names are fixed. tx.catalog() and tx.audit_log() open them without you naming the database, schema or table. In SQL, use the names in Trino and Spark.
  • The Catalog is a point-in-time copy, as described above. Its snapshots are listed with tx.catalog_snapshots(), not with bucket.snapshots() on one of your databases.
  • They aren't replicated to another cluster.

The audit log

Protocol auditing records operations that clients perform over SMB, NFSv3, NFSv4, S3 and VAST DataBase. When your platform team chooses to save audit logs to VAST DataBase, each audited operation is a row in the audit log table. (They can also, or instead, write the records to JSON files in an audit directory. These docs cover the table.)

Each row records one operation:

  • When: time (UTC).
  • Who: uid, sid, login_name, and for S3 s3_used_access_key.
  • From where: client_ip, and the cluster side: cluster_vip, cnode_name.
  • What: protocol (S3, NFSv3, NDB for VAST DataBase, and so on) and rpc_type (the operation, such as PUT_OBJECT or QUERY_DATA), on object_type.
  • On what: path (the full path, when the cluster logs full paths), view_path (the view the request came through), name (the path inside the view), and protocol-specific fields such as s3_bucket_name.
  • Result: status. For S3 and VAST DataBase, Success or the error code, such as NoSuchKey; for NFS, the NFS status, such as NFS3_OK.
  • How much: num_bytes.

Fields that don't apply to a protocol are empty. The full list is in the audit log schema.

What gets recorded is a cluster setting:

  • Which operations. Auditing is set per protocol and per category of operation: create and delete, modify data, modify metadata, read data, read metadata, and session create and close. A global setting applies to every view, and a view policy can add more for its views, but not remove any. For the operations in each category, including the VAST DataBase ones (for example QUERY_DATA, INSERT_ROWS, CREATE_TABLE), see Audited Protocol Operations.
  • Which details. Full paths are logged by default. User names (login_name) are logged only if your platform team turns that on. Without full paths, path is empty; without user names, login_name is an empty string.
  • For how long. Records are kept for a retention period, which defaults to one hour, or forever.

Your own reads of the Catalog and the audit log are audited too, when VAST DataBase operations are: each read request is a QUERY_DATA record. A dashboard that refreshes often adds many rows.

So before you rely on the audit log for a question, ask your platform team which protocols and categories are audited for the paths you care about, and how long records are kept. An operation that isn't audited leaves no row, and that looks the same as an operation that never happened.

See also

Related articles