Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Troubleshooting VAST Catalog

Prev Next

Applies to VAST Cluster 5.5.

Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the page with the details. For SDK errors that aren't specific to these tables, see the VAST DataBase error reference.

The table isn't there

Symptom Likely cause Fix
tx.catalog() raises MissingTable: {'bucket': 'vast-big-catalog-bucket', 'schema': 'vast_big_catalog_schema', 'table': 'vast_big_catalog_table'} The Catalog isn't enabled, or its first indexing hasn't finished. It's disabled by default. Ask your platform team to enable it (Configuring VAST Catalog) and to tell you when the first indexing is complete.
tx.catalog_snapshots() raises NotFound (HTTP 404, NoSuchBucket, "The specified bucket does not exist.") The same: the Catalog isn't enabled. As above.
tx.audit_log() raises MissingTable: {'bucket': 'vast-audit-log-bucket', 'schema': 'vast_audit_log_schema', 'table': 'vast_audit_log_table'} Audit logs aren't being saved to VAST DataBase. They can be saved to JSON files instead. Ask your platform team to turn on "Save audit logs to VAST DB" (Configuring Global Auditing Settings).
The Catalog can't be enabled on your cluster The cluster uses encryption keys managed through EKM, including per-tenant or per-path keys. In 5.5 the Catalog can't be enabled then. None in 5.5.
You're on a replication or Global Access destination and the Catalog or audit log has nothing from the source Neither is replicated, and the Catalog has no information about data cached from a Global Access remote cluster. Query them on the cluster that holds the data.

To handle a missing table in code, tx.catalog(fail_if_missing=False) and tx.audit_log(fail_if_missing=False) return None instead of raising MissingTable, also when the Catalog or audit log has never been enabled. tx.catalog_snapshots() has no such option: catch NotFound. When a request fails, the SDK also logs an RPC failed: … line to stderr.

Access denied

Symptom Likely cause Fix
Forbidden (HTTP 403, AccessDenied) from select() on tx.catalog() or tx.audit_log(), although columns() works Your key's identity policy doesn't allow s3:TabularQueryData on vast-big-catalog-bucket (for the Catalog) or vast-audit-log-bucket (for the audit log). Each table needs its own grant. Send your platform team the policy in access.
Forbidden only when you set search_path to some paths Your policy allows only part of the Catalog. You need permission on the subtree you search. Search within the path your policy allows, or ask for more.
Rows from other tenants are missing Expected. Each tenant's users see only that tenant's data. Ask a cluster administrator for cross-tenant reports.
BadRequest with TabularVirtualBucketNotAllowed: command=INSERT_ROWS is not allowed on virtual bucket_name=vast-big-catalog-bucket (or vast-audit-log-bucket) You tried to write to the Catalog or the audit log. The cluster refuses writes to both, whatever your policy allows. Write your results to a table of your own.
A copied example policy is rejected, or doesn't do what you expect A policy with two Resource keys in one statement isn't valid JSON. Use the policy in access.

For how the SDK reports refused requests in general, see Forbidden.

Results look wrong

Symptom Likely cause Fix
A file you just wrote isn't in the Catalog, or one you just deleted still is The Catalog is as fresh as its last indexed snapshot, every 30 minutes by default. Wait for the next snapshot. Check the newest snapshot's time; see snapshots and freshness.
A new user-defined column is missing or empty User-defined columns are filled in from the next snapshot after they're added. Wait for the next snapshot.
catalog_snapshots() returns only one or two copies, or none Copies are kept only for the "keep catalog copies" period, 30 minutes by default. Ask for a longer retention, or save your own history; see change detection.
catalog_snapshots() returns snapshots tx.catalog(snapshot=...) can't open The list includes snapshots that aren't copies of the Catalog table. Use only the snapshots named bc_table_<date and time>; see list the Catalog snapshots.
A query returns rows from the whole tenant and is slow No search_path filter. Add search_path with the subtree you need; see search_path.
Files with very long paths are missing The Catalog supports paths up to 1,024 characters in 5.5. None in 5.5.
A filter on a tag or metadata raises NotImplementedError The SDK can't push down a filter on the user_tags or user_metadata maps, not even isnull(). Filter in your client, or ask for a user-defined column; see tags and metadata.

The audit log

Symptom Likely cause Fix
The audit log is empty, or has nothing for the operation you're looking for That protocol or category of operation isn't audited, globally or in the view's view policy, or the record has passed its retention period (one hour by default). It can also take a short time for the first records to appear after auditing is turned on. Ask your platform team what's audited for that view and how long records are kept. See concepts: the audit log.
No VAST Event Broker operations in the audit log Event Broker produce, consume and topic management aren't covered by protocol auditing. None.
login_name is always empty Logging user names is off by default, and needs the name to be available from the authentication provider. With it off, login_name is an empty string. Ask your platform team to turn on "Log Username", or use uid.
A filter on status == "Success" misses NFS operations NFS records carry the NFS status code, such as NFS3_OK or NFS3ERR_ACCES, not Success. Filter on the status values of each protocol; see the audit log schema.
Bytes written by an S3 multipart upload are missing Uploading the parts leaves no records. The start and completion of the upload are recorded, with num_bytes 0. Take object sizes from the Catalog instead.
On a user-impersonated view, records have a UID but no login name or SID A known limitation in 5.5: only the original user's UID is logged. Map the UID yourself.
path is empty but view_path is set The cluster is set not to log full paths. view_path is then the view and name the element's path inside it (for NFS, only the file name). Ask for full paths to be logged.
Presigned POST uploads appear as PUT_OBJECT In 5.5, the audit log records a presigned POST upload as a PUT_OBJECT operation. Treat them as uploads.
The Web UI's audit log page shows a random selection of rows when you sort, or negative values in some fields In 5.5, with more than 1,000 matching rows, the Web UI (and the VMS REST API) can't sort the results, and some fields show raw internal values. Query the table with the SDK, Trino or Spark; see query the audit log.

See also

Related articles