--- title: "Querying the VAST Catalog" slug: "querying-the-vast-catalog" description: "Learn to query the VAST Catalog using VMS REST API and VAST DB SDK for efficient access to files, directories, and metadata in your cluster." updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/querying-the-vast-catalog" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Querying the VAST Catalog A guide to querying the VAST Catalog — the cluster's built-in, queryable index of every file, directory, object, table, and symlink — through the database. It starts with the path that works with nothing but VMS credentials, then adds the VAST DB SDK for programmatic access. > [!NOTE] > **Scope.** This guide covers database access to the catalog: the VMS REST API and the VAST DB SDK (plus the Web UI where your version exposes a catalog grid). External Trino query engines and Row and Column Security are out of scope ([Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1)), and so is point-in-time querying of catalog snapshots. For the audit log — the *other* built-in queryable table, which records what happened rather than what exists — see the companion guide *Querying the VAST AuditDB*. This is a getting-started guide. The official documentation is the reference for concepts, the full schema, and Web UI procedures — start with the [VAST Catalog Overview](/docs-5-5/docs/vast-catalog-overview); the other official pages are linked from the sections here that provide overviews ([Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1), [Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)) and are collected in [Section 6](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1). --- ## What is the VAST Catalog? The VAST Catalog (also called the Big Catalog) is a VAST Database table the cluster maintains about its own namespace: one row per element (file, directory, object, DB table, symlink...), with its path, type, size, ownership, timestamps, and S3 user tags/metadata. Instead of walking a filesystem over NFS or listing buckets over S3, which can be slow and inefficient, you query the index: "the 100 largest files under `/projects`", "everything owner X changed this week", "every object tagged `retention=forever`", each as one query. The catalog is an **index, not a live view**: the cluster refreshes it on a schedule you configure (typically every 15–30 minutes, [Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)), so very recent changes appear after the next refresh. --- ## Ways to query it | Method | What you need | Section | | --- | --- | --- | | **VMS REST API** (vastpy) | VMS credentials or an API token | [Section 2, Path A](/develop/docs/querying-the-vast-catalog#path-a-the-vms-rest-api1) | | **VAST DB SDK** | An S3 key and an `s3:Tabular*` identity policy | [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1) | | **Web UI** | A VMS login (where your version exposes a catalog grid) | [Section 2, Path B](/develop/docs/querying-the-vast-catalog#path-b-the-web-ui1) | The VMS REST API reads the catalog through the management plane and works the moment the catalog is enabled — and it sees the **whole cluster**, every tenant. The SDK reads the same table via the S3/tabular data endpoint and is automatically **scoped to the querying key's tenant** (see [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1), *Tenant scoping is automatic*). Start with [Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1) for a quick look; use the SDK for anything programmatic. --- ## Read in this order | # | Section | What you get | | --- | --- | --- | | 0 | [Enable the catalog](/develop/docs/querying-the-vast-catalog#enable-it-or-change-the-cadence) | Turn the catalog on and set its refresh cadence. Skip if already enabled. | | 1 | [Overview & prerequisites](/develop/docs/querying-the-vast-catalog#1-overview-prerequisites1) | What must be true first, the access methods, and how to choose. | | 2 | [Query the catalog](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1) | The VMS REST API path. Get data now. | | 3 | [Troubleshooting](/develop/docs/querying-the-vast-catalog#3-troubleshooting1) | `MissingBucket`, empty results (trailing slashes, tenants, staleness), slow queries. | | 4 | [Schema & query recipes](/develop/docs/querying-the-vast-catalog#4-schema-query-recipes1) | All columns, `element_type` values, and copy-paste queries. | | 5 | [Advanced: SDK access & tenancy](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1) | Programmatic access, the identity policy, and how tenant scoping works. | | 6 | [Appendix: CLI, config links & verification](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1) | vastpy cheat-sheet, official doc links, verification checklist. | Runnable starters, in the same easy-to-expert order: `examples/query_vms_catalog.py` (VMS API path), `examples/first_query.py` (SDK first steps: list a directory, `--types`, `--schema`), and `examples/catalog_report.py` (SDK: largest files, by owner, changed-since, by extension, tagged elements). --- ## The 60-second mental model 1. **Turn the catalog on** and pick a refresh cadence ([Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)). 2. **Query it right away** through the VMS REST API — `bigcatalogconfig/query_data`, with a`path` subtree anchor and at least one filter ([Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)). 3. **For SDK access**, give the user an identity policy with the tabular read actions (`s3:TabularList*`, `s3:TabularQueryData`) and open the table with `tx.catalog()` ([Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1)). 4. **Always cap the read.** Every SDK query is a scan over a table with one row per element on the cluster; `limit_rows` both cap the fetch and let the scan stop early, and low-match queries pay a full sweep (~15 s on a 3-billion-row lab catalog) no matter how selective the predicate looks ([Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1)). --- ## Coming from the AuditDB guide? The access mechanics are identical — same managed-bucket pattern, same identity policy, same`tx.<well-known-name>()` open — but three semantics flip: | | AuditDB | Catalog | | --- | --- | --- | | A row is | an **event** (something happened) | an **element** (something exists) | | SDK tenant visibility | cluster-wide — filter on the `tenant` column yourself | **auto-scoped** to the key's tenant, server-side | | Freshness | near-real-time stream | periodic index (minutes behind) | | The fast query | time window + `limit_rows` | `parent_path` anchor (trailing slash!) + a `limit_rows` the scan can satisfy early | --- ## Conventions - The catalog lives at fixed, well-known coordinates on every cluster:`vast-big-catalog-bucket` / `vast_big_catalog_schema` / `vast_big_catalog_table`. The SDK has them built in (`tx.catalog()`), so most code never spells them out. - Version-sensitive values (like the `element_type` list) are called out in place and collected in the verification checklist in [Section 6](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1). - "Identity policy" and "S3 policy" are the same object in VAST (the `s3policies` API resource). - Cross-references like "[Section 3](/develop/docs/querying-the-vast-catalog#3-troubleshooting1)" or "Recipe 5" are deliberately plain text, not hyperlinks: these files get pasted into GitLab snippets and tickets, where relative links between files break. The only live links are absolute `https://` URLs and anchors within the same file. --- ## Code examples The runnable example scripts this guide refers to — `query_vms_catalog.py` (VMS REST API path), `first_query.py`, and `catalog_report.py` (VAST DB SDK) — live in a companion snippet, with their own README covering setup and a quick tour: ## 0. Enable the Catalog This page turns the VAST Catalog on and sets how often it refreshes. On many clusters it is already enabled — check first; it takes one call. (The official reference for this area, including the Web UI procedure, is [Configuring VAST Catalog](/docs-5-5/docs/configuring-vast-catalog).) The catalog is driven by one config object, the **Big Catalog policy** (`bigcatalogconfig` in the VMS API). It has two knobs that matter: 1. `enable` — whether the cluster maintains the catalog at all. 2. `frames` — the refresh schedule: how often the cluster snapshots its namespace into the catalog table (`every`), and how long those catalog snapshots are retained (`keep-local` / `keep-remote`). The `every` interval is your **data freshness**: an element created a moment ago appears in query results after the next frame. --- If you haven't used the vastpy-cli before, check out this KB article. [Getting Started with vastpy-cli](/develop/docs/getting-started-with-vastpy-cli) ## [](/develop/docs/querying-the-vast-catalog#check-whether-its-already-on)Check whether it's already on ```plaintext pip install vastpy vastpy-cli --address --user admin --password '********' get bigcatalogconfig ``` Healthy output looks like this (from a VAST 5.4.4 lab cluster): ```plaintext [{  "id": 1,  "name": "Vast Catalog Policy",  "enable": true,  "state": "working",  "big_catalog": true,  "prefix": "big_catalog",  "clone_type": "LOCAL",  "frames": [{"every": "30m", "start-at": "2026-04-28 20:01:25",              "keep-local": "1H", "keep-remote": "30m"}] }] ``` `enable: true` plus `state: "working"` means the catalog is being maintained and is currently queryable — skip to [Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1). --- ## [](/develop/docs/querying-the-vast-catalog#enable-it-or-change-the-cadence)Enable it (or change the cadence) Patch the policy by its `id` from the check above (typically `1`). This example enables the catalog with a 15-minute refresh — **reuse the** `start-at` **value the GET returned** so the schedule anchor doesn't move: ```plaintext vastpy-cli --address --user admin --password '********' \  patch bigcatalogconfig/1 \  enable=true \  frames='[{"every": "15m", "every_value": "15m", "every_unit": "m",            "start-at": "2026-04-28 20:01:25",            "keep-local": "2H", "local-period": "H",            "keep-remote": "15m", "remote-period": "m"}]' ``` (`vastpy-cli` parses values that look like JSON as JSON, so the `frames` list goes through as a real structure. If you'd rather keep the body in a file, the same call is `vastpy-cli ... patch bigcatalogconfig/1 -i body.json`.) Then re-run the check above and confirm `enable: true` and `state: "working"`. Cadence trade-off: a shorter `every` means fresher query results and more frequent indexing work on the cluster. The cluster enforces a **15-minute minimum** (the default is 30), and 15–30 minutes is the common range; whatever you choose, remember it when interpreting results — "no rows for the file I just wrote" usually just means "next frame hasn't run yet" ([Section 3, 2c](/develop/docs/querying-the-vast-catalog#2c-the-element-is-newer-than-the-last-refresh1)). The same policy can be managed from the Web UI's Settings on versions that expose it — the field names match the JSON above. The official [Configuring VAST Catalog](/docs-5-5/docs/configuring-vast-catalog) guide documents that path, plus two things this guide deliberately doesn't cover: indexing **user-defined attributes** (S3 tags/metadata as extra catalog columns) and monitoring indexing progress via restore points. This guide uses the CLI because it is consistent across versions; see the verification checklist in [Section 6](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1). --- ## [](/develop/docs/querying-the-vast-catalog#what-you-get)What you get Once enabled, the cluster maintains the catalog table at its fixed coordinates (`vast-big-catalog-bucket` / `vast_big_catalog_schema` / `vast_big_catalog_table`) with one row per element in the namespace. No view, no bucket setup, no per-tenant work — tenancy is handled automatically at query time ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)). ## 1. Overview & Prerequisites ### [](/develop/docs/querying-the-vast-catalog#prerequisites)Prerequisites One admin setting underpins everything in this guide: 1. **The catalog is enabled and healthy**: `bigcatalogconfig` shows `enable: true` and `state: "working"` ([Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)). With that in place, the catalog is immediately queryable via the VMS REST API ([Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)). The VAST DB SDK reads the same table over the S3/tabular data endpoint; it needs an S3 access key whose user has an `s3:Tabular*` identity policy, covered in [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1). Also worth knowing before your first query: - **Results are as fresh as the last refresh frame.** The catalog is rebuilt on the `frames.every` cadence (15–30 minutes typically), not on every write ([Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)). - **What you see depends on the door you use.** The VMS API reads through the management plane and shows the whole cluster — every tenant. The SDK is automatically scoped to the tenant of the querying key ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)). ### [](/develop/docs/querying-the-vast-catalog#catalog-table-coordinates-sdk-path-only)Catalog table coordinates (SDK path only) The catalog has the same well-known coordinates on every cluster: | Piece | Value | Notes | | --- | --- | --- | | Bucket | `vast-big-catalog-bucket` | A **managed** bucket: open it by name; it doesn't appear in S3 bucket listings ([Section 5, *The catalog is a managed bucket*)](/develop/docs/querying-the-vast-catalog#the-catalog-is-a-managed-bucket1). | | Schema | `vast_big_catalog_schema` | Fixed. | | Table | `vast_big_catalog_table` | Fixed. | | DB data endpoint | `http://<cluster-vip>` | The VAST DB data VIP, not the VMS management address. | The SDK has these names built in — `tx.catalog()` returns the table handle directly. The VMS API path needs none of this; it addresses the catalog through the management plane. --- ### [](/develop/docs/querying-the-vast-catalog#choose-your-access-method)Choose your access method | Method | Auth | Setup | Sees | Best for | | --- | --- | --- | --- | --- | | **VMS REST API** (vastpy) | VMS creds or token | Low | Whole cluster (admin view) | Admin scripts, cluster-wide reports, a gated internal service | | **VAST DB SDK** | S3 key + `s3:Tabular*` identity policy | Low | The key's tenant | Analysts, pipelines, tools — anything programmatic | | **Web UI** | VMS login | None | Whole cluster (admin view) | Spot-checks, where your version exposes a catalog grid | - **Cluster-wide question, admin hat on?** ("what's the biggest data on the cluster?", "how many DB tables exist across tenants?") — the VMS REST API ([Section 2, Path A](/develop/docs/querying-the-vast-catalog#path-a-the-vms-rest-api1)). - **Building a tool, or answering questions inside one tenant's namespace?** — the VAST DB SDK ([Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1)); the only setup is a key and one identity policy, and tenancy is enforced for you. - **Handing a team self-service access to *their* data only?** Give them SDK access using a key from their tenant — the server handles scoping. Within a single tenant, the query grant is table-level, so finer path-level scoping needs a mediated service, the same pattern as the AuditDB guide's [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1). --- ### [](/develop/docs/querying-the-vast-catalog#two-audiences-two-starting-points)Two audiences, two starting points - **Administrators/platform team**: [Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1) to get data flowing today, then [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1) to set up SDK access for users and tools. - **Analysts**/**tool builders**: start at [Section 4](/develop/docs/querying-the-vast-catalog#4-schema-query-recipes1) for the schema and recipes, and keep [Section 3](/develop/docs/querying-the-vast-catalog#3-troubleshooting1) (Troubleshooting) nearby — the trailing slash and the refresh cadence cause most first-day confusion. ## 2. Query the Catalog The VMS REST API reads the catalog using nothing more than VMS credentials through the management plane as soon as the catalog is enabled — and it sees every tenant's namespace. Start here, and move to the SDK path ([Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1)) when you need bulk extracts or want to embed queries in tooling. --- ### [](/develop/docs/querying-the-vast-catalog#path-a-the-vms-rest-api)Path A: the VMS REST API The endpoint is `bigcatalogconfig/query_data` (a POST). It requires two things on every call, and rejects the request with a `400` naming the missing field otherwise: - `path` — the subtree to search. It anchors the query and is **recursive**: `path=/` searches the whole namespace, `path=/projects/` everything below it. - `filters` — at least one server-side row filter (shape below). Plus the usual: `fields` (columns to return) and `limit`. The quickest way to run one is `vastpy-cli`: ```plaintext vastpy-cli --address var202.selab.vastdata.com --user admin --password '********' \  post bigcatalogconfig/query_data path=/ limit=20 \  filters='{"element_type":[{"eq":"TABLE"}]}' \  fields='["parent_path","name","element_type","size","mtime","owner_name"]' ``` The same call from Python, for anything scripted (vastpy is the same client under the CLI): ```plaintext import urllib3; urllib3.disable_warnings() from vastpy import VASTClient c = VASTClient(address="var202.selab.vastdata.com", user="admin", password="********") resp = c.bigcatalogconfig.query_data.post(    path="/",    limit=50,    fields=["parent_path", "name", "element_type", "size", "mtime", "owner_name"],    filters={        "element_type": [{"eq": "TABLE"}],          # every DB table on the cluster        # "name":       [{"startswith": "report"}], # startswith also verified on 5.4.4    }, ) # resp is {"prop_list": [...columns...], "results": [[...row...], ...], "count": N} cols = resp["prop_list"] for row in resp["results"]:    print(dict(zip(cols, row))) ``` A ready-made, filterable version is in `examples/query_vms_catalog.py`: ```plaintext export VMS_ADDRESS=var202.selab.vastdata.com VMS_USER=admin VMS_PASSWORD='********' python query_vms_catalog.py --path / --type TABLE --top 50 python query_vms_catalog.py --path /home/ --name-prefix report --top 20 ``` #### [](/develop/docs/querying-the-vast-catalog#authenticate-to-vms)Authenticate to VMS Point vastpy at the management host — the same hostname you open the GUI with, as a bare name (vastpy adds `https://` itself): ```plaintext export VMS_ADDRESS='var202.selab.vastdata.com' export VMS_USER='admin' export VMS_PASSWORD='********' # or, VAST 5.3+:  export VMS_TOKEN='' ``` The S3 data endpoint is a different address and only speaks S3; pointing vastpy at it returns `400 CredentialsNotSupported`. #### [](/develop/docs/querying-the-vast-catalog#filter-syntax)Filter syntax `filters` is shaped `{"<column>": [{"<operator>": <value>}]}`, with the snake_case column names from [Section 4](/develop/docs/querying-the-vast-catalog#4-schema-query-recipes1). Verified operators on VAST 5.4.4: `eq` and `startswith`. Operators the audit endpoint's docs suggest, but this endpoint rejects on 5.4.4 (as `400` with the operator name): `contains`, `gte` and other range forms — the set for your cluster is listed at `https://<vms>/docs/`. For range and aggregate questions, use the SDK path, where the full predicate language is available ([Section 4, *Recipe queries*](/develop/docs/querying-the-vast-catalog#recipe-queries-vast-db-sdk1)). There is no match-all operator: at least one real filter is always required, and an empty-prefix `{"startswith": ""}` is rejected too (`400`). When you just want "everything under a path", filter on something broad like `element_type`. Three response details worth knowing: - If you omit `fields`, results come back with a **default column set** (`parent_path`, `name`, `size`, `mtime`, owner/group columns) — pass `fields` explicitly (POST honors it). - `count` **can be capped** on very large result sets (a broad query on a 5.4.4 lab cluster reports exactly 2097152 = 2²¹). Treat it as "at least this many", not an exact total. - Timestamps come back as ISO strings (`2026-08-06T03:12:10.196574`), in UTC. #### [](/develop/docs/querying-the-vast-catalog#sibling-endpoints)Sibling endpoints Same resource, same auth: | Endpoint | Purpose | Status | | --- | --- | --- | | `GET bigcatalogconfig/columns/` | The catalog schema — every column with its type — without SDK access. | ✅ works | | `GET bigcatalogconfig/stats/` | Catalog statistics: `count_columns`, `count_rows` (total elements — ≈1.0 B on our lab). | ✅ works | | `GET bigcatalogindexedcolumns/` (+ `add`/`remove`) | Which columns are indexed for filtering. | ✅ works (returns `[]` when none configured). | --- ### [](/develop/docs/querying-the-vast-catalog#path-b-the-web-ui)Path B: the Web UI In versions that expose the catalog in VMS, open **Database → VAST Catalog** in the left menu, then select **Open Query Panel**. You build conditions per column (the operators offered vary by data type), OR alternative conditions on the same column, AND conditions across columns, and view results in a grid whose displayed columns you can adjust. The official walkthrough is [Querying VAST Catalog from the VAST Web UI](/docs-5-5/docs/querying-vast-catalog-from-the-vast-web-ui). The column names match [Section 4](/develop/docs/querying-the-vast-catalog#4-schema-query-recipes1), and conditions map onto the `filters` above. Availability and layout vary by release; this guide's programmatic paths are identical everywhere, so they are what the examples use. If the panel exists on your version, it is the fastest way to confirm the catalog has data. --- ### [](/develop/docs/querying-the-vast-catalog#what-this-path-shows-you-and-what-it-doesnt)What this path shows you (and what it doesn't) The VMS API reads as the management plane: results span **every tenant** on the cluster. That makes it the right tool for cluster-wide administrative questions — and the wrong tool for handing directly to a tenant's users. For per-tenant access, use the SDK path, which is automatically scoped to the querying key's tenant ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)); to give users less than a tenant (a path subtree, say), put a small mediated service in front of either path and inject the caller's allowed prefix into `path`/`filters` — the same pattern as the AuditDB guide's [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1). --- ### [](/develop/docs/querying-the-vast-catalog#when-to-use-the-sdk)When to use the SDK Move to [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1) when you need range predicates, large extracts, streaming reads, or queries embedded in tooling. That path reads the table via the S3/tabular data endpoint and requires only an S3 key and a single identity policy. ## 3. Troubleshooting Three problems account for almost every support case: the SDK reports a missing bucket, a query runs but returns nothing (or less than you expected), or a query never seems to finish. All have straightforward fixes. > [!NOTE] > **Fastest way to get unblocked:** the VMS API path ([Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)) reads the same catalog with just VMS credentials — and sees every tenant. If you're stuck on the SDK, use it to keep working while you fix it. --- ### [](/develop/docs/querying-the-vast-catalog#error-1-missingbucket-on-the-sdk-path)Error 1: `MissingBucket` on the SDK path ```plaintext vastdb.errors.MissingBucket: {'bucket': 'vast-big-catalog-bucket'} ``` This means the code opened the bucket with `tx.bucket("vast-big-catalog-bucket")`. That call starts with an S3 `HEAD Bucket` check, and the catalog is a **managed** bucket that doesn't answer S3 bucket calls (it isn't backed by a view, so `HEAD` returns 404, and it's absent from `ListBuckets`) — even though tabular queries against it work fine. The fix is to open it the way the SDK intends, by well-known name: ```plaintext with session.transaction() as tx:    table = tx.catalog()          # instead of tx.bucket("vast-big-catalog-bucket")... ``` `tx.catalog()` skips the S3-level check and goes straight to the tabular API ([Section 5, *The catalog is a managed bucket*](/develop/docs/querying-the-vast-catalog#the-catalog-is-a-managed-bucket1)). If the query then fails with an access error, work down this list: | # | Check | How | | --- | --- | --- | | 1 | **Identity policy with** `s3:TabularList*` **+** `s3:TabularQueryData`**, attached to the user?** | Creating a policy is separate from attaching it. See [Section 5, *Grant access: the identity policy*](/develop/docs/querying-the-vast-catalog#grant-access-the-identity-policy1). | | 2 | **Catalog enabled and healthy?** | `vastpy-cli get bigcatalogconfig` should show `enable: true`, `state: "working"` ([Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)). | | 3 | **Right endpoint?** | Connect to the VAST DB **data VIP** (`http://<data-vip>.<cluster>...`), not the VMS management address — and use a VIP pool that belongs to the key's tenant (see 2b). | One thing that looks related but isn't: **the catalog never shows up in** `aws s3 ls` **or** `tx.bucket()`**.** That's by design, not a permissions problem — don't go looking for a view or View Policy to fix it. --- ### [](/develop/docs/querying-the-vast-catalog#error-2-empty-or-unexpected-results)Error 2: empty or unexpected results The query runs and returns no errors but returns no rows, or fewer than you expect. Causes, most common first: #### [](/develop/docs/querying-the-vast-catalog#2a-directory-listing-without-the-trailing-slash)2a. Directory listing without the trailing slash Directory listing is an **equality** match on `parent_path`, and the stored values end in `/`. The slash is not optional: ```plaintext (_.parent_path == "/projects")    # 0 rows — silently (_.parent_path == "/projects/")   # the directory's entries ``` Normalize paths before querying, as `examples/first_query.py` does. #### [](/develop/docs/querying-the-vast-catalog#2b-the-path-belongs-to-another-tenant)2b. The path belongs to another tenant The SDK is **automatically scoped to the querying key's tenant**. A path that plainly exists — you can see it in the VMS API or the UI — returns zero rows through the SDK when it belongs to a different tenant. That's the feature working as designed ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)), and it's the first thing to check when VMS and SDK results disagree. (Live example: on our lab cluster `/dataengine/` exists in six tenants but not the default tenant, so a default-tenant key gets an empty listing for it while the VMS API shows it.) Two related symptoms: - `403 Forbidden` **at transaction begin**: the key is valid but used against another tenant's VIP pool. Keys work only through their own tenant's VIPs. - **You genuinely need the cross-tenant view**: that's the VMS API path ([Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)), not an SDK setting. #### [](/develop/docs/querying-the-vast-catalog#2c-the-element-is-newer-than-the-last-refresh)2c. The element is newer than the last refresh The catalog is refreshed on the `frames.every` cadence (15–30 minutes typically, [Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)) — it is an index, not a live view. A file created two minutes ago is not a bug when it's missing; it appears after the next frame. (Verified live: an object PUT via S3 was queryable over S3 immediately and absent from the catalog in the same minute, on a cluster with a 30 m cadence.) If freshness matters to a report, read the cadence from `bigcatalogconfig` and say "as of" accordingly. #### [](/develop/docs/querying-the-vast-catalog#2d-wrong-element_type-spelling)2d. Wrong `element_type` spelling `element_type` values are UPPERCASE and the set is version-specific. On VAST 5.4.4 and 5.5.0.1: `FILE`, `DIR`, `SYMLINK`, `TABLE`, `SCHEMA`, `BLOCK_DIR` — so a filter like `element_type = 'File'` or `'file'` matches nothing. List what your cluster has (a handful of cheap probes, never a full scan): ```plaintext $ python3 examples/first_query.py --types Using the number of endpoints as a heuristic for concurrency. Using the number of endpoints as a heuristic for concurrency. Using the number of endpoints as a heuristic for concurrency. Using the number of endpoints as a heuristic for concurrency. Using the number of endpoints as a heuristic for concurrency. Using the number of endpoints as a heuristic for concurrency. element_type values present on this cluster:  FILE  DIR  SYMLINK  TABLE  SCHEMA  BLOCK_DIR (candidate list is version-specific; yours may have more — see ../04-schema-and-query-recipes.md) ``` #### [](/develop/docs/querying-the-vast-catalog#2e-filtering-on-the-map-columns)2e. Filtering on the map columns `user_tags` and `user_metadata` are Arrow **maps**, and map-key predicates don't push down — `(_.user_tags["env"] == "qa")` raises `NotImplementedError` on vastdb 2.x. The pattern that works: push down the counter column, filter the map client-side: ```plaintext from ibis import _ df = table.select(columns=["parent_path", "name", "user_tags"],                  predicate=(_.user_tags_count > 0),        # pushes down                  limit_rows=100_000).read_all().to_pandas() df["tags"] = df["user_tags"].map(dict) hits = df[df["tags"].map(lambda t: t.get("env") == "qa")]   # client-side ``` ([Section 4, Recipe 5](/develop/docs/querying-the-vast-catalog#recipe-5-everything-with-s3-user-tags1) shows the full version.) On the VMS path, the analogous surprise is operators: only `eq` and `startswith` are confirmed on 5.4.4; anything else returns `400` naming the operator ([Section 2, *Filter syntax*](/develop/docs/querying-the-vast-catalog#filter-syntax1)). --- ### [](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes)Error 3: the query never finishes The catalog holds **one row per element on the cluster** — billions on a big system (our 5.4.4 lab: ~1.0 billion elements, 3.3 billion physical rows). Every `select()` is a server-side scan with predicate pushdown, and two things decide how long it runs (timings below measured live on that lab): - **Queries return early once** `limit_rows` **is satisfied.** Matches stream back as the scan finds them: a 1000-entry directory listing with `limit_rows=1000` returned in ~3 s; the same 5-entry directory took ~1.7 s with `limit_rows=5`. - **When matches are fewer than** `limit_rows`**, the scan must sweep the whole catalog** to be sure it's done — ~15 s on our 3.3 B-row lab (~9 s with `num_splits=64`), regardless of how selective the predicate is. Listing a 5-entry directory with `limit_rows=1000` costs the full sweep. That's the per-query floor, and it grows with catalog size. - **Big result sets add transfer time on top.** Streaming a 200 k-row subtree took ~15 s; an unbounded `read_all()` over millions of rows runs for minutes and looks exactly like a hang. The habits that follow: ```plaintext # 1. Always pass limit_rows — it caps the fetch AND lets the scan stop early reader = table.select(columns=cols, predicate=pred, limit_rows=10_000) # 2. Keep limit_rows close to what you'll actually consume (page in modest #    chunks), and anchor subtree predicates as deep as the question allows — #    both raise the odds the scan can stop before sweeping everything ``` For genuinely large scans — space rollups, subtree inventories — iterate the reader in batches instead of `read_all()`, and raise `QueryConfig.num_splits` so the sweep fans out ([Section 5, *Reading at scale*](/develop/docs/querying-the-vast-catalog#reading-at-scale1)). A related hang happens before any query runs, when the endpoint is wrong. The failure mode depends on what's at the address: a host that answers but isn't a VAST DB endpoint (the VMS address, say) fails immediately (`UnsupportedServer`), but an **unreachable** host hits `vastdb.connect()`'s default retry loop — still retrying after 2½ minutes in our test. For interactive tools, fail fast instead (verified: 5 s): ```plaintext from vastdb.config import BackoffConfig session = vastdb.connect(endpoint=EP, access=AK, secret=SK,                         timeout=5, backoff_config=BackoffConfig(max_tries=1)) ``` All three example scripts in `examples/` do exactly this (their `--timeout` flag). --- ### [](/develop/docs/querying-the-vast-catalog#quick-triage-flow)Quick triage flow ```plaintext MissingBucket from the SDK?              → Error 1: open by name — tx.catalog(), not tx.bucket() Directory listing returns 0 rows?        → 2a: trailing slash on parent_path VMS shows the path, SDK returns nothing? → 2b: it's another tenant's namespace (by design) 403 at transaction begin?                → 2b: key used against another tenant's VIP pool A just-created file is missing?          → 2c: wait for the next refresh frame (Section 0 cadence) element_type filter matches nothing?     → 2d: UPPERCASE, version-specific — run --types Predicate raises NotImplementedError?    → 2e: map columns — push down user_tags_count instead Query hangs / runs forever?              → Error 3: unbounded fetch — add limit_rows, batch it Every query takes ~15s even for 5 rows?  → Error 3: normal — low-match queries sweep the catalog Connect itself hangs?                    → Error 3: unreachable endpoint — use the DB data VIP; cap timeout Nothing via the VMS path either?         → catalog isn't enabled/working (Section 0) ``` ## 4. Schema & Query Recipes Recipes are shown with the VAST DB SDK (ibis/pandas). Equality and prefix filters map directly onto the VMS REST API (`fields` + `filters`, [Section 2, *Filter syntax*](/develop/docs/querying-the-vast-catalog#filter-syntax1)); range and aggregate queries are SDK-only there. --- ### [](/develop/docs/querying-the-vast-catalog#the-schema)The schema One row per element in the namespace. The full column list on VAST 5.4.4 (32 columns — also retrievable without the SDK via `GET bigcatalogconfig/columns/`; the official schema reference is in the [VAST Catalog Overview](/docs-5-5/docs/vast-catalog-overview)): #### [](/develop/docs/querying-the-vast-catalog#identity-what-and-where)Identity: what and where | Column | Type | Meaning | | --- | --- | --- | | `parent_path` | `string` | Directory containing the element, **with trailing slash** (`/projects/models/`). Equality on it = directory listing; `startswith` = subtree search. Either way the read is a capped scan — see [Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1) for the cost model. | | `name` | `string` | The element's name. Full path = `parent_path + name`. | | `element_type` | `string` | What the element is. UPPERCASE; on 5.4.4 and 5.5.0.1: `FILE`, `DIR`, `SYMLINK`, `TABLE`, `SCHEMA`, `BLOCK_DIR`. Version-specific — probe with `first_query.py --types`. | | `extension` | `string` | Filename extension without the dot (`log`, `parquet`) — indexed for the classic "find by type" query. | | `symlink_path` | `string` | Symlink target, for `SYMLINK` rows. | | `phandle` | `struct<clone_id, handle_id>` | The element's internal handle. | | `search_path` | `string` | Per the official docs, a virtual column that restricts a query to a subtree; observed on 5.4.4 to mirror `parent_path` in every row. | | `tenant_id` | `int32` | **Internal** tenant id — does **not** match VMS tenant IDs, and your own tenant's rows can show `-1`. Don't filter on it: tenancy is already enforced server-side ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)). | #### [](/develop/docs/querying-the-vast-catalog#size-and-links)Size and links | Column | Type | Meaning | | --- | --- | --- | | `size` | `int64` | Logical size in bytes (directories show the dir-entry size, 4096). | | `used` | `int64` | Bytes actually consumed (0 for directories). | | `nlinks` | `int64` | Hard-link count. | #### [](/develop/docs/querying-the-vast-catalog#ownership-and-permissions)Ownership and permissions | Column | Type | Meaning | | --- | --- | --- | | `uid` / `gid` | `int32` | Numeric owner/group. | | `owner_name` / `group_owner_name` | `string` | Resolved names, when the identity resolves (S3-created data often shows the VAST user's display name). **When unresolved, holds a numeric string** (`"0"`) — and on 5.5 can also be null — filter accordingly. | | `owner_sid` / `group_owner_sid` | `string` | SMB SIDs, when applicable. | | `login_name` | `string` | Qualified login of the owner identity when resolvable (`vastdata@selab.vastdata.com`); `"0"` otherwise. | | `nfs_mode_bits` | `int32` | POSIX mode bits. | | `name_aces_exist` | `bool` | Whether named ACEs exist on the element. | | `abac_tags` | `list<string>` | ABAC security tags. | #### [](/develop/docs/querying-the-vast-catalog#times-all-timestampns-utc)Times (all `timestamp[ns]`, UTC) | Column | Meaning | | --- | --- | | `creation_time` | When the element was created. | | `mtime` | Last data modification — the workhorse for "changed since" ([Recipe 3](/develop/docs/querying-the-vast-catalog#recipe-3-everything-that-changed-since-a-point-in-time)). | | `ctime` | Last metadata change. | | `atime` | Last access. | #### [](/develop/docs/querying-the-vast-catalog#s3-object-metadata)S3 object metadata | Column | Type | Meaning | | --- | --- | --- | | `user_tags` | `map<string,string>` | S3 object tags. Map columns don't take pushdown predicates — see [Recipe 5](/develop/docs/querying-the-vast-catalog#recipe-5-everything-with-s3-user-tags). | | `user_metadata` | `map<string,string>` | S3 user metadata (`x-amz-meta-*`). Same pushdown caveat. | | `user_tags_count` | `int16` | Number of tags — **this** is the pushdown filter for "anything tagged". | | `s3_locks_retention` | `struct<mode, timeout>` | S3 object-lock retention. | | `s3_locks_legal_hold` | `bool` | S3 legal hold. | #### [](/develop/docs/querying-the-vast-catalog#devices)Devices | Column | Type | Meaning | | --- | --- | --- | | `major_device` / `minor_device` | `int32` | Device numbers, for device nodes. | --- ### [](/develop/docs/querying-the-vast-catalog#recipe-queries-vast-db-sdk)Recipe queries (VAST DB SDK) Each assumes an open `table` handle: ```plaintext import vastdb from ibis import _ with vastdb.connect(endpoint=, access=, secret=).transaction() as tx:    table = tx.catalog()   # the VAST Catalog, by its built-in well-known name    ...                    # recipes below run inside this block ``` Two habits to keep: always pass `limit_rows` (it caps the fetch *and* lets the scan stop early), and keep it close to what you'll actually consume — a query whose matches are fewer than its `limit_rows` sweeps the whole catalog before returning ([Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1)). For the most common reports, `examples/catalog_report.py` is ready to run: ```plaintext python catalog_report.py --largest --under /projects/ python catalog_report.py --owner root --changed-since '2026-08-01' python catalog_report.py --ext parquet --min-size 1G python catalog_report.py --tagged ``` #### [](/develop/docs/querying-the-vast-catalog#recipe-1-list-a-directory)Recipe 1: List a directory Equality on `parent_path`, trailing slash mandatory ([Section 3, 2a](/develop/docs/querying-the-vast-catalog#2a-directory-listing-without-the-trailing-slash1)): ```plaintext df = table.select(    columns=["name", "element_type", "size", "used", "mtime", "owner_name"],    predicate=(_.parent_path == "/projects/models/"),    limit_rows=100_000, ).read_all().to_pandas() ``` Filter to subdirectories only with `& (_.element_type == "DIR")` — that's how a tree browser walks the namespace one query at a time. Expect seconds, not milliseconds: a listing that doesn't fill its `limit_rows` sweeps the catalog before it can declare itself done (~15 s on a 3 B-row lab; [Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1)), which is why browsing tools cache listings. #### [](/develop/docs/querying-the-vast-catalog#recipe-2-largest-files-under-a-subtree)Recipe 2: Largest files under a subtree `startswith` scans the subtree, so give the scan a selective size floor and let the sort happen client-side: ```plaintext pred = (_.element_type == "FILE") \       & (_.parent_path.startswith("/projects/")) \       & (_.size >= 1 * 2**30)                      # ≥ 1 GiB df = table.select(columns=["parent_path", "name", "size", "owner_name"],                  predicate=pred, limit_rows=100_000).read_all().to_pandas() top = df.sort_values("size", ascending=False).head(100) ``` Caveat: if the scan hits `limit_rows`, you sorted a *sample*, not the true top-N — raise the size floor until the match set fits under the cap. #### [](/develop/docs/querying-the-vast-catalog#recipe-3-everything-that-changed-since-a-point-in-time)Recipe 3: Everything that changed since a point in time ```plaintext import ibis from datetime import datetime pred = (_.mtime >= ibis.literal(datetime(2026, 8, 1), type="timestamp")) \       & (_.parent_path.startswith("/projects/")) df = table.select(columns=["parent_path", "name", "element_type", "mtime", "owner_name"],                  predicate=pred, limit_rows=100_000).read_all().to_pandas() ``` Times are UTC. `creation_time`, `ctime`, and `atime` take the same shape — "files nobody has read in a year" is this recipe with `_.atime <=`. #### [](/develop/docs/querying-the-vast-catalog#recipe-4-one-owners-data-or-one-extension)Recipe 4: One owner's data (or one extension) ```plaintext pred = (_.element_type == "FILE") & (_.owner_name == "jsmith")       # or _.uid == 1000 pred = pred & (_.extension == "parquet")                              # optional df = table.select(columns=["parent_path", "name", "size", "mtime"],                  predicate=pred, limit_rows=100_000).read_all().to_pandas() print(df["size"].sum(), "bytes in", len(df), "files") ``` #### [](/develop/docs/querying-the-vast-catalog#recipe-5-everything-with-s3-user-tags)Recipe 5: Everything with S3 user tags Map columns don't push down; `user_tags_count` does ([Section 3, 2e](/develop/docs/querying-the-vast-catalog#2e-filtering-on-the-map-columns1)): ```plaintext df = table.select(columns=["parent_path", "name", "user_tags", "user_metadata"],                  predicate=(_.user_tags_count > 0),                  limit_rows=100_000).read_all().to_pandas() df["tags"] = df["user_tags"].map(dict) qa = df[df["tags"].map(lambda t: t.get("env") == "qa")]   # client-side key/value filter ``` #### [](/develop/docs/querying-the-vast-catalog#recipe-6-count-elements-by-type-a-full-scan-report-done-deliberately)Recipe 6: Count elements by type (a full-scan report, done deliberately) Exact cluster-wide counts read one column across the entire catalog — minutes on a big cluster, so stream it instead of `read_all()` and know you asked for it: ```plaintext from collections import Counter counts = Counter() reader = table.select(columns=["element_type"])       # no limit: deliberate full scan for batch in reader:    counts.update(batch.column("element_type").to_pylist()) print(dict(counts)) ``` Scope it to a subtree with a `parent_path.startswith` predicate to make the transfer proportionally cheaper. (For "which types exist at all", skip the scan — that's `first_query.py --types`, six `limit_rows=1` probes. For a plain **total** element count, skip it too: `GET bigcatalogconfig/stats` returns `count_rows` (≈1.0 B on our lab), and the SDK's `table.get_stats()` returns the physical row count — larger, ~3.3 B on the same lab, because the table retains multiple catalog frames.) ## 5. Advanced: SDK Access & Tenancy The VAST DB SDK reads the catalog the same way it reads any VAST Database table: over S3, with an access key. Use it for anything programmatic — reports, pipelines, tools. This section walks through the setup, then the two things that make the catalog different at scale: automatic tenant scoping, and reading efficiently from a table with a row for every element on the cluster. --- ### [](/develop/docs/querying-the-vast-catalog#connect-and-query)Connect and query Three steps: install, point at the data endpoint, open the catalog by name. ```plaintext pip install vastdb pyarrow pandas ibis-framework export VASTDB_ENDPOINT='http://'   # S3/tabular data VIP, not the VMS address export VASTDB_ACCESS_KEY='' export VASTDB_SECRET_KEY='' ``` ```plaintext import os, vastdb from ibis import _ session = vastdb.connect(endpoint=os.environ["VASTDB_ENDPOINT"],                         access=os.environ["VASTDB_ACCESS_KEY"],                         secret=os.environ["VASTDB_SECRET_KEY"]) with session.transaction() as tx:    table = tx.catalog()                  # the VAST Catalog, by its built-in well-known name    batch = table.select(        columns=["name", "element_type", "size", "mtime", "owner_name"],        predicate=(_.parent_path == "/projects/"),        limit_rows=100,    ).read_all()    print(batch.to_pandas()) ``` That's the whole pattern. `tx.catalog()` returns a regular table handle, so everything in [Section 4](/develop/docs/querying-the-vast-catalog#4-schema-query-recipes1) applies as-is. The starter `examples/first_query.py` wraps this with listing, `--types`, and `--schema`; `examples/catalog_report.py` builds it out into the common reports. One habit for command-line tools: pass `timeout=` and `backoff_config=BackoffConfig(max_tries=1)` (from `vastdb.config`) to `vastdb.connect()` so an unreachable endpoint fails in seconds (measured: 5 s, vs. still-retrying at 2½ minutes with the defaults; [Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1)). #### [](/develop/docs/querying-the-vast-catalog#the-catalog-is-a-managed-bucket)The catalog is a managed bucket The catalog lives at fixed coordinates on every cluster — `vast-big-catalog-bucket` / `vast_big_catalog_schema` / `vast_big_catalog_table` — and the SDK knows them (`tx.catalog()`, just like `tx.audit_log()` for the AuditDB). Because the bucket is managed by the cluster rather than backed by a view you created, it behaves differently from your own DB buckets in one visible way: **it doesn't answer S3 bucket calls.** It won't appear in `ListBuckets` (`aws s3 ls`), and `tx.bucket("vast-big-catalog-bucket")` fails with `MissingBucket` because that call begins with an S3 `HEAD Bucket` check. This is normal — open it by name and query it ([Section 3, Error 1](/develop/docs/querying-the-vast-catalog#error-1-missingbucket-on-the-sdk-path1)). On an SDK too old to have `tx.catalog()`, or in a tool that keeps the coordinates configurable, construct the handle by name explicitly — it's the same open-by-name operation: ```plaintext import vastdb.bucket, vastdb.schema b = vastdb.bucket.Bucket("vast-big-catalog-bucket", tx) table = vastdb.schema.Schema("vast_big_catalog_schema", b).table("vast_big_catalog_table") ``` --- ### [](/develop/docs/querying-the-vast-catalog#grant-access-the-identity-policy)Grant access: the identity policy The querying user needs an S3 access key and one read-only identity policy with the tabular actions. That's the entire permission model for the SDK path — the catalog bucket has no View Policy to join, because it isn't backed by a view, and there is no per-tenant catalog setup. ```plaintext {  "Version": "2012-10-17",  "Statement": [    { "Sid": "ListTabular", "Effect": "Allow",      "Action": "s3:TabularList*",      "Resource": ["vast-big-catalog-bucket", "vast-big-catalog-bucket/vast_big_catalog_schema/*"] },    { "Sid": "QueryTabular", "Effect": "Allow",      "Action": "s3:TabularQueryData",      "Resource": ["vast-big-catalog-bucket/vast_big_catalog_schema/*"] }  ] } ``` - `s3:TabularList*` lets the SDK resolve the schema and table; `s3:TabularQueryData` performs the read. Both are needed for a working read-only client. - **Resource forms** (from VAST's *Managing Permissions for Accessing VAST Tabular Databases*): `"<bucket>"` for bucket-level ops, `"<bucket>/<schema>/*"` for schema-level, `"<bucket>/<schema>/<table>"` for table-level. The `arn:aws:s3:::<bucket>/...` form is also accepted on some versions. A broad `Resource: "*"` also matches the managed catalog bucket — verified live on VAST 5.5.0.1 with a from-scratch user whose only grant was a tabular-actions policy on `"*"`. Setup order, all with vastpy (see the cheat-sheet in [Section 6](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1)): create the user, create their access key, create the identity policy, attach it to the user. --- ### [](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic)Tenant scoping is automatic The catalog is **scoped, server-side, to the tenant of the S3 identity you query with**. A tenant's key sees exactly its own namespace — its paths, its rows, nothing else. There is no tenant parameter to pass and no way to opt out from the SDK side. Verified live (VAST 5.4.4): a default-tenant key listing `/` sees only default-tenant paths; a second tenant's subtrees simply aren't there; a key used against another tenant's VIP pool gets `403 Forbidden` at transaction begin (the tenant is bound to the VIP pool, so use the tenant's own VIPs). Practical consequences: - **Giving a team access to *their* namespace is zero extra work**: create a user in their tenant, attach the identity policy above, hand out a key for their tenant's VIP pool. The server does the scoping. - **Cross-tenant reporting is not an SDK capability** — that's the VMS API path ([Section 2](/develop/docs/querying-the-vast-catalog#2-query-the-catalog1)), which reads as the management plane and sees everything. - **Don't try to reimplement tenancy with the** `tenant_id` **column.** It holds internal IDs that don't match VMS tenant IDs (and your own rows can show `-1`). The scoping already happened before your predicate ran. - **If you also use the AuditDB, note the inversion**: audit rows are cluster-wide to any key that can read the table (you filter on its `tenant` name column yourself), while the catalog is auto-scoped. Same access pattern, opposite tenancy defaults. #### [](/develop/docs/querying-the-vast-catalog#scoping-below-a-tenant)Scoping below a tenant Within one tenant, the `s3:TabularQueryData` grant is table-level — all-or-nothing. To give a user only a path subtree, put a small **mediated query service** in front: it holds the credential, maps each authenticated caller to their allowed prefixes, and injects the prefix into the predicate (`_.parent_path.startswith(allowed)`) or the VMS call's `path`/`filters`. The pattern (and its design notes) is the AuditDB guide's [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1) — it applies here unchanged. #### [](/develop/docs/querying-the-vast-catalog#out-of-scope-trino-and-row-and-column-security)Out of scope: Trino and Row and Column Security External Trino query engines can also query the catalog, and VAST Database Row and Column Security can make the database itself filter rows and columns per user — but RCS is enforced only by Trino engines with end-user impersonation, so both are out of scope for this guide. If you later need a hard, database-enforced boundary below the tenant level, start with VAST's [*VAST Database Row and Column Security*](/docs-5-5/docs/vast-database-row-and-column-security) doc. --- ### [](/develop/docs/querying-the-vast-catalog#reading-at-scale)Reading at scale Two SDK features matter once queries leave "one directory" territory: **Stream, don't buffer.** `select()` returns a RecordBatch reader. `read_all()` buffers everything; iterating processes batches as they arrive, in bounded memory: ```plaintext reader = table.select(columns=["parent_path", "size", "used"],                      predicate=(_.element_type == "FILE")                                & (_.parent_path.startswith("/projects/"))) total = 0 for batch in reader:                      # arrives while the scan still runs    total += sum(batch.column("used").to_pylist()) ``` **Fan the scan out.** `QueryConfig` controls parallelism — `num_splits` (how the table scan is partitioned) and `num_sub_splits` per split. Measured on a 3.3 B-row lab catalog: `num_splits=64` cut a full-sweep query from ~15 s to ~9 s and a 200 k-row subtree stream from ~15 s to ~10 s; production tools also spread requests across all the data VIPs (`QueryConfig(data_endpoints=[...])`): ```plaintext from vastdb.config import QueryConfig config = QueryConfig(num_splits=64, num_sub_splits=4) reader = table.select(columns=cols, predicate=pred, config=config) ``` **Know the size without scanning.** `table.get_stats()` returns the table's physical row count and byte size in one cheap call (`num_rows=3_300_583_568` on our lab — note this is physical rows including retained catalog frames, larger than the element count that `GET bigcatalogconfig/stats` reports). #### [](/develop/docs/querying-the-vast-catalog#what-you-can-build-on-this-catwalk)What you can build on this: Catwalk A worked example of everything in this section is **Catwalk**, an interactive web file browser for the VAST Catalog: pick a view, browse the tree, page through directories with millions of entries, and get per-directory space rollups aggregated over all descendants — all from catalog queries, never walking the filesystem. Internally it is exactly this guide's patterns: `tx.catalog()`, equality-on-`parent_path` listings (cached, because even a small listing costs a catalog sweep — [Section 3, Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1)), streamed `startswith` scans with `num_splits` fan-out for rollups, and per-tenant deployment by credential (run it with a tenant's key and it browses that tenant's namespace — no tenant option needed, [see above](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)). It also ships a mock mode (`CATWALK_MOCK=1`) that serves a synthetic namespace, handy for demos without a cluster. ## 6. Appendix: CLI, Config Links & Verification ### [](/develop/docs/querying-the-vast-catalog#environment-variables-sdk-companion-scripts)Environment variables (SDK / companion scripts) The `examples/` scripts read these: ```plaintext # SDK path (first_query.py, catalog_report.py) export VASTDB_ENDPOINT='http://'   # VAST DB data endpoint (not VMS) export VASTDB_ACCESS_KEY='' export VASTDB_SECRET_KEY='' # VMS path (query_vms_catalog.py) export VMS_ADDRESS=''                 # bare hostname; vastpy adds https:// export VMS_USER='admin' export VMS_PASSWORD='********'                  # or VMS_TOKEN (VAST 5.3+) ``` The catalog bucket/schema/table names are fixed (`vast-big-catalog-bucket` / `vast_big_catalog_schema` / `vast_big_catalog_table`) and built into the SDK, so they never need configuring. --- ### [](/develop/docs/querying-the-vast-catalog#vastpy-cheat-sheet)vastpy cheat-sheet [vastpy](https://pypi.org/project/vastpy/) is a thin VMS REST client (SDK and `vastpy-cli`). It's a schema-less passthrough: `vastpy-cli <get|post|patch|delete> <endpoint> key=value ...`. Values that look like JSON are parsed as JSON, and `-i file.json` supplies a POST/PATCH body from a file. ```plaintext # --- Auth (env or per-call flags) --- export VMS_ADDRESS=vms.example.com export VMS_USER=admin export VMS_PASSWORD='********' # --- Catalog config (admin; full enable procedure in Section 0) --- vastpy-cli get bigcatalogconfig                       # enabled? state? refresh cadence? vastpy-cli patch bigcatalogconfig/1 enable=true \  frames='[{"every":"15m","every_value":"15m","every_unit":"m",            "start-at":"",            "keep-local":"2H","local-period":"H",            "keep-remote":"15m","remote-period":"m"}]' # --- Query the catalog via the VMS API --- # POST; both `path` (subtree anchor, recursive) and `filters` are required. vastpy-cli post bigcatalogconfig/query_data path=/ limit=20 \  filters='{"element_type":[{"eq":"TABLE"}]}' \  fields='["parent_path","name","element_type","size","mtime","owner_name"]' # --- Schema without SDK access --- vastpy-cli get bigcatalogconfig/columns # --- Sibling endpoints (see Section 2) --- vastpy-cli get bigcatalogconfig/stats vastpy-cli get bigcatalogindexedcolumns # --- Users, keys & identity policies (SDK-path setup; see Section 5) --- vastpy-cli post users name='cataloguser' user_type='LOCAL' vastpy-cli post users//access_keys           # or .../s3_access_keys on some versions vastpy-cli post s3policies name='catalog-readonly' policy="$(cat policy.json)" tenant_id=1 vastpy-cli patch s3policies/ users='["cataloguser"]'   # attach (version-dependent) ``` A note on TLS: vastpy doesn't verify certificates unless you pass `--cert-file` / `cert_file`, and the same goes for `vastdb.connect(..., ssl_verify=False)`. Fine for labs; set a CA bundle in production. --- ### [](/develop/docs/querying-the-vast-catalog#vast-db-tabular-permissions)VAST DB tabular permissions Tabular operations are granted by an **identity policy** with `s3:Tabular*` actions, scoped by Resource: `"<bucket>"` (bucket-level, e.g. list schemas), `"<bucket>/<schema>/*"` (schema-level), `"<bucket>/<schema>/<table>"` (table-level). For the **catalog** (a managed bucket, like the AuditDB), the identity policy is the whole model — there is no view or View Policy involved, and tenancy is enforced server-side at query time ([Section 5, *Tenant scoping is automatic*](/develop/docs/querying-the-vast-catalog#tenant-scoping-is-automatic1)). For DB buckets you create yourself (S3-enabled `DATABASE` views), the user must additionally be in the view's View Policy with bucket-listing permission for the bucket to be visible. Common tabular actions (identity-policy `Action` values, all `s3:`-prefixed): | Action | Purpose | | --- | --- | | `TabularQueryData` | Read / SELECT rows | | `TabularListSchemas` / `TabularListTables` / `TabularListColumns` | Resolve schema → table → columns | | `TabularGetTableStats` | Row count / used bytes | --- ### [](/develop/docs/querying-the-vast-catalog#official-vast-references)Official VAST references This guide gets you started; these are the reference docs: - [VAST Catalog Overview](/docs-5-5/docs/vast-catalog-overview) — concepts, the official schema reference, and the supported query tools. - [Configuring VAST Catalog](/docs-5-5/docs/configuring-vast-catalog) — the Web UI enablement procedure, snapshot schedules and retention, user-defined attribute indexing (S3 tags/metadata), and monitoring indexing progress. - [Querying VAST Catalog from the VAST Web UI](/docs-5-5/docs/querying-vast-catalog-from-the-vast-web-ui) — the Query Panel walkthrough (Database → VAST Catalog → Open Query Panel). - *Managing Permissions for Accessing VAST Tabular Databases* (support KB) — the identity-policy model and Resource forms used in [Section 5](/develop/docs/querying-the-vast-catalog#5-advanced-sdk-access-tenancy1). - *VAST Database Row and Column Security* (support KB) — only if you later need database-enforced scoping (Trino-only; out of scope here). - VAST's [data-platform-field-docs](https://vast-data.github.io/data-platform-field-docs/) — runnable catalog notebooks, including the Python SDK examples this guide's recipes align with. --- ### [](/develop/docs/querying-the-vast-catalog#verification-checklist)Verification checklist Facts this guide asserts, and where they were last confirmed. A full skeptical re-verification of every claim ran 2026-08-06 against var202 (VAST 5.4.4.0, vastdb 2.0.14); rows below are from that pass unless noted. The examples were additionally re-run the same day against var204 (VAST 5.5.0.1). Re-verify the version-specific rows on a new VAST release. #### [](/develop/docs/querying-the-vast-catalog#enablement-vms-api)Enablement & VMS API | Fact | Status | | --- | --- | | `bigcatalogconfig` GET shows `enable`/`state`/`frames`; PATCH with [Section 0](/develop/docs/querying-the-vast-catalog#0-enable-the-catalog1)'s exact field shape accepted (idempotent PATCH left config unchanged, `state: working`) | ✅ (the Section 0 `vastpy-cli patch` line itself ran live on 2026-08-07) | | `query_data` requires `path` + `filters` (400 naming the missing field); POST honors `fields`+`limit`; omitted `fields` → default 10-column set | ✅ | | `path` anchors recursively AND scopes: a file under `/sven/` found with `path=/sven/` and `path=/`, not with `path=/bgolliher/` | ✅ | | Operators: `eq`, `startswith` work; `contains`/`gte` → 400; **no match-all** (empty-prefix `startswith` → 400 too) | ✅ (version-specific) | | `count` caps at exactly 2 097 152 = 2²¹ (reproduced) — treat as lower bound | ✅ | | VMS timestamps are the same UTC instants the SDK returns (same file compared both paths) | ✅ | | `columns` works (schema + the well-known coordinates); `stats` works (`count_columns: 32`, `count_rows` ≈ 1.00 B elements); `bigcatalogindexedcolumns` works (`[]` when unset) | ✅ | | `aggregate` — internal server error for every parameter form tried; identical failure on 5.5.0.1 (and POST → 405) | ⚠️ in progress 5.4.4 and 5.5.0.1 | | vastpy at the S3 data endpoint → `This request does not support credentials.` | ✅ | | All [Section 6](/develop/docs/querying-the-vast-catalog#6-appendix-cli-config-links-verification1) `vastpy-cli` lines (get bigcatalogconfig, post query_data, get columns/stats) | ✅ | #### [](/develop/docs/querying-the-vast-catalog#sdk-path)SDK path | Fact | Status | | --- | --- | | `tx.catalog()` works with only the `s3:Tabular*` identity policy; explicit `Bucket("vast-big-catalog-bucket", tx)` construction works too | ✅ | | From-scratch minimal user: a new local user whose only grant is a tabular-actions policy with `Resource: "*"` can `tx.catalog()` and query — `Resource: "*"` matches the managed bucket; no view involved | ✅ var204 (5.5.0.1), 2026-08-06 | | `tx.bucket("vast-big-catalog-bucket")` → `MissingBucket: {'bucket': 'vast-big-catalog-bucket'}` via S3 `HEAD` 404; bucket absent from `ListBuckets`; `head_bucket` → 404 (boto3) | ✅ | | 32-column schema as in Section 4; `element_type` = FILE, DIR, SYMLINK, TABLE, SCHEMA, BLOCK_DIR | ✅ same set on 5.4.4 and 5.5.0.1 (version-specific) | | Trailing slash mandatory on `parent_path` equality (`'/sven'` → 0 rows, `'/sven/'` → 5) | ✅ | | `owner_name` falls back to numeric string (`"0"`) when unresolved — on 5.5.0.1 unresolved rows can also be null; `login_name` is qualified (`user@domain`) or `"0"`; `search_path` mirrors `parent_path`; `used`=0 for DIRs | ✅ | | Map-key predicates → `NotImplementedError`; `user_tags_count > 0` pushes down; `.map(dict)` + client-side key filter works (Recipe 5 code ran as written) | ✅ | | All Section 4 recipes ran as written (1, 2 incl. its hit-the-cap caveat, 3, 4, 5, 6-scoped) | ✅ | | All three `examples/` scripts pass on VAST 5.5.0.1 (var204): `first_query.py --types` / `--path /`, `catalog_report.py --largest`, `query_vms_catalog.py --type TABLE` | ✅ 2026-08-06 | | `table.get_stats()`: `num_rows=3 300 583 568` physical (> the ~1.0 B element count — retained frames), `size_in_bytes≈1.4 TB` | ✅ | #### [](/develop/docs/querying-the-vast-catalog#performance-measured-the-section-3-error-3-cost-model)Performance (measured — the [Section 3 Error 3](/develop/docs/querying-the-vast-catalog#error-3-the-query-never-finishes1) cost model) | Measurement | Result | | --- | --- | | Query returns early when `limit_rows` is met | 5-entry dir, `limit_rows=5`: 1.7 s · 1000-entry dir, `limit_rows=1000`: 3.3 s | | Low-match query sweeps the catalog | 5-entry dir, `limit_rows=1000`: 14.4–15.2 s (3 runs) | | `num_splits=64` speeds the sweep | 15 s → 9.0 s; 200 k-row subtree stream 14.7 s → 10.0 s | | Fail-fast connect (`timeout=5, max_tries=1`) vs default, unreachable host | 5.0 s vs still retrying at 150 s | | Wrong-but-responding endpoint (the VMS host) | fails instantly (`UnsupportedServer`) |