--- title: "How To Audit DB Query Guide" slug: "how-to-audit-db-query-guide" description: "Enable audit logging for NFS and S3 views in VAST Database with simple settings. Learn to track operations and enhance your data security efficiently." updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/how-to-audit-db-query-guide" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # How To Audit DB Query Guide ## Enable Audit Logging to the Database --- This page turns on protocol auditing for **all NFS and S3 views** on a VAST cluster and stores the records in the VAST Database (the AuditDB), from which the rest of this guide can query them. It takes two settings and about five minutes. The two settings: 1. **Save audit logs to VAST DB:** Routes audit records into a database table. 2. **Global Baseline Audit Settings:** Choose which protocols and operations get recorded. Whatever you enable here applies to **every view on the cluster**, regardless of view policy. That is why covering all NFS and S3 views takes no per-view or per-policy work. (View policies can *add* auditing selectively on top of the baseline; you only need that when you want some views audited more than others.) --- ### Enable it in the Web UI 1. Log in to VMS and go to **Settings → Auditing**. 2. Leave the **General** settings at their defaults. The audit directory name and the Read-access Users/Groups fields are part of the JSON file export, which is a separate path from the database. 3. Under **Define retention period**, set the length of time audit data is retained. The default is a manually defined period of one hour, which is short; pick a window that matches how far back you need to look. 4. Under **Auditing**, turn on **Save audit logs to VAST DB**. This is the switch that creates the AuditDB table. (**Saving audit logs to a file (JSON format)** is independent; you can enable both.) 5. Under **Global Baseline Audit Settings**: 1. Toggle **Enable** on. 2. Under **Select protocols to assign operations**, select the protocols your views serve. For all NFS and S3 views, select **NFSv3**, **NFSv4.1**, and **S3**. (The full list on 5.5 is NFSv3, NFSv4.1, SMB, S3, and NDB, where NDB is VAST Database traffic.) 3. Under **Operations to audit**, choose the categories to record: | Category | Records | | --- | --- | | Create/Delete Files/Directories/Objects | Creates and deletes | | Modify Data | Writes, including size changes | | Modify Metadata | Permission, ownership, attribute changes | | Read Data | Reads and downloads | | Read Metadata | Stats, lists, lookups | | Session create/close | Kerberos session open/close | A good starting set is Create/Delete, Modify Data, and Modify Metadata. Add the Read categories deliberately: on a busy cluster, reads dominate traffic, so they multiply the row volume. 4. Under the audit record options, leave **Log Full Path** on (the default; each record carries the full Element Store path, at a small performance cost) and turn on **Log Username** so records carry the resolved username, not just the UID. 6. Click **Save**. That's it. Rows begin to appear as audited operations occur. --- ### Or enable it with the VMS REST API The same settings live at the `clusters/<id>/auditing` endpoint, so this script cleanly with [vastpy](https://pypi.org/project/vastpy/): ```bash pip install vastpy export VMS_ADDRESS=        # management host, bare hostname export VMS_USER=admin export VMS_PASSWORD='********' vastpy-cli get clusters              # find your cluster id (usually 1) vastpy-cli get clusters/1/auditing   # current settings ``` Write the config you want as `audit-config.json`. Field names below are as returned by the GET on VAST 5.5.0.1; run the GET first and mirror the names your version returns (`modify_data_md` and `read_data_md` are the metadata categories): ```json {  "enable_vast_db_audit": true,  "protocols": ["NFSv3", "NFSv4.1", "S3"],  "protocols_audit": {    "create_delete_files_dirs_objects": true,    "modify_data": true,    "modify_data_md": true,    "read_data": true,    "read_data_md": true,    "log_full_path": true,    "log_username": true  } } ``` Apply it, then confirm: ```bash vastpy-cli patch clusters/1/auditing --file-input audit-config.json vastpy-cli get   clusters/1/auditing ``` The confirming GET should show `enable_vast_db_audit: True` and, once the table exists, `vast_audit_log_state: CREATED`. VAST CLI works too if you prefer it: `cluster show --audit` displays the current settings and `cluster modify --audit-protocols ... --audit-operations ... --enable-audit-settings`changes them; see VAST's *Configuring Global Auditing Settings* KB page. --- ### Confirm it's working Generate some traffic first (write a file over NFS, or put an object into an S3 bucket), then read it back either way: - **Web UI:** go to **DataBase → VAST Audit Log**. Your operations appear as rows. **Scripted:** `python examples/query_vms_api.py --bucket <bucket> --top 10` ([§2](/develop/docs/how-to-audit-db-query-guide#path-b-the-vms-rest-api) covers this path). From here, the audit table is fully queryable through the Web UI, the VMS REST API, and the VAST DB SDK. The SDK path just needs an S3 key and an identity policy, covered in [§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1). --- ## 1. Overview & Prerequisites ### [](/develop/docs/how-to-audit-db-query-guide#prerequisites)Prerequisites Everything in this guide builds on three admin settings. Confirm them once, and you're set (if any are off, [§0](/develop/docs/how-to-audit-db-query-guide#enable-audit-logging-to-the-database) walks through enabling them): 1. **Protocol auditing is enabled** on the cluster. 2. **Audit logs are saved to VAST DB.** In VMS: `Settings → Auditing → Save audit logs to VAST DB`. Both Save to VAST DB" and "Save to file (JSON)" can be on at once; this guide uses the VAST DB destination. (To read the JSON files over NFS, S3, or SMB instead, see VAST's *Accessing Audit Log JSON Files via NFS, S3, and SMB*.) 3. **The operations you care about are audited.** Auditing is opt-in per category (Create/Delete, Modify Data, Modify Metadata, Read Data, Read Metadata, Sessions). Rows appear only for enabled categories. With those in place, the audit table is readable through the Web UI and the VMS REST API immediately ([§2](/develop/docs/how-to-audit-db-query-guide#2-query-the-auditdb1)). The VAST DB SDK reads the same table over the S3/tabular data endpoint; it needs an S3 access key whose user has an `s3:Tabular*` identity policy, covered in [§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1). The step-by-step enable procedure is in [§0](/develop/docs/how-to-audit-db-query-guide#enable-audit-logging-to-the-database); the authoritative references are VAST's *Configuring Global Auditing Settings* and *Configuring Auditing with View Policies* KB pages ([§6](/develop/docs/how-to-audit-db-query-guide#official-vast-configuration-docs1)). ### [](/develop/docs/how-to-audit-db-query-guide#audit-table-coordinates-sdk-path-only)Audit table coordinates (SDK path only) The AuditDB has the same well-known coordinates on every cluster: | Piece | Value | Notes | | --- | --- | --- | | Bucket | `vast-audit-log-bucket` | A **managed** bucket: open it by name; it doesn't appear in S3 bucket listings ([§5](/develop/docs/how-to-audit-db-query-guide#the-auditdb-is-a-managed-bucket1)). | | Schema | `vast_audit_log_schema` | Fixed. | | Table | `vast_audit_log_table` | Fixed. | | DB data endpoint | `http://<cluster-vip>` | The VAST DB data VIP, not the VMS management address. | The SDK has these names built in — `tx.audit_log()` returns the table handle directly. The Web UI and VMS API paths do not need this; they access the audit log through the management plane. --- ### [](/develop/docs/how-to-audit-db-query-guide#choose-your-access-method)Choose your access method | Method | Auth | Setup | Best for | | --- | --- | --- | --- | | **Web UI Query Panel** | VMS login | None | Spot-checks, confirming data, non-technical users | | **VMS REST API** (vastpy) | VMS creds or token | Low | Admin scripts, a gated internal service | | **VAST DB SDK** | S3 key + `s3:Tabular*` identity policy | Low | Trusted analysts, bulk programmatic extracts | - **Just need to look, or confirm data exists?** Use the Web UI Query Panel ([§2A](/develop/docs/how-to-audit-db-query-guide#path-a-the-web-ui-query-panel)). No keys, no policy, nothing to install. - **Automating, or building a service that hands out per-user scoped results?** Use the VMS REST API ([§2B](/develop/docs/how-to-audit-db-query-guide#path-b-the-vms-rest-api)). One privileged credential: your service applies any per-user path scoping ([§5](/develop/docs/how-to-audit-db-query-guide#scoping-what-users-see-by-path1)). - **Bulk extracts, pipelines, embedding queries in tooling?** Use the VAST DB SDK ([§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1)); the only setup is a key and one identity policy. One design point worth knowing up front: the query grant (`s3:TabularQueryData`) operates at the table level, so per-user path scoping is done in a small query service rather than in the policy itself. Section 5 shows the pattern. --- ### [](/develop/docs/how-to-audit-db-query-guide#two-audiences-two-starting-points)Two audiences, two starting points - **Administrators/platform team**: [§2](/develop/docs/how-to-audit-db-query-guide#2-query-the-auditdb1) to get data flowing today, then [§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1) to set up SDK access and scoped access. - **Analysts**: start at [§2](/develop/docs/how-to-audit-db-query-guide#2-query-the-auditdb1) and keep [§3 Troubleshooting](/develop/docs/how-to-audit-db-query-guide#3-troubleshooting1) nearby. --- ## 2. Query the AuditDB Two paths read audit records with nothing more than VMS credentials: the Web UI Query Panel and the VMS REST API. Both read the audit table through the management plane, so they work as soon as auditing is enabled. Start here and move to the [SDK path](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1) when you need bulk extracts or want to embed queries in tooling. --- ## [](/develop/docs/how-to-audit-db-query-guide#path-a-the-web-ui-query-panel)Path A: the Web UI Query Panel The fastest way to look at audit data. Nothing to install. 1. In VMS, go to `DataBase → VAST Audit Log`. The grid shows recent audit records. 2. Use the **Protocol** drop-down (upper-right) to filter by NFS, SMB, S3, and so on. 3. Click the **column-selector** icon to choose which fields are shown. 4. Click **Open Query Panel** to build a query: - In the **Active Query** tab, pick a **Column name**, choose a **Select operator** (the operators offered depend on the column's type, e.g., *is exactly*, *is null*), and enter a value. - Add more clauses with **& (AND)**, or alternatives with **OR**. Click **Add** to add each condition to the list. - Click **Execute Query**. Results appear in the grid. 5. Use the **download** icon to export the currently displayed rows to **CSV**. Good for spot-checks, confirming data exists, and giving a non-technical user a way to look without credentials or code. --- ## [](/develop/docs/how-to-audit-db-query-guide#path-b-the-vms-rest-api)Path B: the VMS REST API For anything programmatic, query the same table through the VMS management API with [vastpy](/develop/docs/how-to-audit-db-query-guide#vastpy-cheatsheet1). This is the path to automate on and the natural base for a gated internal service. ### [](/develop/docs/how-to-audit-db-query-guide#authenticate-to-vms)Authenticate to VMS Point vastpy at the management host, the same hostname you open the GUI with, as a bare name (vastpy adds `https://` itself): ```plaintext export VMS_ADDRESS='var204.selab.vastdata.com' export VMS_USER='admin' export VMS_PASSWORD='********' # or, VAST 5.3+:  export VMS_TOKEN='' ``` The S3 data endpoint is a different address and only speaks S3; pointing vastpy at it returns `400 CredentialsNotSupported`. Management calls always go to the management host. ### [](/develop/docs/how-to-audit-db-query-guide#pull-audit-rows)Pull audit rows The endpoint is `vastauditlog/query_data` (a POST). Pass `fields` (columns to return), `filters` (server-side row filters), and `limit`: ```plaintext import urllib3; urllib3.disable_warnings() from vastpy import VASTClient c = VASTClient(address="var204.selab.vastdata.com", user="admin", password="********") resp = c.vastauditlog.query_data.post(    limit=50,    fields=["time", "login_name", "protocol", "rpc_type", "view_path", "num_bytes"],    filters={        "s3_bucket_name": [{"eq": "mputest"}],   # server-side filter (operator: eq)        # "view_path":    [{"startswith": "/bgolliher/"}],   # verify operator name on your version    }, ) # resp is {"prop_list": [...columns...], "results": [[...row...], ...], "count": N} cols = resp["prop_list"] for row in resp["results"]:    print(dict(zip(cols, row))) ``` The same call from the CLI (which renders the result as a table): ```plaintext vastpy-cli --address var204.selab.vastdata.com --user admin --password '********' \  post vastauditlog/query_data limit=20 \  filters='{"s3_bucket_name":[{"eq":"mputest"}]}' \  fields='["time","login_name","protocol","rpc_type","view_path","num_bytes"]' ``` A ready-made, filterable version is in `examples/query_vms_api.py`: ```plaintext export VMS_ADDRESS=var204.selab.vastdata.com VMS_USER=admin VMS_PASSWORD='********' python query_vms_api.py --bucket mputest --top 10 python query_vms_api.py --path /bgolliher/ --top 20 ``` ### [](/develop/docs/how-to-audit-db-query-guide#filter-syntax)Filter syntax Every call includes a `filters` field shaped as `{"<field>": [{"<operator>": <value>}]}`. The endpoint requires it; a call without one returns `400 filters - field required`. The always-available operator is `eq`; others such as `startswith` and range operators vary by version, and the full set for your cluster is listed at `https://<vms>/docs/`. Field names are the snake_case column names from [§4](/develop/docs/04-schema-and-query-recipes.md); `view_path` is handy for path filtering because it's a flat string (`/bgolliher/mputest`). --- ### [](/develop/docs/how-to-audit-db-query-guide#scoping-results-by-path)Scoping results by path Both paths filter by path: a `view_path` filter in Path B, or a Query Panel condition in Path A. That filtering is applied by the caller, so it's a convenience rather than a per-user restriction. - For a convenience filter ("just show me `/bgolliher/*`"), use the `view_path` filter above. - To restrict each requester to their own prefix, wrap Path B in a thin service that injects the caller's allowed prefix into `filters` and holds the VMS credential itself. Your code is the enforcement point, and this is the recommended way to give UserA `/bgolliher/*` and nobody else's paths. See [§5](/develop/docs/how-to-audit-db-query-guide#scoping-what-users-see-by-path1). --- ### [](/develop/docs/how-to-audit-db-query-guide#when-to-use-the-sdk)When to use the SDK Move to [§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1) when you need the VAST DB SDK for large programmatic extracts or to embed queries in tooling. That path reads the table over the S3/tabular data endpoint and needs only an S3 key and one identity policy, covered there. --- ## 3. Troubleshooting Three problems account for almost every support case: the SDK reports a missing bucket, a query runs but returns nothing, or a query never seems to finish. All have straightforward fixes. > **Fastest way to get unblocked:** the [Web UI and VMS API paths](/develop/docs/how-to-audit-db-query-guide#2-query-the-auditdb1) read the same audit data with just VMS credentials. If you're stuck on the SDK, use those to keep working while you fix it. --- ### [](/develop/docs/how-to-audit-db-query-guide#error-1-missingbucket-on-the-sdk-path)Error 1: `MissingBucket` on the SDK path ```plaintext vastdb.errors.MissingBucket: {'bucket': 'vast-audit-log-bucket'} ``` This means the code opened the bucket with `tx.bucket("vast-audit-log-bucket")`. That call starts with an S3 `HEAD Bucket` check, and the AuditDB is a **managed** bucket that doesn't answer S3 bucket calls (it isn't backed by a view, so `HEAD` returns 404, and it's absent from `ListBuckets`) — even though tabular queries against it work fine. The fix is to open it the way the SDK intends, by a well-known name: ```plaintext with session.transaction() as tx:    table = tx.audit_log()          # instead of tx.bucket("vast-audit-log-bucket")... ``` `tx.audit_log()` skips the S3-level check and goes straight to the tabular API ([§5](/develop/docs/how-to-audit-db-query-guide#5-advanced-sdk-access-path-scoping1)). If the query then fails with an access error, work down this list: | # | Check | How | | --- | --- | --- | | 1 | **Identity policy with** `s3:TabularList*` **+** `s3:TabularQueryData`**, attached to the user?** | Creating a policy is separate from attaching it. See [§5](/develop/docs/how-to-audit-db-query-guide#grant-access-the-identity-policy1). | | 2 | **Auditing routed to VAST DB?** | `vastpy-cli get clusters/1/auditing` should show `enable_vast_db_audit: true` and `vast_audit_log_state: CREATED` ([§0](/develop/docs/how-to-audit-db-query-guide#enable-audit-logging-to-the-database)). | | 3 | **Right endpoint?** | Connect to the VAST DB **data VIP** (`http://main.<cluster>...`), not the VMS management address. | A few things that look related but aren't: - **The AuditDB never shows up in** `aws s3 ls` **or** `tx.bucket()`**.** That's by design, not a permissions problem — don't go looking for a view or View Policy to fix it. - **Read-access Users / Groups** (in the global auditing settings) control access to the audit **JSON files**, not the database. Per VAST's *Accessing Audit Log JSON Files via NFS, S3, and SMB*, "access to the Audit Directory and its contents is controlled by the Audit settings." Being on that list is neither necessary nor sufficient for DB access. - **An S3 bucket named** `audit` is the JSON file bucket (it holds `audit_env_N/` folders of `audit_log_*` blobs, owned by root, read-only). It's a different path with a different access model; the database is reached only through the tabular API. --- ### [](/develop/docs/how-to-audit-db-query-guide#error-2-empty-or-unexpected-results)Error 2: empty or unexpected results The query runs without errors but returns no rows or gives different results than you expect. Causes, most common first: #### [](/develop/docs/how-to-audit-db-query-guide#2a-wrong-rpc_type-or-protocol-value)2a. Wrong `rpc_type` or `protocol` value `rpc_type` values are case-sensitive, and the exact spelling is cluster/version-specific. The styles, by protocol: | Protocol | Style(s) | Examples | | --- | --- | --- | | NFSv3 / NFSv4 / SMB | `UPPERCASE` | `WRITE`, `READ`, `CREATE`, `RENAME`, `REMOVE` | | S3 | varies by version: `UPPER_SNAKE` or AWS CamelCase | `PUT_OBJECT` / `GET_OBJECT` / `COPY_OBJECT` (VAST 5.4/5.5) vs `PutObject` / `GetObject` (VAST's *Audited Protocol Operations* doc) | | VAST Database | `UPPER_SNAKE_CASE` | `INSERT_ROWS`, `QUERY_DATA`, `SELECT` | On VAST 5.4/5.5, S3 writes are stored as `PUT_OBJECT`, `COPY_OBJECT`, and `COMPLETE_MULTIPART_UPLOAD`, not the CamelCase `PutObject` the operation-list doc implies. The doc names the *operations*; the audit table stores VAST's internal `rpc_type` spelling, which can change between versions. So a filter like `rpc_type = 'PutObject'` can return nothing on one cluster and everything on another. The fix is simple: list the real values from your own data before writing a filter. ```plaintext python examples/first_query.py --distinct-ops ``` Or directly with the SDK: ```plaintext # SDK: list the distinct rpc_type values in a recent sample import vastdb with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:    t = tx.audit_log()    df = t.select(columns=["protocol", "rpc_type"], limit_rows=100_000).read_all().to_pandas()    print(df.value_counts(["protocol", "rpc_type"])) ``` One version note: before VAST 5.4.1, S3 `DeleteByLifecycleRule` operations are recorded as `DeleteObject`. Filter accordingly on older clusters. The `protocol` column has the same property: the stored spelling can vary by release (`NFSv3` vs `NFS3` vs `NFSV3`), and NFSv4.0 and 4.1 share a single NFSv4 value. Filter with `isin` over the known variants rather than a single equality — `examples/audit_timeslice.py` shows the pattern: ```plaintext predicate &= _.protocol.isin(["NFSv3", "NFS3", "NFSV3"])   # matches every spelling ``` #### [](/develop/docs/how-to-audit-db-query-guide#2b-time-range-filters)2b. Time-range filters The `time` column is a `timestamp(9)`. If your window is wrong (timezone, UTC vs local, or a range in the future), you get zero rows. Widen the window to sanity-check, then narrow. #### [](/develop/docs/how-to-audit-db-query-guide#2c-the-path-filter-and-the-nested-struct)2c. The `path` filter and the nested struct `path` is a struct; the string lives at `path.path`. Filtering on `path` directly won't match the way you expect. See [§4](/develop/docs/04-schema-and-query-recipes.md#the-nested-path-column). The audit stream also contains internal noise you usually want to exclude: rows where `path.path` is null or starts with `(null)`, rows under `/.vast_audit_dir/`, and the `audit` service account. In the SDK, exclude the service account in the predicate, and the path noise client-side — **negated prefix filters (**`~startswith`**) don't push down** (they raise `NotImplementedError` on vastdb 2.x; positive `startswith` filters push down fine): ```plaintext from ibis import _ predicate = predicate & (_.login_name != "audit")      # pushes down df = table.select(columns=cols, predicate=predicate, limit_rows=cap).read_all().to_pandas() df["path"] = df["path"].map(lambda p: p.get("path", "") if isinstance(p, dict) else p or "") df = df[~df["path"].str.startswith(("(null)", "/.vast_audit_dir/"))]   # client-side ``` #### [](/develop/docs/how-to-audit-db-query-guide#2d-a-policy-that-seems-to-scope-by-path-only-sometimes)2d. A policy that seems to scope by path only sometimes If a per-user policy appears to restrict paths inconsistently, it is because it expects row-level scoping from a table-level grant. `s3:TabularQueryData` grants the whole table, and narrowing the policy `Resource` to imply a path prefix doesn't filter rows. To scope what a user sees by path, enforce it in a mediated service that adds the caller's allowed prefix to the query filter ([§5](/develop/docs/05-advanced-sdk-access.md#scoping-what-users-see-by-path)). --- ### [](/develop/docs/how-to-audit-db-query-guide#error-3-the-query-never-finishes)Error 3: the query never finishes An SDK query with no row cap fetches **every** matching row before returning, and a busy cluster writes millions of audit rows per hour, so an open-ended `table.select(...).read_all()` can run for a very long time while looking "hung". Filter pushdown itself is fast — a selective predicate plus a row cap return in well under a second even on a large table. Two habits prevent it: ```plaintext # 1. Always pass limit_rows unless you truly want the full result set reader = table.select(columns=cols, predicate=pred, limit_rows=10_000) # 2. Keep time windows tight, and remember rows are not returned newest-first: #    narrow with the predicate, then sort what comes back ``` For genuinely large extracts, iterate the reader in batches instead of `read_all()`, or narrow the window and paginate by time. A related hang happens before any query runs: pointed at the wrong endpoint (the VMS address or a host that isn't reachable), `vastdb.connect()` has no request timeout and retries for about a minute by default. For interactive tools, fail fast instead: ```plaintext from vastdb.config import BackoffConfig session = vastdb.connect(endpoint=EP, access=AK, secret=SK,                         timeout=5, backoff_config=BackoffConfig(max_tries=1)) ``` `examples/audit_timeslice.py` does exactly this (its `--timeout` flag), so a mistyped endpoint fails in seconds with a pointer to the fix. --- ### [](/develop/docs/how-to-audit-db-query-guide#quick-triage-flow)Quick triage flow ```plaintext MissingBucket from the SDK?             → Error 1: open by name — tx.audit_log(), not tx.bucket() Query runs but returns 0 rows?          → 2a rpc_type value, then 2b time, then 2c path/noise Data shows in the UI but not my query?  → connection/filters (2a-2c), not auditing Trying to scope a policy by path?       → 2d: use a mediated service (§5) Query hangs / runs forever?             → Error 3: add limit_rows, tighten the window Connect itself hangs?                   → Error 3: wrong endpoint — use the DB data VIP; cap timeout Nothing in the UI grid either?          → auditing isn't writing to the DB (§1 prerequisites) ``` --- ## 4. Schema & Query Recipes Recipes are shown with the VAST DB SDK (ibis/pandas). The same filters map directly onto the VMS REST API (`fields` + `filters`, see [§2](/develop/docs/how-to-audit-db-query-guide#filter-syntax)) and onto the Web UI Query Panel's conditions. --- ### [](/develop/docs/how-to-audit-db-query-guide#the-columns-youll-actually-use)The columns you'll actually use The audit table has around 50 columns. These are the ones most queries touch: | Column | Type | Meaning | | --- | --- | --- | | `time` | `timestamp(9)` | When the operation happened (UTC). | | `login_name` | `varchar` | Resolved username, when **Log Username** is on, and the identity resolves to a name (S3: the VAST user owning the access key; NFS: resolved from the UID via the cluster's provider). Blank when unresolvable. | | `uid` | `integer` | UNIX user ID, when the requesting identity has one (e.g. AD users with UID mappings, including over S3). **Null when it doesn't** (common for pure S3/SMB identities), which pandas shows as `NaN` — expected, not a bug. | | `sid` | `varchar` | SMB SID. | | `client_ip` | `varchar` | Client IP the request came from (see also `true_client_ip`). | | `cnode_name` | `varchar` | Which CNode served the request (with `cluster_name` / `cluster_vip`). | | `protocol` | `varchar` | `NFSv3`, `NFSv4`, `SMB`, `S3`, `NDB` (VAST DB traffic). Spelling can vary by release (`NFSv3` vs `NFS3`), and NFSv4.0/4.1 share one NFSv4 value — filter with `isin` over the variants ([§3 2a](/develop/docs/how-to-audit-db-query-guide#2a-wrong-rpctype-or-protocol-value1)). | | `rpc_type` | `varchar` | The operation. Case-sensitive and version-specific; see below. | | `path` | `row(path varchar, phandle ...)` | Nested. The path string is `path.path`. | | `view_path` | `varchar` | Path relative to the view (present when full-path logging is off). | | `status` | `varchar` | The protocol-level result: `Success`, or the protocol's error name (e.g. `NFS3ERR_STALE` for NFS, `NoSuchKey` for S3). There is no separate numeric response-code column. | | `s3_used_access_key` | `varchar` | Which access key made the S3 request. | | `tenant` | `varchar` | Tenant the access was in. | | `num_ops` | `integer` | Operation count for the record. | | `num_bytes` | `bigint` | Bytes moved (great for "who wrote how much"). | | `s3_bucket_name` | `varchar` | S3 bucket, for S3 operations. | | `object_type` | `varchar` | `FILE`, `DIRECTORY`, `OBJECT`, `BUCKET`, `SYMLINK`, `UNKNOWN`. | The full column list is in `../db-audit-qry/audit-schema.md`. ### [](/develop/docs/how-to-audit-db-query-guide#s3-specific-columns)S3-specific columns On `protocol = 'S3'` rows, these are also available (the `nfs*_` / `smb_*` columns are null): | Column | Populated | Meaning | | --- | --- | --- | | `s3_bucket_name` | always | The bucket — the most reliable S3 filter (bucket-level ops have no `view_path`). | | `name` | always | Nested struct; `name.name` is the object key within the bucket. | | `s3_used_access_key` | always | The access key that made the request. | | `s3_access_keys` | always | The identity's access keys (list). | | `s3_request_id` | always | Request ID, correlates with client-side S3 errors. | | `connection_type` | always | `HTTP` or `HTTPS`. | | `object_type` | always | `OBJECT`, `BUCKET`, ... | | `s3_version_id` | always | Object version; `-1` on unversioned buckets. | | `s3_multipart_upload_id` | multipart ops | Ties `*_MULTIPART_UPLOAD` parts together. | | `s3_source_object` | `COPY_OBJECT` | The copy source. | | `true_client_ip` | behind a proxy/LB | Original client IP when `client_ip` is the proxy. | ### [](/develop/docs/how-to-audit-db-query-guide#a-note-on-field-names-three-different-spellings)A note on field names (three different spellings) The same field appears under three naming conventions depending on where you look: | Where | Convention | Example | | --- | --- | --- | | This DB table (SDK / VMS-API queries) | `snake_case` | `login_name`, `rpc_type`, `client_ip` | | VAST's *Audit Log Record Fields* doc | `CamelCase` | `LoginName`, `RPCType`, `ClientIP` | | The exported JSON audit files | `PascalCase` | `LoginName`, `RPCType`, `ClientIP` | When querying, always use the `snake_case` column names. ### [](/develop/docs/how-to-audit-db-query-guide#the-nested-path-column)The nested `path` column `path` is a struct: `row(path varchar, phandle row(...))`. The readable path string is the inner `.path` field. ```plaintext # SDK (ibis predicate) from ibis import _ predicate = _.path.path.startswith("/projects/") ``` ```plaintext # After read_all() → pandas, the column comes back as a dict: df["full_path"] = df["path"].apply(lambda p: p["path"] if isinstance(p, dict) else None) ``` For the VMS API path, filter on the flat `view_path` column instead (e.g. `/bgolliher/mputest`). --- ### [](/develop/docs/how-to-audit-db-query-guide#rpc_type-value-reference)`rpc_type` value reference Values are case-sensitive and version-specific, so treat these as starting points and confirm against your own data (one query, shown below). - **NFSv3 / NFSv4** (`UPPERCASE`): `CREATE`, `OPEN`, `WRITE`, `READ`, `SETATTR`, `MKDIR`, `REMOVE`, `RMDIR`, `RENAME`, `LINK`, `SYMLINK`, `READDIR`, `GETATTR`, `ACCESS`. - **SMB** (`UPPERCASE`): `CREATE`, `WRITE`, `READ`, `SET_INFO`, `CLOSE`, `IOCTL`, `QUERY_DIRECTORY`, `SESSION_SETUP`, `LOGOFF`. - **S3**: spelling varies by version. On VAST 5.4/5.5, the stored values are `UPPER_SNAKE`: `PUT_OBJECT`, `GET_OBJECT`, `COPY_OBJECT`, `COMPLETE_MULTIPART_UPLOAD`, `DELETE_OBJECT`. VAST's *Audited Protocol Operations* doc lists them in AWS CamelCase (`PutObject`, ...); the audit table stores VAST's internal spelling. - **VAST Database** (`UPPER_SNAKE_CASE`): `INSERT_ROWS`, `UPDATE_ROWS`, `DELETE_ROWS`, `IMPORT_DATA`, `SELECT`, `QUERY_DATA`, `CREATE_TABLE`, `DROP_TABLE`, `ALTER_TABLE`. List the real values from your data with the distinct-values query in [§3](/develop/docs/03-troubleshooting.md#2a-wrong-rpc_type-or-protocol-value), or: ```plaintext python examples/first_query.py --distinct-ops ``` --- ### [](/develop/docs/how-to-audit-db-query-guide#recipe-queries-vast-db-sdk)Recipe queries (VAST DB SDK) Each assumes an open `table` handle: ```plaintext import vastdb from ibis import _ with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:    table = tx.audit_log()   # the AuditDB, by its built-in well-known name    ...                      # recipes below run inside this block ``` One habit to keep: pass `limit_rows` unless you want every matching row — on a busy cluster, an unbounded window holds millions of rows ([§3 Error 3](/develop/docs/how-to-audit-db-query-guide#3-troubleshooting1)). Fuller implementations (CSV/JSON export, path aggregation) are in `../db-audit-qry/`. And for the most common report of all — everything one protocol did in a time window — `examples/audit_timeslice.py` is ready to run: ```plaintext python examples/audit_timeslice.py --s3 --minutes 60 --filter getObject,putObject python examples/audit_timeslice.py --nfs3 --filter ACCESS python examples/audit_timeslice.py --nfsv4.1 --start '2026-07-15 10:00' --json | jq . ``` It normalizes operations’ spellings to the stored `UPPER_SNAKE` form, matches every `protocol` spelling variant, excludes the audit noise, and caps the fetch — the habits this section teaches, in one place. #### [](/develop/docs/how-to-audit-db-query-guide#recipe-1-recent-activity-for-one-user)Recipe 1: Recent activity for one user ```plaintext import ibis pred = (_.login_name == "jsmith") & (_.time >= ibis.literal(one_day_ago, type="timestamp")) rows = table.select(    columns=["time", "protocol", "rpc_type", "path", "status", "num_bytes"],    predicate=pred, limit_rows=10_000, ).read_all().to_pandas() ``` #### [](/develop/docs/how-to-audit-db-query-guide#recipe-2-bytes-written-by-path-prefix-depth-3-folders)Recipe 2: Bytes written by path prefix (depth-3 folders) Query WRITE rows, then drop the noise paths and aggregate the truncated path in pandas (negated prefix filters don't push down, so the exclusion is client-side — [§3 2c](/develop/docs/how-to-audit-db-query-guide#2c-the-path-filter-and-the-nested-struct1)): ```plaintext pred = (_.rpc_type == "WRITE") & (_.time >= ibis.literal(week_ago, type="timestamp")) df = table.select(columns=["path", "num_bytes", "num_ops"], predicate=pred,                  limit_rows=1_000_000).read_all().to_pandas() df["full"] = df["path"].apply(lambda p: p.get("path") if isinstance(p, dict) else None) df = df[~df["full"].str.startswith(("(null)", "/.vast_audit_dir/"), na=True)] df["prefix"] = df["full"].str.split("/").str[:4].str.join("/")   # /a/b/c depth-3 agg = df.groupby("prefix").agg(bytes=("num_bytes", "sum"), ops=("num_ops", "sum")) \        .sort_values("bytes", ascending=False) ``` The full version is `../db-audit-qry/audit_writes_by_path.py`. For S3 writes, filter `rpc_type == "PUT_OBJECT"` (5.4/5.5) rather than `"WRITE"`. #### [](/develop/docs/how-to-audit-db-query-guide#recipe-3-top-writers-across-everyone-last-7-days)Recipe 3: Top writers across everyone (last 7 days) ```plaintext pred = (_.rpc_type.startswith("WRITE")) & (_.time >= ibis.literal(week_ago, type="timestamp")) \       & (_.login_name != "audit") df = table.select(columns=["login_name", "uid", "num_bytes"], predicate=pred).read_all().to_pandas() df["user"] = df["login_name"].where(df["login_name"] != "", df["uid"].astype(str)) top = df.groupby("user")["num_bytes"].sum().sort_values(ascending=False).head(50) ``` #### [](/develop/docs/how-to-audit-db-query-guide#recipe-4-who-deleted-or-renamed-something-under-a-path)Recipe 4: Who deleted or renamed something under a path ```plaintext pred = (_.rpc_type.isin(["REMOVE", "RMDIR", "RENAME", "DELETE_OBJECT"])) \       & (_.path.path.startswith("/projects/")) \       & (_.time >= ibis.literal(month_ago, type="timestamp")) rows = table.select(    columns=["time", "login_name", "rpc_type", "path", "rename_path", "name"],    predicate=pred, limit_rows=10_000, ).read_all().to_pandas() ``` S3 delete/rename spelling is version-specific; confirm with `--distinct-ops` before relying on `DELETE_OBJECT`. --- ## 5. Advanced: SDK Access & Path Scoping The VAST DB SDK reads the audit table the same way it reads any VAST Database table: over S3, with an access key. Use it for bulk extracts, pipelines, and embedding audit queries in tooling. This section walks through the setup, then shows how to scope what different users see by path. --- ### [](/develop/docs/how-to-audit-db-query-guide#connect-and-query)Connect and query Three steps: install, point at the data endpoint, open the audit table by name. ```plaintext pip install vastdb pyarrow pandas ibis-framework export VASTDB_ENDPOINT='http://'   # S3/tabular data VIP, not the VMS address export VASTDB_ACCESS_KEY='' export VASTDB_SECRET_KEY='' ``` ```plaintext import os, vastdb from ibis import _ session = vastdb.connect(endpoint=os.environ["VASTDB_ENDPOINT"],                         access=os.environ["VASTDB_ACCESS_KEY"],                         secret=os.environ["VASTDB_SECRET_KEY"]) with session.transaction() as tx:    table = tx.audit_log()                # the AuditDB, by its built-in well-known name    batch = table.select(        columns=["time", "login_name", "protocol", "rpc_type", "path", "status"],        predicate=(_.login_name != "audit"),        limit_rows=100,    ).read_all()    print(batch.to_pandas()) ``` That's the whole pattern. `tx.audit_log()` returns a regular table handle, so everything in [§4](/develop/docs/04-schema-and-query-recipes.md) (predicates, the nested `path` column, recipes) applies as-is. The starter `examples/first_query.py` wraps this with time and path filters; `examples/audit_timeslice.py` builds it out into a full per-protocol report (NFSv3, NFSv4, or S3, one time window). One habit for command-line tools: pass `timeout=` and `backoff_config=BackoffConfig(max_tries=1)` (from `vastdb.config`) to `vastdb.connect()` so a mistyped endpoint fails in seconds instead of retrying for a minute ([§3 Error 3](/develop/docs/how-to-audit-db-query-guide#error-3-the-query-never-finishes1)). ### [](/develop/docs/how-to-audit-db-query-guide#the-auditdb-is-a-managed-bucket)The AuditDB is a managed bucket The audit table lives at fixed coordinates on every cluster — `vast-audit-log-bucket` / `vast_audit_log_schema` / `vast_audit_log_table` — and the SDK knows them (`tx.audit_log()`, just like `tx.catalog()` for the VAST Catalog). Because the bucket is managed by the cluster rather than backed by a view you created, it behaves differently from your own DB buckets in one visible way: **it doesn't answer S3 bucket calls.** It won't appear in `ListBuckets` (`aws s3 ls`), and `tx.bucket("vast-audit-log-bucket")` fails with `MissingBucket` because that call begins with an S3 `HEAD Bucket` check. This is normal — open it by name and query it ([§3 Error 1](/develop/docs/how-to-audit-db-query-guide#error-1-missingbucket-on-the-sdk-path1)). On an SDK too old to have `tx.audit_log()` (it's present in vastdb 2.x) — or in a tool that keeps the coordinates configurable, as `examples/audit_timeslice.py` does — construct the handle by name explicitly. It's the same open-by-name operation: ```plaintext import vastdb.bucket, vastdb.schema b = vastdb.bucket.Bucket("vast-audit-log-bucket", tx) table = vastdb.schema.Schema("vast_audit_log_schema", b).table("vast_audit_log_table") ``` --- ### [](/develop/docs/how-to-audit-db-query-guide#grant-access-the-identity-policy)Grant access: the identity policy The querying user needs an S3 access key and one read-only identity policy with the tabular actions. That's the entire permission model for the SDK path — the audit bucket has no View Policy to join, because it isn't backed by a view. ```plaintext {  "Version": "2012-10-17",  "Statement": [    { "Sid": "ListTabular", "Effect": "Allow",      "Action": "s3:TabularList*",      "Resource": ["vast-audit-log-bucket", "vast-audit-log-bucket/vast_audit_log_schema/*"] },    { "Sid": "QueryTabular", "Effect": "Allow",      "Action": "s3:TabularQueryData",      "Resource": ["vast-audit-log-bucket/vast_audit_log_schema/*"] }  ] } ``` - `s3:TabularList*` lets the SDK resolve the schema and table; `s3:TabularQueryData` performs the read. Both are needed for a working read-only client. - **Resource forms** (from VAST's *Managing Permissions for Accessing VAST Tabular Databases*): `"<bucket>"` for bucket-level ops, `"<bucket>/<schema>/*"` for schema-level, `"<bucket>/<schema>/<table>"` for table-level. The `arn:aws:s3:::<bucket>/...` form is also accepted on some versions. - This grant is **table-level**, so everyone who has it can query every row. To give a user only their paths, see [Scoping](/develop/docs/how-to-audit-db-query-guide#scoping-what-users-see-by-path1) below. Set up order, all with vastpy (see the [cheat sheet](/develop/docs/how-to-audit-db-query-guide#vastpy-cheatsheet1)): create the user, create their access key, create the identity policy, attach it to the user. --- ### [](/develop/docs/how-to-audit-db-query-guide#scoping-what-users-see-by-path)Scoping what users see by path The goal: UserA sees only `/bgolliher/*`, UserB sees only their own subtree. The clean way to do this is a small **mediated query service**: a thin service that holds one privileged credential and, for each request, adds that caller's allowed path prefix to the query filter. Callers never query the table directly, so they can only ever see their own paths. - Build it on the [VMS REST API](/develop/docs/how-to-audit-db-query-guide#path-b-the-vms-rest-api) (`vastauditlog/query_data` with a `view_path` filter), which works with just VMS credentials, or on the SDK (a `path.path.startswith(prefix)` predicate). - Your service is the enforcement point: map each authenticated user to their allowed prefixes, reject anything else, and add the prefix to `filters` server-side. ```plaintext # sketch: inside your service, per authenticated request allowed_prefix = lookup_prefix(request.user)          # e.g. "/bgolliher/" resp = client.vastauditlog.query_data.post(    limit=200,    fields=["time", "login_name", "rpc_type", "view_path", "num_bytes"],    filters={"view_path": [{"startswith": allowed_prefix}]},   # confirm operator name on your version ) ``` Two design notes worth knowing: - The file view policy governs file and bucket visibility, not which audit rows a query returns. Row filtering belongs in your service. - The `s3:TabularQueryData` grant is table-level, so the service, not the policy, is where per-user scoping happens. This model is service-enforced (the boundary is your code), which fits most "scope results to the relevant team" needs. ### [](/develop/docs/how-to-audit-db-query-guide#out-of-scope-row-and-column-security)Out of scope: Row and Column Security VAST also offers VAST Database Row and Column Security: identity-policy statements (`"Effect":"RowColumnSecurity"`) that make the database itself filter rows and columns per user. It's the strongest boundary, but it's enforced only by Trino query engines and needs end-user impersonation, so it's out of scope for this guide. If you later need a hard database-enforced boundary, see VAST's *VAST Database Row and Column Security* doc. --- ## 6. Appendix: CLI, Config Links & Verification --- ### [](/develop/docs/how-to-audit-db-query-guide#environment-variables-sdk-companion-scripts)Environment variables (SDK / companion scripts) The `examples/` scripts and the `../db-audit-qry/` tools read these: ```plaintext export VASTDB_ENDPOINT='http://'   # VAST DB data endpoint (not VMS) export VASTDB_ACCESS_KEY='' export VASTDB_SECRET_KEY='' ``` The audit bucket/schema/table names are fixed (`vast-audit-log-bucket` / `vast_audit_log_schema` / `vast_audit_log_table`) and built into the SDK, so they never need to be configured. --- ### [](/develop/docs/how-to-audit-db-query-guide#vastpy-cheat-sheet)vastpy cheat-sheet [vastpy](https://pypi.org/project/vastpy/) is a thin VMS REST client (SDK and `vastpy-cli`). It's a schema-less passthrough: `vastpy-cli <get|post|patch|delete> <endpoint> key=value ...`. Values that look like JSON are parsed as JSON. ```plaintext # --- Auth (env or per-call flags) --- export VMS_ADDRESS=vms.example.com export VMS_USER=admin export VMS_PASSWORD='********' # or:  export VMS_TOKEN=     # VAST 5.3+ # --- Users & keys --- vastpy-cli post users name='auditor' full_name='Audit Analyst' user_type='LOCAL' vastpy-cli post users//access_keys          # or .../s3_access_keys on some versions # --- Identity (S3) policies --- vastpy-cli get  s3policies vastpy-cli post s3policies name='auditdb-readonly' policy="$(cat policy.json)" tenant_id=1 vastpy-cli patch s3policies/ users='["auditor"]'   # attach user (version-dependent) # --- Audit config (admin; full enable procedure in §0) --- vastpy-cli get   clusters                              # find cluster id vastpy-cli get   clusters//auditing        # current settings vastpy-cli patch clusters//auditing --file-input audit-config.json # --- Query audit data via the VMS API (alternative to the SDK) --- # Use POST, and always include `filters` (the endpoint requires it). vastpy-cli post vastauditlog/query_data limit=20 \  filters='{"s3_bucket_name":[{"eq":"mputest"}]}' \  fields='["time","login_name","rpc_type","view_path","num_bytes"]' ``` Example `audit-config.json` (enables auditing and routes to VAST DB; see [§0](/develop/docs/how-to-audit-db-query-guide#or-enable-it-with-the-vms-rest-api) for the full field set and protocol selection): ```plaintext {  "read_access_users": ["vastadmin"],  "read_access_users_groups": ["vastadmin"],  "protocols_audit": {    "log_full_path": true,    "modify_data_md": true,    "create_delete_files_dirs_objects": true  },  "enable_vast_db_audit": true } ``` A note on TLS: vastpy doesn't verify certificates unless you pass `--cert-file` / `cert_file`, and the same goes for `vastdb.connect(..., ssl_verify=False)`. Fine for labs; set a CA bundle in production. --- ### [](/develop/docs/how-to-audit-db-query-guide#vast-db-tabular-permissions)VAST DB tabular permissions Tabular operations are granted by an **identity policy** with `s3:Tabular*` actions, scoped by Resource: `"<bucket>"` (bucket-level, e.g., list schemas), `"<bucket>/*"` (object-level), `"<bucket>/<schema>/*"` (schema-level), `"<bucket>/<schema>/<table>"` (table-level). For the **AuditDB** (a managed bucket, like the VAST Catalog), the identity policy is the whole model — there is no view or View Policy involved ([§5](/develop/docs/how-to-audit-db-query-guide#the-auditdb-is-a-managed-bucket1)). For DB buckets you create yourself (S3-enabled `DATABASE` views), the user must additionally be in the view's View Policy with bucket-listing permission for the bucket to be visible (VAST *Creating View Policies*, step 6). Common tabular actions (identity-policy `Action` values, all `s3:`-prefixed): | Action | Purpose | | --- | --- | | `TabularQueryData` | Read / SELECT rows (returns pages of columns) | | `TabularListSchemas` / `TabularListTables` / `TabularListColumns` | Resolve schema → table → columns | | `TabularGetTableStats` | Row count / used bytes | | `TabularInsertRows` / `TabularUpdateRows` / `TabularDeleteRows` / `TabularImportData` | Write ops | | `TabularCreateSchema` / `TabularCreateTable` / `TabularAddColumns` / `TabularAlter*` / `TabularDrop*` | DDL | | `TabularBeginTransaction` / `TabularCommitTransaction` / `TabularRollbackTransaction` | Transactions | A read-only audit client needs `s3:TabularList*` plus `s3:TabularQueryData`. Setup order: create the user and access/secret key, create the identity policy, assign it to the user. --- ### [](/develop/docs/how-to-audit-db-query-guide#smoke-testing-sdk-access)Smoke-testing SDK access The audit table's coordinates are fixed and built into the SDK, so the smoke test is one call: ```plaintext import vastdb with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:    t = tx.audit_log()    print(t.select(columns=["time", "protocol", "rpc_type"], limit_rows=5)           .read_all().to_pandas()) ``` Remember that the audit bucket won't show up in `aws s3 ls` or answer `tx.bucket()` — That's expected for a managed bucket ([§5](/develop/docs/how-to-audit-db-query-guide#the-auditdb-is-a-managed-bucket1)). --- ### [](/develop/docs/how-to-audit-db-query-guide#official-vast-configuration-docs)Official VAST configuration docs The configuration side (turning auditing on, choosing what to audit) is covered by these VAST KB pages; link to your customer's versioned copies: | Topic | VAST KB page | | --- | --- | | Concept overview | *Protocol Auditing Overview* | | Turn auditing on, route to VAST DB, retention | *Configuring Global Auditing Settings* | | Per-view-policy auditing | *Configuring Auditing with View Policies* | | Which operations are captured (per protocol) | *Audited Protocol Operations* | | Field/column reference | *Audit Log Record Fields* | | Viewing logs (UI query panel, JSON files) | *Viewing Protocol Audit Logs* | | Tabular DB access: View Policy + identity policy, tabular actions | *Managing Permissions for Accessing VAST Tabular Databases* | | JSON file export over NFS/S3/SMB (out of scope here) | *Accessing Audit Log JSON Files via NFS, S3, and SMB* | | Per-user row/column boundary (Trino; out of scope here) | *VAST Database Row and Column Security* | --- ### [](/develop/docs/how-to-audit-db-query-guide#verification-checklist)Verification checklist Items below marked as verified were confirmed live on VAST 5.5.0.1 and/or 5.4.4.0 clusters. Anything version-sensitive is worth re-confirming on your own cluster before you rely on it. **Verified live (5.5.0.1 and 5.4.4.0):** - The VMS API path works with just VMS credentials: `vastauditlog/query_data` via vastpy returned real rows on both versions. Params used: `limit`, `fields` (snake_case columns), and `filters` (`{"<field>":[{"eq":<v>}]}`; the `filters` field is required). - The SDK path works with just an S3 key and identity policy (verified on 5.4.4.0, vastdb 2.0.14): `tx.audit_log()` opened the table and predicate queries returned rows. No `DATABASE` view for the audit bucket exists or is needed; `tx.bucket()` / `ListBuckets` correctly do not see it. - Predicate pushdown is fast (a time+protocol+rpc_type filter with `limit_rows=20` returned in under a second), but an **unbounded** `read_all()` over a busy window can run for many minutes — always cap or batch ([§3 Error 3](/develop/docs/how-to-audit-db-query-guide#error-3-the-query-never-finishes1)). - Pushdown supports equality, ranges, `isin`, `isnull`, and positive `startswith` (including on the nested `path.path`), but **not negated** `startswith` (`~_.x.startswith(...)` raises `NotImplementedError` on vastdb 2.0.14) — apply noise-exclusion filters client-side ([§3 2c](/develop/docs/03-troubleshooting.md#2c-the-path-filter-and-the-nested-struct)). - `rpc_type` values for S3 are stored as `UPPER_SNAKE` on both versions: `PUT_OBJECT`, `GET_OBJECT`, `COPY_OBJECT`, `COMPLETE_MULTIPART_UPLOAD`. List the distinct values on your own cluster before writing filters. (Before 5.4.1, `DeleteByLifecycleRule` is recorded as `DeleteObject`.) - `status` holds `Success` or the protocol's error name (observed: `NFS3ERR_STALE`, `NoSuchKey`); `cnode_name` records which CNode served each request. - The S3/tabular data VIP and the VMS management host are different addresses. vastpy talks to the management host; pointing it at the S3 endpoint returns `400 CredentialsNotSupported`. - `vastdb.connect()` has no request timeout and retries for about a minute by default, so a wrong endpoint looks like a hang. The example scripts pass `timeout=` and `backoff_config=BackoffConfig(max_tries=1)` to fail fast ([§3 Error 3](/develop/docs/03-troubleshooting.md#error-3-the-query-never-finishes)). **Confirm on your cluster:** - [ ] VMS API filter operators beyond `eq` (such as `startswith` for `view_path`), from `https://<vms>/docs/`. - [ ] The vastpy key-creation sub-resource (`access_keys` vs `s3_access_keys`) and the policy-attachment call on your version. Row and Column Security (a Trino-enforced per-user boundary) is out of scope for this guide; see VAST's *VAST Database Row and Column Security* doc if you ever need a hard database-enforced boundary. --- **audit_timeslice.py** ```python #!/usr/bin/env python3 """ audit_timeslice.py: show AuditDB records for one protocol in a time window. Queries the audit table via the vastdb SDK, opening the managed audit bucket by its well-known name (like tx.audit_log(); see ../05-advanced-sdk-access.md). The querying user just needs an identity policy with s3:TabularList* + s3:TabularQueryData. SETUP    pip install vastdb pyarrow pandas ibis-framework    # Either export these...    export VASTDB_ENDPOINT='http://'   # VAST DB data endpoint, NOT the VMS address    export VASTDB_ACCESS_KEY=''    export VASTDB_SECRET_KEY=''    # ...or pass --endpoint/--access-key/--secret-key. Missing config fails immediately;    # an unreachable endpoint gives up after --timeout seconds (default 5). USAGE    python audit_timeslice.py --s3                                  # last 15 min of S3 activity    python audit_timeslice.py --s3 --filter getObject,putObject    python audit_timeslice.py --nfs3 --minutes 60 --filter ACCESS    python audit_timeslice.py --nfsv4.1 --start '2026-07-15 10:00' --filter open    python audit_timeslice.py --s3 --json | jq . """ import argparse import os import re import sys from datetime import datetime, timedelta, timezone try:    import ibis    import vastdb    from ibis import _    from vastdb.config import BackoffConfig except ImportError:    sys.exit("Missing deps. Run: pip install vastdb pyarrow pandas ibis-framework") # The stored `protocol` spelling varies by release, so match every known variant. PROTOCOLS = {    "nfs3": ["NFSv3", "NFS3", "NFSV3"],    "nfsv41": ["NFSv4", "NFS4", "NFSV4", "NFSv4.1"],  # the table doesn't split 4.0/4.1    "s3": ["S3"], } COLUMNS = ["time", "login_name", "uid", "client_ip",           "protocol", "rpc_type", "path", "status", "num_bytes"] def parse_args():    p = argparse.ArgumentParser(        description="Show all AuditDB records for one protocol in a time window.",        formatter_class=argparse.ArgumentDefaultsHelpFormatter,    )    proto = p.add_mutually_exclusive_group(required=True)    proto.add_argument("--nfs3", action="store_true", help="Show NFSv3 records.")    proto.add_argument("--nfsv4.1", dest="nfsv41", action="store_true", help="Show NFSv4 records.")    proto.add_argument("--s3", action="store_true", help="Show S3 records.")    p.add_argument("--minutes", type=int, default=15, help="Length of the time window.")    p.add_argument("--start", metavar="'YYYY-MM-DD HH:MM'",                   help="Window start (UTC unless an offset is given); default is MINUTES ago.")    p.add_argument("--filter", dest="ops", metavar="OP[,OP...]",                   help="Only these rpc_type ops, e.g. getObject,putObject or ACCESS."                        "Spelling is normalized to the stored UPPER_SNAKE form.")    p.add_argument("--json", action="store_true", help="Emit JSON lines instead of a table.")    p.add_argument("--limit", type=int, default=1000,                   help="Max rows to fetch; busy windows can hold millions. 0 = no cap.")    p.add_argument("--endpoint", default=os.environ.get("VASTDB_ENDPOINT"),                   help="VAST DB data endpoint (env: VASTDB_ENDPOINT)")    p.add_argument("--access-key", default=os.environ.get("VASTDB_ACCESS_KEY"),                   help="S3 access key (env: VASTDB_ACCESS_KEY)")    p.add_argument("--secret-key", default=os.environ.get("VASTDB_SECRET_KEY"),                   help="S3 secret key (env: VASTDB_SECRET_KEY)")    p.add_argument("--bucket", default=os.environ.get("VASTDB_AUDIT_BUCKET", "vast-audit-log-bucket"))    p.add_argument("--schema", default=os.environ.get("VASTDB_AUDIT_SCHEMA", "vast_audit_log_schema"))    p.add_argument("--table", default=os.environ.get("VASTDB_AUDIT_TABLE", "vast_audit_log_table"))    p.add_argument("--insecure", action="store_true",                   help="Disable TLS verification (self-signed lab certs only).")    p.add_argument("--timeout", type=float, default=5,                   help="Seconds before giving up on an unresponsive endpoint.")    args = p.parse_args()    missing = [n for n in ("endpoint", "access_key", "secret_key") if not getattr(args, n)]    if missing:        p.error("missing required config: " + ", ".join(missing)                + " (set VASTDB_ENDPOINT / VASTDB_ACCESS_KEY / VASTDB_SECRET_KEY or pass flags)")    return args def normalize_op(op):    """Map CLI-friendly spellings to the stored form: getObject -> GET_OBJECT, access -> ACCESS."""    return re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", op.strip()).upper() def time_window(args):    """Return the half-open [start, end) window, anchored at --start or trailing from now."""    if args.start:        start = datetime.fromisoformat(args.start)        start = start if start.tzinfo else start.replace(tzinfo=timezone.utc)    else:        start = datetime.now(timezone.utc) - timedelta(minutes=args.minutes)    return start, start + timedelta(minutes=args.minutes) def build_predicate(args, start, end):    flag = next(name for name in PROTOCOLS if getattr(args, name))    predicate = (        (_.time >= ibis.literal(start, type="timestamp"))        & (_.time < ibis.literal(end, type="timestamp"))        & (_.protocol.isin(PROTOCOLS[flag]))        & (_.login_name != "audit")  # the cluster's own audit-table writes    )    if args.ops:        ops = [normalize_op(op) for op in args.ops.split(",") if op.strip()]        predicate &= _.rpc_type.isin(ops)    return predicate def fetch(args, predicate):    # The SDK's defaults (no request timeout, ~60s of retries) hang for minutes on a    # bad endpoint, so cap the timeout and disable retries.    kwargs = dict(endpoint=args.endpoint, access=args.access_key, secret=args.secret_key,                  timeout=args.timeout, backoff_config=BackoffConfig(max_tries=1))    if args.insecure:        kwargs["ssl_verify"] = False    print(f"Connecting to {args.endpoint} (timeout {args.timeout:g}s)...", file=sys.stderr)    try:        session = vastdb.connect(**kwargs)    except Exception as e:        sys.exit(f"Could not connect to {args.endpoint}: {e}\n"                 "Check --endpoint / $VASTDB_ENDPOINT: it must be the VAST DB data VIP "                 "(http://main....), not the VMS/GUI host.")    with session.transaction() as tx:        try:            # Open by well-known name, like tx.audit_log(): the managed audit bucket is not            # an S3 view, so tx.bucket()'s HEAD-bucket pre-check 404s (MissingBucket) even            # when tabular queries against it are allowed.            b = vastdb.bucket.Bucket(args.bucket, tx)            table = vastdb.schema.Schema(args.schema, b).table(args.table)        except Exception as e:            sys.exit(f"Could not open {args.bucket}/{args.schema}/{args.table}: {e}\n"                     "Does your identity policy grant s3:Tabular* on this resource? "                     "See ../03-troubleshooting.md (Error 1).")        return table.select(columns=COLUMNS, predicate=predicate,                            limit_rows=args.limit or None).read_all().to_pandas() def main():    args = parse_args()    start, end = time_window(args)    df = fetch(args, build_predicate(args, start, end))    # `path` is a nested struct (a dict after to_pandas); the string is at .path.    df["path"] = df["path"].map(lambda p: p.get("path", "") if isinstance(p, dict) else p or "")    df = df[~df["path"].str.startswith("/.vast_audit_dir/")].sort_values("time")    if df.empty:        sys.exit(f"No rows between {start:%Y-%m-%d %H:%M:%S} and {end:%H:%M:%S} UTC. "                 "Widen the window, or check the ops with first_query.py --distinct-ops.")    if args.json:        print(df.to_json(orient="records", lines=True, date_format="iso"), end="")    else:        capped = " (hit --limit; not the full window)" if args.limit and len(df) >= args.limit else ""        print(f"{len(df)} row(s){capped}, {start:%Y-%m-%d %H:%M:%S} -> {end:%H:%M:%S} UTC:\n")        print(df.to_string(index=False)) if __name__ == "__main__":    main() ``` **first_query.py** ```python #!/usr/bin/env python3 """ first_query.py: a clean, minimal starting point for querying the VAST AuditDB. WHAT IT DOES    1. Connects to the VAST DB data endpoint with an S3 access/secret key.    2. Opens the audit table by its built-in well-known name: tx.audit_log().       (The AuditDB is a managed bucket; it doesn't appear in bucket listings.)    3. Shows the two things people most often get wrong:         - the nested `path` column (the string is at path.path)         - protocol-specific, case-sensitive rpc_type values    4. Demonstrates a path-scoped query. SCOPING NOTE    The `--path` filter below is applied by this script (client-side); the SDK grant    itself is table-level. To enforce per-user path visibility (UserA sees    /bgolliher/*, UserB can't), front the AuditDB with a mediated service that injects    each caller's allowed prefix; see ../05-advanced-sdk-access.md (Scoping what users    see by path). SETUP    pip install vastdb pyarrow pandas ibis-framework    export VASTDB_ENDPOINT='http://'   # VAST DB data endpoint, NOT the VMS address    export VASTDB_ACCESS_KEY=''    export VASTDB_SECRET_KEY='' USAGE    python first_query.py                                  # last 24h, all paths, 20 rows    python first_query.py --hours 6 --limit 50    python first_query.py --path /bgolliher/ --hours 168   # last 7 days under a path    python first_query.py --distinct-ops                   # list the rpc_type values you actually have """ import argparse import os import sys from datetime import datetime, timedelta, timezone try:    import vastdb    import ibis    from ibis import _ except ImportError:    sys.exit("Missing deps. Run: pip install vastdb pyarrow pandas ibis-framework") def parse_args():    p = argparse.ArgumentParser(        description="Minimal VAST AuditDB query starter.",        formatter_class=argparse.ArgumentDefaultsHelpFormatter,    )    p.add_argument("--endpoint", default=os.environ.get("VASTDB_ENDPOINT"),                   help="VAST DB data endpoint, e.g. http:// (env: VASTDB_ENDPOINT)")    p.add_argument("--access-key", default=os.environ.get("VASTDB_ACCESS_KEY"),                   help="S3 access key (env: VASTDB_ACCESS_KEY)")    p.add_argument("--secret-key", default=os.environ.get("VASTDB_SECRET_KEY"),                   help="S3 secret key (env: VASTDB_SECRET_KEY)")    p.add_argument("--hours", type=int, default=24, help="Look back this many hours.")    p.add_argument("--path", default="", help="Only rows whose path starts with this prefix (client-side).")    p.add_argument("--limit", type=int, default=20, help="Max rows to print.")    p.add_argument("--distinct-ops", action="store_true",                   help="Instead of rows, print the distinct (protocol, rpc_type) values present.")    p.add_argument("--insecure", action="store_true",                   help="Disable TLS verification (self-signed lab certs only).")    args = p.parse_args()    missing = [n for n in ("endpoint", "access_key", "secret_key") if not getattr(args, n)]    if missing:        p.error("missing required config: " + ", ".join(missing) +                " (set VASTDB_ENDPOINT / VASTDB_ACCESS_KEY / VASTDB_SECRET_KEY or pass flags)")    return args def connect(args):    kwargs = dict(endpoint=args.endpoint, access=args.access_key, secret=args.secret_key)    if args.insecure:        kwargs["ssl_verify"] = False    try:        return vastdb.connect(**kwargs)    except Exception as e:        sys.exit(f"Failed to connect to {args.endpoint}: {e}") def get_table(tx):    """Open the AuditDB by its built-in well-known name, with a helpful message on failure."""    try:        # NOT tx.bucket("vast-audit-log-bucket"): the AuditDB is a managed bucket that        # doesn't answer S3 bucket calls, so tx.bucket() raises MissingBucket by design.        return tx.audit_log()    except Exception as e:        sys.exit(            f"\nCould not open the audit table: {e}\n"            "Check, in order:\n"            "  - Does your user have an identity policy allowing s3:TabularList* and\n"            "    s3:TabularQueryData on vast-audit-log-bucket, and is it attached?\n"            "  - Is 'Save audit logs to VAST DB' enabled? (vast_audit_log_state: CREATED)\n"            "  - Is VASTDB_ENDPOINT the VAST DB data VIP (not the VMS address)?\n"            "Tip: the Web UI Query Panel and VMS API path (examples/query_vms_api.py) work\n"            "with just VMS credentials. See ../03-troubleshooting.md (Error 1)."        ) def path_str(cell):    """The `path` column is a nested struct; the string is at .path (a dict after to_pandas)."""    if isinstance(cell, dict):        return cell.get("path") or ""    return cell or "" def main():    args = parse_args()    session = connect(args)    since = datetime.now(timezone.utc) - timedelta(hours=args.hours)    with session.transaction() as tx:        table = get_table(tx)        if args.distinct_ops:            # The single most useful troubleshooting query: what rpc_type values do I actually have?            # Sampled (not exhaustive): an uncapped scan of a busy audit table takes minutes.            df = table.select(columns=["protocol", "rpc_type"],                              limit_rows=200_000).read_all().to_pandas()            counts = df.value_counts(["protocol", "rpc_type"]).reset_index(name="count")            print(f"\nDistinct (protocol, rpc_type) values in a {len(df)}-row sample:")            print(counts.to_string(index=False))            print("\nrpc_type spelling is version-specific: NFS/SMB are UPPERCASE, VAST DB is "                  "UPPER_SNAKE, and S3 is UPPER_SNAKE on 5.5 (PUT_OBJECT) but CamelCase "                  "(PutObject) in VAST's op-list doc. Trust the values printed above.")            return        # Time window + drop the audit service account. (The other noise filters are        # negated prefix matches, which don't push down — they're applied client-side below.)        predicate = (            (_.time >= ibis.literal(since, type="timestamp"))            & (_.login_name != "audit")        )        if args.path:            # Server-side prefix filter on the nested path string (positive prefix: pushes down).            predicate = predicate & (_.path.path.startswith(args.path))        cols = ["time", "login_name", "uid", "client_ip",                "protocol", "rpc_type", "path", "status", "num_bytes"]        # Cap the fetch: a busy window can hold millions of rows (../03, Error 3).        fetch_cap = max(args.limit, 100_000)        df = table.select(columns=cols, predicate=predicate,                          limit_rows=fetch_cap).read_all().to_pandas()        capped = len(df) >= fetch_cap    # Client-side noise exclusion (negated prefix filters can't be pushed down).    df["path"] = df["path"].map(path_str)    df = df[~df["path"].str.startswith(("(null)", "/.vast_audit_dir/"))]    if df.empty:        print(f"\nNo rows in the last {args.hours}h"              + (f" under '{args.path}'" if args.path else "") + ".")        print("If you expected data: check rpc_type casing, the time window, and the path prefix "              "(see ../03-troubleshooting.md, Error 2).")        return    df = df.sort_values("time", ascending=False).head(args.limit)    note = " (sampled: fetch cap hit, so these may not be the newest)" if capped else ""    print(f"\nMost recent {len(df)} of matching rows{note}"          + (f" under '{args.path}'" if args.path else "") + ":\n")    with_cols = ["time", "login_name", "protocol", "rpc_type", "path", "status", "num_bytes"]    print(df[with_cols].to_string(index=False)) if __name__ == "__main__":    main() ``` **query_vms.py** ```plaintext #!/usr/bin/env python3 """ query_vms_api.py: query the VAST AuditDB through the VMS REST API (vastpy). Reads the audit table through the management plane, so it works with just VMS credentials, as soon as auditing is on. Verified on VAST 5.5.0.1. (For the SDK path and per-user path scoping, see ../05-advanced-sdk-access.md.) SETUP    pip install vastpy    export VMS_ADDRESS='var204.selab.vastdata.com'   # VMS/GUI host (bare), not the S3 endpoint    export VMS_USER='admin'    export VMS_PASSWORD='********'    # or:  export VMS_TOKEN=''   (VAST 5.3+) USAGE    python query_vms_api.py --bucket mputest --top 10    python query_vms_api.py --path /bgolliher/ --top 20    python query_vms_api.py --user blake.golliher --op PUT_OBJECT --top 50 NOTES    The query_data endpoint requires a 'filters' field, so pass at least one of    --bucket/--user/--op/--path. The 'eq' operator is confirmed; 'startswith' (used    for --path) may vary by version; see https:///docs/. rpc_type values are    version-specific (on 5.5.0.1, S3 writes are PUT_OBJECT, not PutObject); run with    a broad filter first to see what you have. """ import argparse import os import sys try:    import urllib3    urllib3.disable_warnings()    from vastpy import VASTClient except ImportError:    sys.exit("Missing deps. Run: pip install vastpy") DEFAULT_FIELDS = ["time", "login_name", "uid", "client_ip", "protocol",                  "rpc_type", "view_path", "num_bytes", "status"] def build_client():    address = os.environ.get("VMS_ADDRESS")    if not address:        sys.exit("$VMS_ADDRESS is required (the VMS/GUI hostname, not the S3 data endpoint).")    if address.startswith(("http://", "https://")):        sys.exit("$VMS_ADDRESS must be a bare hostname; vastpy adds https:// itself.")    kwargs = {"address": address}    if os.environ.get("VMS_TOKEN"):        kwargs["token"] = os.environ["VMS_TOKEN"]    else:        user, pwd = os.environ.get("VMS_USER"), os.environ.get("VMS_PASSWORD")        if not (user and pwd):            sys.exit("Set $VMS_USER and $VMS_PASSWORD (or $VMS_TOKEN).")        kwargs["user"], kwargs["password"] = user, pwd    if os.environ.get("VMS_TENANT_NAME"):        kwargs["tenant_name"] = os.environ["VMS_TENANT_NAME"]    return VASTClient(**kwargs) def build_filters(args):    filters = {}    if args.bucket:        filters["s3_bucket_name"] = [{"eq": args.bucket}]    if args.user:        filters["login_name"] = [{"eq": args.user}]    if args.op:        filters["rpc_type"] = [{"eq": args.op}]    if args.path:        # 'startswith' operator may differ by version; see module docstring.        filters["view_path"] = [{"startswith": args.path}]    return filters def normalize(resp):    """query_data returns {'prop_list': [...], 'results': [[...], ...]}. Yield dict rows."""    if not isinstance(resp, dict) or "results" not in resp:        sys.exit(f"unexpected response: {resp!r}")    cols = resp.get("prop_list") or DEFAULT_FIELDS    for row in resp["results"]:        if isinstance(row, (list, tuple)):            yield dict(zip(cols, row))        elif isinstance(row, dict):            yield row        else:            yield {"value": row} def main():    p = argparse.ArgumentParser(description="Query the VAST AuditDB via the VMS REST API.")    p.add_argument("--bucket", help="Filter by S3 bucket name (exact).")    p.add_argument("--user", help="Filter by login_name (exact).")    p.add_argument("--op", help="Filter by rpc_type (exact), e.g. PUT_OBJECT, WRITE.")    p.add_argument("--path", help="Filter by view_path prefix.")    p.add_argument("--top", type=int, default=20, help="Max rows to fetch (default 20).")    p.add_argument("--fields", help="Comma-separated columns to return (default: a useful set).")    args = p.parse_args()    client = build_client()    fields = [f.strip() for f in args.fields.split(",")] if args.fields else DEFAULT_FIELDS    try:        resp = client.vastauditlog.query_data.post(            limit=args.top, fields=fields, filters=build_filters(args),        )    except Exception as e:        hint = ("Check that $VMS_ADDRESS is the management host (not the S3 endpoint) "                "and that auditing is enabled. See ../03-troubleshooting.md.")        if "filters" in str(e).lower():            hint = ("The query_data endpoint requires a non-empty 'filters'. Pass a filter, "                    "e.g. --bucket mputest (or --user/--op/--path).")        sys.exit(f"query_data failed: {e}\n{hint}")    rows = list(normalize(resp))    if not rows:        print("No matching audit rows. Loosen filters, or run without --op to see real rpc_type values.")        return    print(f"{len(rows)} row(s):\n")    for r in rows:        print("  " + "  ".join(f"{k}={r.get(k)}" for k in fields if k in r)) if __name__ == "__main__":    main() ```