--- title: "Capacity Analysis with vastpy-cli" slug: "capacity-analysis-with-vastpy-cli" description: "Analyze VAST Storage Capacity & File Data Efficiently Using Version 5+, Direct Query + Catalog Insight with vastPy-CLI" updated: 2026-11-03T18:09:19Z published: 2026-11-03T18:09:19Z canonical: "docs.vastdata.com/capacity-analysis-with-vastpy-cli" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Capacity Analysis with vastpy-cli ## Summary This document provides a comprehensive guide on how to analyze storage capacity and file data on a VAST Data platform using `vastpy-cli`. We cover both the direct capacity queries and the more advanced catalog-based analysis. ## Methods ### Via VAST REST API We will cover two methods of obtaining this data. The first, using the VAST Capacity Estimations data, leverages the VAST REST API (via the `vastpy-cli` utility) to query the capacity statistics for any directory within the VAST Platform. This API is available in all versions of VAST from 4.0 onward and requires no configuration. For more information, see [Capacity Overview](/docs-5-5/docs/capacity-overview). This approach is useful if: 1. The VAST Catalog is not enabled on your cluster 2. You want insight into the DataReduction ratio across different folders on the system. ### Via VAST Catalog The Second approach is to use `vastpy-cli` to query the VAST Catalog, a feature that has been available since the 4.6 release. This does require that the VAST Catalog is enabled; see [VAST Catalog Overview](/docs-5-5/docs/vast-catalog-overview) for more details on the feature, as well as how to configure. This approach is preferred if: 1. The VAST Catalog is enabled 2. You want more granular insight and filtering capabilities. Examples include: - Determine capacity used per user/group - Determine capacity per file extension - Find individual file locations based on a filename search - Much, much more. Regardless of which of these approaches you choose, you will be using the same basic tool (`vastpy-cli`). There are other methods of accessing this data, which will be covered in another document. ## Prerequisites A host with: - Python installed - Network access to the VMS IP address, specifically on port 443 (HTTPS) - VMS Manager account with appropriate permissions ## Setting Up `vastpy-cli` ### Installing `vastpy` Install the `vastpy` package, which includes the `vastpy-cli` command: ```bash pip install --upgrade vastpy ``` You may also want to install the `jq` utility: ```bash sudo apt-get install -y jq #ubuntu/debian variants sudo yum install -y jq #rocky/rhel variants ``` ### Setting Up Authentication Set up environment variables for authentication (recommended method): ```bash # Set these environment variables once per session export VMS_USER=admin export VMS_PASSWORD=your_password  # Add a space before this command to avoid storing the password in command history export VMS_ADDRESS=vast-vms-address ``` Alternatively, you can specify credentials with each command: ```bash vastpy-cli --user=admin --password=your_password --address=vast-vms-address [command] ``` ## 1. Analyzing Directory Capacity with `vastpy-cli` The `capacity` subcommand in `vastpy-cli`allows you to retrieve detailed capacity information for specific directories in your VAST Data platform. Generally, you filter by specifying the path you want to analyze. It can be a top-level directory or any subdirectory within the system. Note that the data underpinning this API is an estimate based on sampling and should not be used for exact calculations. Also, there may be a delay between when data is ingested into a folder and when it appears in the statistics. This is because sampling can occur as the data is migrated from SCM -> QLC flash, which occurs asynchronously. Lastly, folders that do not have enough aggregate capacity will not appear on their own in these metrics. If you are unsure of the information you are seeing, contact CS with specific commands and output so they can best assist. ### Basic Usage To get capacity information for a specific directory: ```bash # Basic capacity command vastpy-cli get capacity path=/path/to/directory ``` ### Retrieving JSON Output For more structured output and easier parsing, you can request JSON format: ```bash # Get capacity information in JSON format vastpy-cli get --json capacity path=/path/to/directory ``` ### Example with jq Parsing You can pipe the JSON output to tools like `jq` for further filtering and analysis: ```bash # Get available capacity keys vastpy-cli get --json capacity path=/maria | jq '.keys' ``` Output: ```json [  "usable",  "unique",  "logical" ] ``` To get detailed information about the directory: ```bash # Get detailed capacity information for the directory vastpy-cli get --json capacity path=/maria | jq '.details[0]' ``` Output: ```json [  "/maria",  {    "data": [      7546623507,      5237582703,      25589375620    ],    "parent": "/",    "percent": 100,    "average_atime": "2025-03-29 07:30"  } ] ``` (The output above is in `bytes`) ### Understanding Capacity Output The capacity output contains three key metrics: 1. **Usable** - The usable space in bytes on the physical media 2. **Unique** - The unique space in bytes (after deduplication) 3. **Logical** - The logical space in bytes (what users/applications see) The `average_atime` field shows the average last access time for files in the directory, which can be helpful for understanding data usage patterns. ## 2. Analyzing Data with VAST Catalog The VAST Catalog provides a more powerful way to query and analyze your data, especially for large datasets. You can use the `bigcatalogconfig` subcommand in `vastpy-cli`. Some notes: 1. Depending on the filter and the number of files on the system, these queries can take some time to complete. Most should complete within seconds, but some may take longer 2. There are no built-in 'aggregation' functions in this API at the moment. If you wish to determine the capacity of a folder, for example, you have the following options: - Pipe through `jq` (described in some of the examples below) - Use another method (such as the `vastdb-sdk`), as described in the document here: [VAST DB Python SDK](https://vastdb-sdk.readthedocs.io/) . - Use an application such as Trino, as described here: [Trino Quickstart](https://vast-data.github.io/data-platform-field-docs/vast_database/trino/quickstart.html) 3. `limit` is a **preview control, not a pagination control**. Use `limit=50` when first experimenting with a query so it doesn't overrun your terminal. Be aware of two consequences: any aggregation you run with a limit in place will be computed over the truncated rows and will therefore be wrong, and setting `limit` forces the response's `next` field to `null`, which disables paging entirely. To retrieve a complete result set, drop `limit` and page through the results as described in [Retrieving Complete Result Sets](/kb/docs/capacity-analysis-with-vastpy-cli#3-retrieving-complete-result-sets). 4. Always pass `--json`. The default table output prints rows only and **discards the** `count` **and** `next` **fields**, which are the fields that tell you whether more results exist. ### Using vastpy-cli for Catalog Queries First, an example to list all files that were not accessed in the last 90 days: ```bash # Basic catalog query for files not accessed in the last 90 days vastpy-cli post --json bigcatalogconfig/query_data \  limit=50 \  path="/vperfsanity" \  fields='["parent_path", "name", "atime", "size"]' \  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq ``` Output (truncated): ```json {  "count": 256,  "next": null,  "prop_list": [    "parent_path",    "name",    "atime",    "size"  ],  "results": [    [      "/vperfsanity/",      "vperfsanity21",      "2025-01-14T20:14:49.499298",      4194304000    ],    [      "/vperfsanity/",      "vperfsanity155",      "2025-01-14T20:14:47.415081",      4194304000    ]  ] } ``` Same query, but this time only retrieve the `size` field and perform a sum (notice that we removed the `limit` from this example to get an accurate sum. ```bash vastpy-cli post --json bigcatalogconfig/query_data \  path="/vperfsanity" \  fields='["size"]' \  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq '([.results[][]] | add)' ``` Output: ```bash 1073741824000 ``` Obviously, you may want to convert from bytes into something more useful, like GiB or TiB: ```bash vastpy-cli post --json bigcatalogconfig/query_data \  path="/vperfsanity" \  fields='["size"]' \  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq '([.results[][]] | add) / (1024*1024*1024)' ``` ## 3. Retrieving Complete Result Sets Every example above returns a single response. When a query matches more rows than the server returns at once, the response carries a `next` field holding a URL ending in `?page=N`. Paging means re-issuing the same query with an added `page` field until `next` comes back `null`. `vastpy-cli` is a thin wrapper over the REST API, so the paging rules are the API's rules. Two CLI behaviors shape how you apply them: - **POST puts every argument in the JSON body; GET puts them on the query string.** There is no way to mix a query-string `page` with body filters in one invocation. - `--file-input` **replaces** `key=value` **arguments; the two cannot be combined.** If you supply a JSON file, any `key=value` arguments on the same command line are ignored silently. For page 2 and beyond, put `page` inside the JSON file. ### The two rules that matter 1. **Never send** `limit` **together with** `page`**.** The combination returns a `400` with an empty `detail` body. 2. **Never send** `limit` **at all when you intend to page.** `limit` on its own forces `next` to `null`, so you cannot tell whether more rows exist. ### Recommended: POST with `--file-input` Request the first page with **no** `limit` **and no** `page`: ```bash export VMS_USER=admin export VMS_PASSWORD='***' export VMS_ADDRESS=vast-vms-address export VMS_API_VERSION=v5   # or latest cat > catalog_query.json << 'EOF' {  "path": "/",  "snapshot": "bc_table_2026-09-08_10_00_00",  "fields": ["parent_path", "name", "size", "mtime"],  "filters": {"size": [{"ge": 1048576}]} } EOF vastpy-cli post --json bigcatalogconfig/query_data --file-input catalog_query.json ``` Inspect `next` in the response. If it is a URL containing `?page=2`, wait for the call to finish, then add `"page": 2` to the **same file**, leaving everything else unchanged: ```json {  "path": "/",  "snapshot": "bc_table_2026-09-08_10_00_00",  "fields": ["parent_path", "name", "size", "mtime"],  "filters": {"size": [{"ge": 1048576}]},  "page": 2 } ``` ```bash vastpy-cli post --json bigcatalogconfig/query_data --file-input catalog_query.json ``` Repeat with `"page": 3`, `"page": 4`, and so on until `next` returns `null`. Pinning `snapshot` to a specific catalog table ensures a consistent multi-page walk for the lifetime of the query. The query will work without it, but it is not recommended when point-in-time accuracy matters. ### Paging constraints The server enforces the following. Violating any of them produces skipped or duplicated rows rather than an error: | Constraint | Detail | | --- | --- | | Consecutive pages | Request pages in order: 1, then 2, then 3. Do not skip ahead. | | Identical query | `path`, `snapshot`, `filters`, and `fields` must not change between pages. | | One at a time | Wait for page N to return before requesting page N+1. Firing them concurrently silently duplicates or skips rows. | | 300-second window | Issue the next page within 300 seconds of the previous one, or the server-side cursor expires. | ### Looping through every page ```bash page=1 cp catalog_query.json /tmp/q.json   # no page, no limit while true; do  if [ "$page" -gt 1 ]; then    jq --argjson p "$page" '.page = $p' catalog_query.json > /tmp/q.json  fi  vastpy-cli post --json bigcatalogconfig/query_data --file-input /tmp/q.json \    > /tmp/page.json  count=$(jq '.count' /tmp/page.json)  next=$(jq -r '.next // empty' /tmp/page.json)  echo "page $page: $count rows  next=${next:-null}"  [ -z "$next" ] && break  page=$((page + 1)) done ``` ### GET form GET sends parameters as a query string, which VMS also accepts. Keep `limit` out and keep `--json`. Note that `filters` supplied as CLI JSON is parsed by vastpy into a dict and can serialize badly, so prefer the POST `--file-input` form whenever filters are nested: ```bash vastpy-cli get --json bigcatalogconfig/query_data \  path=/ \  fields=parent_path,name,size,mtime vastpy-cli get --json bigcatalogconfig/query_data \  path=/ \  fields=parent_path,name,size,mtime \  page=2 ``` ### Python SDK For anything scripted, the SDK is cleaner than the CLI. POST keyword arguments become the request body, so `page` is simply another field: ```python from vastpy import VASTClient client = VASTClient(address='vast-vms-address', user='admin', password='***', version='v5') body = {    'path': '/',    'snapshot': 'bc_table_2026-09-08_10_00_00',    'fields': ['parent_path', 'name', 'size', 'mtime'],    'filters': {'size': [{'ge': 1048576}]}, } page = 1 while True:    kwargs = dict(body)    if page > 1:        kwargs['page'] = page    resp = client.bigcatalogconfig.query_data.post(**kwargs)    print('page', page, 'count', resp['count'], 'next', resp.get('next'))    if not resp.get('next'):        break    page += 1 ``` ## Advanced Query Examples The examples below use `limit=50` to keep the output readable. Remove `limit` and page through the results, as described above, whenever you need the complete set. ### Finding Large Files ```bash # Find files larger than 1GB vastpy-cli post --json bigcatalogconfig/query_data \  limit=50 \  path="/vperfsanity" \  fields='["parent_path", "name", "size"]' \  filters='{"size": [{"gt": 1073741824}]}' ``` ### Finding Recently Modified Files (Note the use of `gt` as opposed to `lt`) ```bash # Find files modified in the last day vastpy-cli post --json bigcatalogconfig/query_data \  limit=50 \  path="/" \  fields='["parent_path", "name", "mtime"]' \  filters='{"mtime": [{"gt": "'$(date -d '1 day ago' +%s)'000"}]}' | jq ``` ### Finding Files by Owner ```bash # Find files owned by a specific user vastpy-cli post --json bigcatalogconfig/query_data \  limit=50 \  path="/" \  fields='["parent_path", "name", "login_name"]' \  filters='{"login_name": "maria.gutierrez@selab.vastdata.com"}' ``` ### Find Files by filename ```bash vastpy-cli post --json bigcatalogconfig/query_data \  limit=50 \  path="/" \  fields='["parent_path", "name", "login_name"]' \  filters='{"name": "comment.mp4"}' ``` Output ```json {  "count": 2,  "next": null,  "prop_list": [    "parent_path",    "name",    "login_name"  ],  "results": [    [      "/scratch/home/rayc-flow/MusM/",      "comment.mp4",      null    ],    [      "/scratch/home/jhays-flow/feasibleness/slim/",      "comment.mp4",      null    ]  ] } ``` ## Tips for Large-Scale Analysis When working with large datasets: 1. **Filter Early** - Apply specific filters to reduce the dataset size 2. **Page, don't limit** - `limit` truncates the result and suppresses `next`. For a complete result set, drop `limit` and walk the pages consecutively, pinning `snapshot` so the view stays consistent. See [Retrieving Complete Result Sets](/kb/docs/capacity-analysis-with-vastpy-cli#3-retrieving-complete-result-sets). 3. **Aggregate over every page** - Any `jq` sum or count must accumulate across all pages. Aggregating a single page silently under-reports. 4. **Schedule Regular Analysis** - Set up regular analysis jobs to track capacity trends over time ## Conclusion The VAST capacity and catalog analysis tools provide powerful ways to understand your data usage patterns, identify optimization opportunities, and manage your storage more effectively. By combining direct capacity queries with catalog analysis, you can gain comprehensive insights into your VAST Data platform. For more detailed information about the VAST API, please refer to your VAST system's documentation at `https://<your-vast-vms-address>/docs`.