Tags and metadata

Applies to VAST Cluster 5.5.

If you classify data with S3 object tags or user metadata (x-amz-meta-* headers), the Catalog lets you find objects by them across a whole namespace: for example, every object whose processed tag is false. This guide shows the two ways: the user_tags and user_metadata map columns, which every Catalog has, and user-defined columns, which your platform team adds for the keys you use most.

Each code block is complete on its own. They read your connection settings from the environment, as in the quickstart, and use CATALOG_PATH as the subtree to search.

Where tags and metadata are in the Catalog

You set tags and metadata with any S3 client: tags with PutObjectTagging (or on upload), metadata as x-amz-meta-<key> headers on PutObject. They show up in the Catalog after its next snapshot.

Find objects by a tag

The Python SDK can't push down a filter on one key inside a map, so do it in two steps: filter on the cluster to objects that have tags at all (user_tags_count > 0), then look up the key in your client.

This example needs objects under your path tagged processed=false:

import os

import pyarrow.compute as pc
import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["parent_path", "name", "user_tags"],
        predicate=(_.search_path == path) & (_.element_type == "FILE") & (_.user_tags_count > 0),
    ).read_all()

processed = pc.map_lookup(rows["user_tags"], "processed", "first")
todo = rows.filter(pc.equal(processed, "false"))

print(rows.num_rows, "tagged objects,", todo.num_rows, "with processed=false")
for row in todo.slice(0, 10).to_pylist():
    print(row["parent_path"] + row["name"], dict(row["user_tags"]))

9 tagged objects, 4 with processed=false
/cat-data/projects/apollo/part-008.parquet {'processed': 'false', 'project': 'apollo'}
/cat-data/projects/apollo/part-007.parquet {'processed': 'false', 'project': 'apollo'}
/cat-data/projects/apollo/part-006.parquet {'processed': 'false', 'project': 'apollo'}
/cat-data/projects/apollo/model.bin {'processed': 'false', 'project': 'apollo'}

map_lookup returns null for objects that don't have the key, and the comparison leaves them out.

Find objects by user metadata

User metadata works the same way, with user_metadata. The map has no count column, and a filter on a map column can't run on the cluster (isnull() on it raises NotImplementedError), so fetch the column and drop the objects without metadata in your client. On a large namespace, narrow the query with search_path, or ask for the key to be indexed (next section). This example needs an object under your path uploaded with the header x-amz-meta-project: apollo:

import os

import pyarrow.compute as pc
import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["parent_path", "name", "user_metadata"],
        predicate=(_.search_path == path) & (_.element_type == "FILE"),
    ).read_all()

# A filter on a map column can't run on the cluster, so drop the objects without metadata here.
rows = rows.filter(pc.is_valid(rows["user_metadata"]))
for row in rows.slice(0, 10).to_pylist():
    print(row["parent_path"] + row["name"], dict(row["user_metadata"]))

project = pc.map_lookup(rows["user_metadata"], "project", "first")
print(rows.filter(pc.equal(project, "apollo")).num_rows, "objects with project=apollo")

/cat-data/reports/q3-summary.pdf {'department': 'finance', 'project': 'apollo'}
/cat-data/projects/gemini/sales-4.csv {'project': 'gemini'}
/cat-data/projects/gemini/sales-3.csv {'project': 'gemini'}
/cat-data/projects/gemini/sales-1.csv {'project': 'gemini'}
/cat-data/projects/gemini/sales-2.csv {'project': 'gemini'}
1 objects with project=apollo

The first lines print each object's metadata as it's stored, so you can see the exact keys. The keys are stored without the x-amz-meta- prefix.

Filter on a user-defined column

When you filter on the same key often, or on a large namespace, ask your platform team to index it as a user-defined attribute. Admin task: send to your platform team: Configuring VAST Catalog, "User defined attributes". Tell them the type (tag, metadata or NFSv4 xattr) and the key.

From the next snapshot on, the Catalog has a column for that key, such as tag_processed, and a filter on it pushes down like any other column. The tag is still in user_tags too.

This example needs a user-defined tag attribute processed:

import os

import vastdb
from ibis import _

path = os.environ.get("CATALOG_PATH", "/")

session = vastdb.connect()
with session.transaction() as tx:
    rows = tx.catalog().select(
        columns=["parent_path", "name", "tag_processed"],
        predicate=(_.search_path == path) & (_.tag_processed == "false"),
    ).read_all()

print(rows.num_rows, "objects with processed=false")

4 objects with processed=false

To see which user-defined columns your Catalog has, list its columns and look for the tag_, metadata_ and nfs4_xattr_ prefixes:

import vastdb

session = vastdb.connect()
with session.transaction() as tx:
    fields = tx.catalog().columns()

for field in fields:
    if field.name.startswith(("tag_", "metadata_", "nfs4_xattr_")):
        print(field.name, field.type)

metadata_project string
tag_processed string

In SQL

With Trino, filter on a map key with element_at, for example WHERE element_at(user_tags, 'processed') = 'false'. Trino evaluates that itself; whether the connector pushes it down to the cluster isn't documented, so keep a search_path filter in the query. See Trino and Spark.

See also

• Concepts: indexed and user-defined columns.
• Catalog schema.
• Find files.