Security

Applies to VAST Cluster 5.5.

Vector search uses the same identities, keys and policies as the rest of VAST DataBase. This page covers what's specific to searching vectors through the VAST Query Engine. For the general model (who the cluster thinks you are, refused requests, row and column security, impersonation), see VAST DataBase security.

Actions

Vector DDL, inserts and searches use the same IAM-style JSON policies as S3. Through the Query Engine they need:

The policy can be an identity policy on the user or a bucket policy. Setup has a request you can send. For the actions each SDK call needs, see setup prerequisites.

Who sees which vectors

You can filter vector searches with ordinary WHERE clauses. You can also filter them by the user's permissions, so that each user's search returns only the rows they may see (see Vector Search). This is done with row and column security statements in the user's identity policy, which your platform team sets up. The order in which the cluster checks policies and then applies these rules is the same as for any VAST DataBase request; see the diagram in who the cluster thinks you are.

Permission filtering isn't a substitute for WHERE. Use it for access control, and WHERE for what the query is about.

A search filtered by permissions still returns LIMIT k rows when the user may read at least k rows: the nearest ones they're allowed to see. In a test where every one of the nearest 200 rows was hidden from the user, searches with LIMIT 10, 100 and 1000 returned 10, 100 and 1,000 permitted rows. Such a search reads further into the index, so it takes longer: about twice as long in that test.

Which rules work with vector search

On 5.5, what applies to vector search is:

• The row filter by S3 path applies to Query Engine searches and to SDK select(). See below.
• Column allow and deny rules are applied by the Query Engine. See column security.
• Row-filter predicates and column masks are supported for Trino only. If a table's policy has them for your user, the Query Engine refuses to query that table, rather than returning unfiltered rows. That includes DESCRIBE, and a RowFilter whose clause names only Trino ("QueryEngine": "Trino"). Use Trino for those tables; Trino doesn't use the vector index (limits).

The Query Engine's refusal (abridged):

failed with: Permission Error: [req=0x<id>]: query_engine is not supported for tables with row_filter=0xff

The SDK's own select() on such a table is refused with HTTP 403 and the message cluster version before 5.4.0 but row_filter=0xff column_mask=0x0, even on a 5.5 cluster.

Row filter by S3 path

A common case for retrieval-augmented generation (RAG) is "return a chunk only if the user may read the document it came from". The RowFilterByS3PathAuth statement does this:

{
  "Effect": "RowColumnSecurity",
  "Resource": "arn:aws:s3:::my-db-bucket/*",
  "RowFilterByS3PathAuth": "True"
}

With it, a row is returned only if the user may read (get-object) every S3 path listed in the row's vastdb_s3_path_auth column (KB). So the table needs that column, holding the path of each embedding's source object. See the vastdb_s3_path_auth column.

A RAG table therefore needs two decisions when you create it, because neither can be made later: the vector index, and the vastdb_s3_path_auth column (the column can't be added, changed or dropped after the table is created). A vector-indexed table can have the column.

How the filter behaves:

• The filter hides a chunk from users who may not read its source document, for Query Engine searches and SDK select() alike.
• A row listing several paths is returned only if the user may read all of them.
• Paths must be quoted ("bucket/object"); an unquoted path is rejected on insert with InvalidArgument.
• A GetObject deny without the row-filter statement doesn't hide rows.

The filter also applies to an impersonated end user. A service account that connects with vast.db.end_user=<user> gets only the rows that user may read; without the option, it gets the rows the service account may read. An end user the cluster doesn't know is refused with login_name=<user> was not found.

The RAG retrieval guide has the worked example with its output.

Column security

Column allow and deny rules are applied by the Query Engine. The Query Engine sees only the columns the user is allowed, so a denied column behaves as if it isn't in the table. Use this to keep the embedding column away from callers who should only see IDs and text: don't grant the embedding column to those users.

What a denied column does to searches:

• The embedding column denied: every search on the table fails, through SQL and through the SDK's vector_search(), even one that selects only the distance. DESCRIBE lists only the allowed columns, and the SDK's own select() of the allowed columns still works.
• A payload column denied: a search that selects that column fails; a search that doesn't select it works.

The error names the column as if it didn't exist (abridged):

failed with: Binder Error: Referenced column "e" not found in FROM clause!\nCandidate bindings: "id"

Keys and logs

• Keep the access and secret keys in environment variables or a secret store, and read them in your code. Never put them in source code.
• Never log the vast.db.secret_key option, or a dict that contains it.
• The ADBC driver writes its own log file. With adbc-driver-vastdb 0.0.32 it's ~/.local/share/VastAdbcDriver/vastdb_driver.log. Set VAST_ADBC_LOG_DIR to write it somewhere else.

See also

• VAST DataBase security: row and column security, masks, impersonation.
• Setup: what to ask your platform team for.
• RAG retrieval: where the path column fits in a RAG pipeline.