Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Troubleshooting

Prev Next

Applies to VAST Cluster 5.5.

Find what you're seeing in the tables below: symptom, likely cause, first fix. If you have an error message, search the error index for it first.

Connecting and permissions

Symptom Likely cause First fix
ADBC connect or the first SQL statement fails with SignatureDoesNotMatch Wrong secret key Check the key pair (errors).
ADBC connect or SQL fails with an authorization error The endpoint isn't a Query Engine endpoint, or the policy lacks s3:QueryEngine* or s3:Tabular* Use the Query Engine pool's address, and ask for both groups of actions (setup).
vector_search() raises NoAdbcConnectionError The ADBC driver isn't installed pip install 'vastdb[adbc]' (errors).
ADBC connect fails with Connection refused (os error 111) The endpoint URL has the wrong port, or none Add the cluster's S3 port, for example :9090 (setup).
CREATE SCHEMA and CREATE TABLE work, but every query, even SELECT 1, fails with status: 17 The cluster has no virtual IP pool with the Query Engine role Ask your platform team for one (errors).

Creating a table

Symptom Likely cause First fix
Binder Error: ... must be a vector column You clustered by a VARCHAR, INTEGER or FLOAT[] column Use VECTOR(FLOAT, <d>) (errors).
internal error: Invalid Argument when you create the table An illegal quantization pair (scalar_8bit cluster with none search), an unsupported value such as rabitq_4bit, or an unknown metric Use none/none, none/scalar_8bit or scalar_8bit/scalar_8bit (quantization).
You need a vector index on a table that already has data An index can only be defined when the table is created Create a new table with the index and copy the data into it (move vectors into an indexed table).

Searching

Symptom Likely cause First fix
The search runs, but slowly, as if it ignores the index The query doesn't have the shape the index needs: it uses OFFSET, orders descending or by more than one key, has no LIMIT, or uses a distance function that doesn't match the table's metric. Such queries run as full scans. Run EXPLAIN on the query: a search that uses the index has a VAST_VEC_INDEX_SCAN step. If it doesn't, order by the one distance, ascending, with LIMIT k and no OFFSET, and use the function for the table's metric (query design).
Recall is low, latency is fine Wrong embedding model, dimension or normalization; random test data; recent rows not clustered yet; probe settings too low Check the embedding contract (ingest). Wait for clustering grade A or A+ (check the index first). Then raise the probe settings (configuration).
Latency is high, and another system answers faster Selecting wide columns, no warm-up, different probe settings, or an unclustered table Compare the same SQL. Time only execute and fetch. Return only the columns you need.
Latency is high right after a bulk load Recent rows are searched by brute force, and background clustering uses CNode CPU Wait for clustering to finish (performance).
Throughput (queries per second) is low One ADBC connection, or all connections on one address Open several connections and spread them over the Query Engine pool's addresses (performance).
Python SDK vector_search() on an ip (inner product) table returns the least similar rows first A known SDK 2.1.0 problem with ip tables Search ip tables with SQL (Python SDK).
Search through Trino or Spark is slow The Trino and Spark connectors don't use the vector index (limits) Search through the Query Engine with ADBC or the Python SDK.
A search copied from a webinar or slide fails with Scalar Function with name cosine_similarity does not exist!, or, rewritten with array_cosine_similarity and ORDER BY ... DESC, runs slowly There's no cosine_similarity function. The rewrite runs, but orders descending by a function that isn't the table's distance function, so it scans every row. Use array_cosine_distance(...) with ORDER BY ... ASC LIMIT k (query design).

Changing data

Symptom Likely cause First fix
You need to change an embedding Updating the vector column isn't supported: it fails with Invalid Argument DELETE the row and INSERT it again with the new vector.
ALTER TABLE fails with Invalid Argument Adding or dropping columns on an indexed table isn't supported in 5.5 Plan the columns at creation (design a vector table), or copy into a new table (move vectors into an indexed table).

Before you ask for help

Collect:

  • Your cluster version, and your Python SDK and ADBC driver versions; see find your versions.
  • The query ID of the slow or failing search (the vast.db.last_query_id statement option; see ADBC).
  • The table's metric, quantization and clustering grade (check the index first).
  • The exact SQL, and the full error message if there is one.

Remove addresses and keys before you post logs anywhere.

See also

Related articles