Monitoring your workload

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

This page shows what you can see about your own queries and loads from the client: SDK logging, table statistics, sort progress, the partitions view and the imports table. Cluster-wide dashboards belong to your platform team. The last section says what they can see for you.

SDK logging

The SDK uses Python's standard logging module. Every module logs under its own name, so vastdb is the parent logger for all of them:

To see what your code is doing, set the vastdb logger to INFO:

import logging

logging.basicConfig(format="%(asctime)s %(name)s %(levelname)s %(message)s")
logging.getLogger("vastdb").setLevel(logging.INFO)

# More detail for queries and inserts only:
logging.getLogger("vastdb.table").setLevel(logging.DEBUG)

Be careful with DEBUG on vastdb._internal and vastdb.errors. Those messages include request URLs, headers and response bodies, which name your bucket, schemas and tables, and the cluster's addresses. They're useful for a support case, but don't leave them on in production logs.

Retries are logged at the level set in BackoffConfig.backoff_log_level, which is DEBUG by default. See Retries.

Set up

The examples below use the environment variables from the quickstart and a schema called ops_monitoring. Run the blocks in order in one Python session.

import os
import time
import pyarrow as pa
import vastdb

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]

columns = pa.schema([
    ("id", pa.int64()),
    ("region", pa.utf8()),
    ("value", pa.float64()),
])
rows = pa.table({
    "id": list(range(1000)),
    "region": ["eu", "us"] * 500,
    "value": [float(i) for i in range(1000)],
}, schema=columns)

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).create_schema("ops_monitoring", fail_if_exists=False)
    table = schema.create_table("events", columns, fail_if_exists=False)
    row_ids = table.insert(rows)
    print(f"Inserted {len(row_ids)} rows")

Inserted 1000 rows

Table statistics

table.get_stats() returns a TableStats object with the table's row count and size, as reported by the cluster. Each call fetches new numbers. It needs the s3:TabularGetTableStats permission (see setup prerequisites).

The statistics are kept up to date by background work, not as part of your transaction. A reading taken straight after a load can be behind, and for a while it can also include rows from transactions that haven't committed, or count deleted rows that haven't been cleaned up yet. On sorted and partitioned tables the totals can lag for minutes (write path). To check a load exactly, count the rows with a query. This example polls until the count matches:

def wait_for_row_count(table_name, expected, timeout=60):
    deadline = time.time() + timeout
    while True:
        with session.transaction() as tx:
            stats = tx.bucket(BUCKET).schema("ops_monitoring").table(table_name).get_stats()
        if stats.num_rows == expected:
            return stats
        if time.time() > deadline:
            raise TimeoutError(f"{table_name}: stats show {stats.num_rows} rows, expected {expected}")
        time.sleep(1)

stats = wait_for_row_count("events", 1000)
print("num_rows:", stats.num_rows)
print("size_in_bytes > 0:", stats.size_in_bytes > 0)
print("sorting_key_enabled:", stats.sorting_key_enabled)
print("partitioning_info:", stats.partitioning_info)

num_rows: 1000
size_in_bytes > 0: True
sorting_key_enabled: False
partitioning_info: None

Use the statistics for trends and rough checks, such as "did the nightly load land?". For an exact count at a point in time, query the table.

TableStats also has vips and endpoints: the virtual IP addresses the SDK uses to spread queries and imports across the cluster (see performance). Don't print them into shared logs. All the fields are listed in the SDK reference.

Sort progress

On a sorted table, the cluster sorts new rows in the background after they're committed. Two methods report its progress:

• sorting_done() returns True when the background sorter has nothing left to do. That doesn't mean every row is sorted: the newest block of rows is sorted only once it's full (512K rows), so a small sorted table reports done even though none of its rows are sorted (write path). It's always False for unsorted tables.
• sorting_score() returns a progress score. It's always 0 for unsorted tables.

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("ops_monitoring")
    sorted_table = schema.create_table("events_sorted", columns,
                                       fail_if_exists=False, sorting_key=["id"])
    sorted_table.insert(rows)

deadline = time.time() + 120
while True:
    with session.transaction() as tx:
        sorted_table = tx.bucket(BUCKET).schema("ops_monitoring").table("events_sorted")
        is_sorted = sorted_table.sorted_table
        done = sorted_table.sorting_done()
    if done or time.time() > deadline:
        break
    time.sleep(2)

print("sorted_table:", is_sorted)
print("sorting_done:", done)

with session.transaction() as tx:
    events = tx.bucket(BUCKET).schema("ops_monitoring").table("events")
    print("unsorted table, sorting_done:", events.sorting_done())

sorted_table: True
sorting_done: True
unsorted table, sorting_done: False

Sorting works best on tables with more than 1 million rows, because tables are sorted in chunks of 512K rows. While sorting runs, the table's row count and size can go up for a while and then come back down, but count(*) stays correct (see Sorted tables). So expect get_stats() to move around on a sorted table that's still being sorted.

The partitions view

On a partitioned table, table.partitions() returns a read-only view with one row per partition. Each row holds the partition values and two estimates, est_row_count and est_byte_size. Like the table statistics, the estimates can lag behind recent writes.

from pyiceberg.transforms import IdentityTransform
from vastdb import PartitionKey, PartitionSpec

spec = PartitionSpec([PartitionKey("region", IdentityTransform())])

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("ops_monitoring")
    by_region = schema.create_table("events_by_region", columns,
                                    fail_if_exists=False, partition_spec=spec)
    by_region.insert(rows)

with session.transaction() as tx:
    by_region = tx.bucket(BUCKET).schema("ops_monitoring").table("events_by_region")
    parts = by_region.partitions().select().read_all()

print(sorted(parts.schema.names))
print(sorted(parts["region"].to_pylist()))

['est_byte_size', 'est_row_count', 'region']
['eu', 'us']

Use the view to check that rows land in the partitions you expect, and to spot skew, such as one partition much larger than the rest. Only partitioned tables have it. On any other table, partitions() raises AssertionError.

The imports table

When you load Parquet files with import_files(), the table keeps a record of the files it imported. The record is a second table, the imports table, which you query with imports_table(). It needs cluster 5.2 or later.

On a 5.5 cluster, a new table already has an imports table, so you don't need to create one. Calling create_imports_table() on it raises InternalServerError with failed creating import table (res=EXIST). In SDK 2.1.0 this happens even if you pass fail_if_exists=False, because the method ignores that argument.

The example uploads two Parquet files to the same bucket with boto3, under a prefix that isn't a schema name, and then imports them:

import io
import boto3
import pyarrow.parquet as pq

s3 = boto3.client("s3", endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"])
buf = io.BytesIO()
pq.write_table(rows, buf)
keys = ["ops-monitoring-files/part-0.parquet", "ops-monitoring-files/part-1.parquet"]
for key in keys:
    s3.put_object(Bucket=BUCKET, Key=key, Body=buf.getvalue())

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("ops_monitoring")
    schema.create_table("events_imported", columns, fail_if_exists=False)

with session.transaction() as tx:
    imported = tx.bucket(BUCKET).schema("ops_monitoring").table("events_imported")
    imported.import_files([f"/{BUCKET}/{key}" for key in keys])

with session.transaction() as tx:
    imported = tx.bucket(BUCKET).schema("ops_monitoring").table("events_imported")
    log = imported.imports_table().select(columns=["BucketName", "ObjectName"]).read_all()

print(log.schema.names)
print(sorted(log["ObjectName"].to_pylist()))

['BucketName', 'ObjectName']
['ops-monitoring-files/part-0.parquet', 'ops-monitoring-files/part-1.parquet']

The imports table also records each object's creation and modification times and size, and the start and end of each import (ImportStartTime and ImportEndTime, plus ...UTC string versions). Before you import a batch of files, you can check it against ObjectName to skip files you've already loaded. See ingest.

To follow an import while it runs, set vastdb.table and vastdb._internal to INFO. The SDK logs each batch it starts, the endpoint it uses, and each file as it finishes.

Clean up

with session.transaction() as tx:
    schema = tx.bucket(BUCKET).schema("ops_monitoring")
    for t in schema.tables():
        t.drop()
    schema.drop()

for key in keys:
    s3.delete_object(Bucket=BUCKET, Key=key)
print("Dropped ops_monitoring and its files")

Dropped ops_monitoring and its files

What your platform team can see

The cluster keeps database metrics that the SDK can't read. According to Monitoring VAST DataBases, the VAST Web UI shows:

• On the dashboard: database bandwidth, the number of running queries, and rows scanned per second.
• In predefined analytics reports: query latency and the number of running queries, bytes returned or read per second, rows scanned, imported, inserted, updated, deleted or filtered out per second, and requests per type (query, import, insert, update, delete).
• Custom analytics reports built from database metrics, and capacity analytics for each database view.

If a query or load is slow and nothing on the client side looks wrong, ask for these numbers for the time it ran.

Admin task: send to your platform team. See Monitoring VAST DataBases.

See also

• Performance: what you can change when the numbers look wrong.
• Errors: what the SDK's exceptions mean.
• Python SDK reference: TableStats, sorting, partitions and the imports table.