---
title: "Monitoring"
slug: "dev-event-broker-operations-monitoring"
description: "Learn to monitor Kafka consumer lag and manage topics effectively with VAST Analytics, ensuring optimal performance during load tests."
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-event-broker-operations-monitoring"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

*Applies to VAST Cluster 5.5.*

This page covers what to watch while you produce and consume: the Kafka reports your platform team can open in VAST Analytics, and the consumer lag you can check yourself from a Kafka client.

## On the cluster

Your platform team can open these in the VAST Web UI. Ask them to share the numbers during load tests.

| What | Where | What it tells you |
| --- | --- | --- |
| Kafka produce and fetch latency and bandwidth | Analytics → Predefined Analytics, the reports with **Kafka** in the name, filtered by view or by CNode | Whether producing and fetching are slow, and whether load spreads over the CNodes. If one CNode carries most of it, check that clients use all the virtual IPs (see [architecture](/develop/docs/dev-event-broker-concepts-architecture#every-vip-is-a-broker)). |
| Events and capacity per topic | The topic's details in the VAST Web UI, CLI `topic show`, or REST `GET /topics/show/` | How many events a topic holds and how much space it takes, so you can check retention is keeping it in bounds. |
| Blob expansion statistics | Analytics, as custom analytics | How many events were parsed into the target table, and how many failed. See [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion). |
| Publishing to third-party brokers | Analytics, the published-events reports; and the Prometheus endpoint for external brokers | Only for [S3 bucket events](/develop/docs/dev-event-broker-guides-s3-bucket-events) sent to an external Kafka cluster. |

On the cluster dashboard, the **Latency** popup for **Kafka** also shows an **Avg** field, which is always 0 in 5.5. Ignore it.

The [Prometheus Kafka endpoint](https://kb.vastdata.com/api-5-5/apidocs/return-prometheus-kafka-broker-metrics) covers events the cluster publishes to external brokers. Measure consumer lag on VAST Event Broker topics from your clients, as below.

Producing, consuming and topic management aren't covered by VAST protocol auditing or Quality of Service limits, so neither shows Kafka traffic.

## From your client

### Consumer lag

Lag is how far a consumer group is behind: for each partition, the high watermark minus the group's committed offset. On VAST Event Broker the high watermark is the offset after the last committed event.

Offsets can have gaps (a rolled-back produce request uses up offsets, and compaction removes events), so treat lag as an upper bound on the number of events still to read, not an exact count. The other way round, lag can read low for a short while: the end offset that the broker reports (through `get_watermark_offsets()`, or a `ListOffsets` request for the latest offset) can trail the newest events by tens of seconds: it can catch up 10 to 55 seconds after a burst of writes, while a consumer can already read the new events. Watch lag over minutes, not seconds.

#### Set up

Set `KAFKA_BOOTSTRAP` (and, for SASL/PLAIN, `KAFKA_USERNAME` and `KAFKA_PASSWORD`; see [setup](/develop/docs/dev-event-broker-operations-setup#check-that-it-works)). This creates a one-partition topic, produces 10 events, and has a consumer group read and commit 4 of them:

```
import os
import time
from confluent_kafka import Consumer, Producer, TopicPartition
from confluent_kafka.admin import AdminClient, NewTopic

TOPIC = "docs-monitoring-lag"
GROUP = "docs-monitoring-lag-group"

conf = {"bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"]}
if os.environ.get("KAFKA_USERNAME"):
    conf.update({
        "security.protocol": "SASL_PLAINTEXT",
        "sasl.mechanism": "PLAIN",
        "sasl.username": os.environ["KAFKA_USERNAME"],
        "sasl.password": os.environ["KAFKA_PASSWORD"],
    })

admin = AdminClient(conf)
admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=1)])[TOPIC].result()

producer = Producer({**conf, "compression.type": "none", "enable.idempotence": False})
for i in range(10):
    producer.produce(TOPIC, key=f"key-{i}".encode(), value=f"event {i}".encode())
producer.flush()

consumer = Consumer({**conf, "group.id": GROUP, "auto.offset.reset": "earliest",
                     "enable.auto.commit": False})
consumer.subscribe([TOPIC])
read = 0
deadline = time.time() + 60
while read < 4 and time.time() < deadline:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    read += 1
    if read == 4:
        consumer.commit(message=msg, asynchronous=False)
print("read and committed:", read)
```

```
read and committed: 4
```

#### Check the lag

`committed()` returns the group's committed offset per partition, and `get_watermark_offsets()` the low and high watermarks:

```
partitions = [TopicPartition(TOPIC, p) for p in admin.list_topics(TOPIC).topics[TOPIC].partitions]
committed = consumer.committed(partitions, timeout=10)

total = 0
for tp in committed:
    low, high = consumer.get_watermark_offsets(tp, timeout=10)
    position = tp.offset if tp.offset >= 0 else low   # no commit yet: lag from the start
    total += high - position
print("lag:", total)
```

```
lag: 6
```

Any consumer can run this check for any group: create a consumer with that `group.id`, but don't subscribe, so it doesn't join the group. With authorization on, it needs `KafkaRead` on the group and `KafkaDescribe` on the topic (see [security](/develop/docs/dev-event-broker-operations-security#actions)).

#### Clean up

```
consumer.close()
admin.delete_consumer_groups([GROUP])[GROUP].result()
admin.delete_topics([TOPIC])[TOPIC].result()
print("deleted", TOPIC, "and", GROUP)
```

```
deleted docs-monitoring-lag and docs-monitoring-lag-group
```

A group can only be deleted once it has no members, so close the consumer first.

### Fetch rate when caught up

When a consumer has read everything, VAST Event Broker answers its next fetch straight away with no records, instead of holding the request open until data arrives. A caught-up consumer therefore sends more fetch requests than it would against Apache Kafka. That's expected. If many idle consumers matter for your load, make them wait between polls when they get nothing back.

### Topic size from the table

A topic is a table, so its row count and size are also available through the VAST DataBase SDK. The figures cover all partitions.

```
with session.transaction() as tx:
    stats = tx.bucket(os.environ["EVENTS_BUCKET"]).schema("kafka_topics").table("<topic>").get_stats()
    print(stats.num_rows, stats.size_in_bytes)
```

See [VAST DataBase monitoring](/develop/docs/dev-database-operations-monitoring-your-workload#table-statistics) for how these statistics behave.

## See also

- [Architecture](/develop/docs/dev-event-broker-concepts-architecture): leaders, coordinators and how fetches work.
- [Consuming](/develop/docs/dev-event-broker-guides-consuming): consumer groups and committing offsets.
- [Troubleshooting](/develop/docs/dev-event-broker-troubleshooting-index): empty fetches, rebalances and errors.
- [Monitoring Kafka performance with VAST Analytics](/docs-5-5/docs/publishing-events-to-vast-event-broker#monitoring-kafka-performance-with-vast-analytics).