Monitoring

Applies to VAST Cluster 5.5.

This page covers what to watch while you produce and consume: the Kafka reports your platform team can open in VAST Analytics, and the consumer lag you can check yourself from a Kafka client.

On the cluster

Your platform team can open these in the VAST Web UI. Ask them to share the numbers during load tests.

What	Where	What it tells you
Kafka produce and fetch latency and bandwidth	Analytics → Predefined Analytics, the reports with Kafka in the name, filtered by view or by CNode	Whether producing and fetching are slow, and whether load spreads over the CNodes. If one CNode carries most of it, check that clients use all the virtual IPs (see architecture).
Events and capacity per topic	The topic's details in the VAST Web UI, CLI `topic show`, or REST `GET /topics/show/`	How many events a topic holds and how much space it takes, so you can check retention is keeping it in bounds.
Blob expansion statistics	Analytics, as custom analytics	How many events were parsed into the target table, and how many failed. See blob expansion.
Publishing to third-party brokers	Analytics, the published-events reports; and the Prometheus endpoint for external brokers	Only for S3 bucket events sent to an external Kafka cluster.

On the cluster dashboard, the Latency popup for Kafka also shows an Avg field, which is always 0 in 5.5. Ignore it.

The Prometheus Kafka endpoint covers events the cluster publishes to external brokers. Measure consumer lag on VAST Event Broker topics from your clients, as below.

Producing, consuming and topic management aren't covered by VAST protocol auditing or Quality of Service limits, so neither shows Kafka traffic.

From your client

Consumer lag

Lag is how far a consumer group is behind: for each partition, the high watermark minus the group's committed offset. On VAST Event Broker the high watermark is the offset after the last committed event.

Offsets can have gaps (a rolled-back produce request uses up offsets, and compaction removes events), so treat lag as an upper bound on the number of events still to read, not an exact count. The other way round, lag can read low for a short while: the end offset that the broker reports (through get_watermark_offsets(), or a ListOffsets request for the latest offset) can trail the newest events by tens of seconds: it can catch up 10 to 55 seconds after a burst of writes, while a consumer can already read the new events. Watch lag over minutes, not seconds.

Set up

Set KAFKA_BOOTSTRAP (and, for SASL/PLAIN, KAFKA_USERNAME and KAFKA_PASSWORD; see setup). This creates a one-partition topic, produces 10 events, and has a consumer group read and commit 4 of them:

import os
import time
from confluent_kafka import Consumer, Producer, TopicPartition
from confluent_kafka.admin import AdminClient, NewTopic

TOPIC = "docs-monitoring-lag"
GROUP = "docs-monitoring-lag-group"

conf = {"bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"]}
if os.environ.get("KAFKA_USERNAME"):
    conf.update({
        "security.protocol": "SASL_PLAINTEXT",
        "sasl.mechanism": "PLAIN",
        "sasl.username": os.environ["KAFKA_USERNAME"],
        "sasl.password": os.environ["KAFKA_PASSWORD"],
    })

admin = AdminClient(conf)
admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=1)])[TOPIC].result()

producer = Producer({**conf, "compression.type": "none", "enable.idempotence": False})
for i in range(10):
    producer.produce(TOPIC, key=f"key-{i}".encode(), value=f"event {i}".encode())
producer.flush()

consumer = Consumer({**conf, "group.id": GROUP, "auto.offset.reset": "earliest",
                     "enable.auto.commit": False})
consumer.subscribe([TOPIC])
read = 0
deadline = time.time() + 60
while read < 4 and time.time() < deadline:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    read += 1
    if read == 4:
        consumer.commit(message=msg, asynchronous=False)
print("read and committed:", read)

read and committed: 4

Check the lag

committed() returns the group's committed offset per partition, and get_watermark_offsets() the low and high watermarks:

partitions = [TopicPartition(TOPIC, p) for p in admin.list_topics(TOPIC).topics[TOPIC].partitions]
committed = consumer.committed(partitions, timeout=10)

total = 0
for tp in committed:
    low, high = consumer.get_watermark_offsets(tp, timeout=10)
    position = tp.offset if tp.offset >= 0 else low   # no commit yet: lag from the start
    total += high - position
print("lag:", total)

lag: 6

Any consumer can run this check for any group: create a consumer with that group.id, but don't subscribe, so it doesn't join the group. With authorization on, it needs KafkaRead on the group and KafkaDescribe on the topic (see security).

Clean up

consumer.close()
admin.delete_consumer_groups([GROUP])[GROUP].result()
admin.delete_topics([TOPIC])[TOPIC].result()
print("deleted", TOPIC, "and", GROUP)

deleted docs-monitoring-lag and docs-monitoring-lag-group

A group can only be deleted once it has no members, so close the consumer first.

Fetch rate when caught up

When a consumer has read everything, VAST Event Broker answers its next fetch straight away with no records, instead of holding the request open until data arrives. A caught-up consumer therefore sends more fetch requests than it would against Apache Kafka. That's expected. If many idle consumers matter for your load, make them wait between polls when they get nothing back.

Topic size from the table

A topic is a table, so its row count and size are also available through the VAST DataBase SDK. The figures cover all partitions.

with session.transaction() as tx:
    stats = tx.bucket(os.environ["EVENTS_BUCKET"]).schema("kafka_topics").table("<topic>").get_stats()
    print(stats.num_rows, stats.size_in_bytes)

See VAST DataBase monitoring for how these statistics behave.