---
title: "Troubleshooting VAST Event Broker"
slug: "dev-event-broker-troubleshooting-index"
description: "Troubleshoot Kafka errors with detailed causes and fixes, including connection issues, authorization, and producer initialization. Get insights now!"
updated: 2026-10-30T18:18:46Z
published: 2026-10-30T18:18:46Z
canonical: "docs.vastdata.com/dev-event-broker-troubleshooting-index"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting VAST Event Broker

*Applies to VAST Cluster 5.5.*

Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the page with the details. Kafka error names are shown as the broker or client reports them, for example `INVALID_PARTITIONS`. Your client library may print them in another form. For errors from the VAST DataBase SDK that aren't about topics, see the VAST DataBase [troubleshooting](/develop/docs/dev-database-troubleshooting-index) page.

## Connecting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| The client connects, then the broker closes the connection with no error, often over and over | The client sent an API, or an API version, the broker doesn't support. The broker closes the connection instead of returning an error. | See [connection closed with no error](/develop/docs/dev-event-broker-troubleshooting-index#connection-closed-with-no-error). |
| The client reaches the bootstrap address, then times out connecting to other brokers | The broker lists every VIP in the Kafka VIP pool as a broker. The client can't reach some of them. | Make sure the client can reach every VIP in the pool. Ask the platform team for the full list. See [setup](/develop/docs/dev-event-broker-operations-setup). |
| The connection is closed straight away on one port but works on the other | The Kafka view doesn't allow that connection type. Each view allows unencrypted connections (port 9092), TLS connections (port 9093), or both. The broker accepts the TCP connection and closes it at the first request. | Use the port and `security.protocol` the view is set up for. See [security](/develop/docs/dev-event-broker-operations-security). |
| The TLS handshake fails with a certificate verification error | The client doesn't trust the CA that issued the cluster's Kafka certificate, or the certificate has no SAN entry for the VIP you connected to. The broker advertises IP addresses, not host names. | Set `ssl.ca.location` to the internal CA certificate. Ask the platform team to check the certificate has one IP-address SAN entry for every VIP; with those, hostname verification passes. You can inspect it with `openssl s_client -connect <vip>:9093 -showcerts`. |
| The TLS handshake fails with `handshake failure` | The client allows only TLS 1.3. The broker negotiates TLS 1.2. | Allow TLS 1.2 in the client. |
| Requests fail with `NOT_LEADER_OR_FOLLOWER` or `NOT_COORDINATOR` for a short time | The cluster is moving VIPs, for example during an upgrade, and the client has stale metadata. | Nothing, if it clears up. Kafka clients refresh metadata and retry these errors. |

### Connection closed with no error

When a request uses an API key the broker doesn't support, or a version outside the range it advertises, VAST Event Broker closes the TCP connection without sending an error code. Apache Kafka brokers do the same. The client sees a disconnect, reconnects and may send the same request again.

Most clients check the broker's supported versions first and fail on the client side instead: `confluent-kafka` raises `_UNSUPPORTED_FEATURE`. You'll see the disconnects when a client, a tool or a framework sends one of these anyway:

- `InitProducerId`, from an idempotent or transactional producer. See [idempotence](/develop/docs/dev-event-broker-troubleshooting-index#the-producer-fails-to-initialize-with-idempotence).
- `CreatePartitions`, `IncrementalAlterConfigs`, `DeleteRecords`, `DescribeCluster`, or the ACL calls, from an admin client or a command-line tool.
- `ConsumerGroupHeartbeat`, from a consumer set to `group.protocol=consumer`.

Find which call the client was making when it was disconnected, and turn off or replace that call. [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility) lists the supported APIs and versions.

## Authentication and authorization

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `SASL_AUTHENTICATION_FAILED`; `confluent-kafka` logs `SASL authentication error:` with no text, and requests fail with `_TRANSPORT` | The user name or password is wrong, or the user isn't a VAST local user. Active Directory and LDAP users aren't supported, and S3 access keys don't work. The SASL authorization identity, if set, must be the same as the user name: otherwise the broker answers `SASL PLAIN authorization identity (authzid) must match authentication identity (authcid)`. | Use a VAST local user and its password: `sasl.username`, `sasl.password`. Ask the platform team to check the user exists in the view's tenant. |
| `SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure` | The client asked for SCRAM, GSSAPI (Kerberos) or OAUTHBEARER. Only PLAIN is supported: the broker lists `PLAIN` as its only mechanism and closes the connection. | Set `sasl.mechanism=PLAIN`. |
| `ILLEGAL_SASL_STATE`, which `confluent-kafka` shows as a broker transport failure | The client tries to authenticate with SASL, but the view doesn't use SASL for this connection type. | Use the `security.protocol` the view is set up for: `PLAINTEXT` or `SSL` instead of `SASL_PLAINTEXT` or `SASL_SSL`. |
| The connection is closed after the first request, with no error. `confluent-kafka` logs `Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication` | The view requires SASL and the client didn't authenticate. Before authenticating, the broker accepts only the version and handshake requests. | Set `security.protocol` to `SASL_PLAINTEXT` (port 9092) or `SASL_SSL` (port 9093), with `sasl.mechanism=PLAIN`. |
| An mTLS connection fails during the TLS handshake (`unknown ca`, `certificate revoked`, `handshake failure`), or right after it: `Disconnected while requesting ApiVersion: might be caused by broker version < 0.10` | During the handshake: the certificate isn't signed by a CA the cluster trusts for your tenant, it's in the CA's revocation list, or the client allows only TLS 1.3. Right after it: the client didn't present a certificate, or its common name isn't a VAST user. | Set `ssl.certificate.location` and `ssl.key.location`. Allow TLS 1.2 in the client. Ask the platform team which CA the view trusts and which user your CN must match. |
| Identity-policy statements have no effect: a denied user can still produce, consume or delete topics | Authorization isn't turned on for the Kafka view. Until it is, every operation is allowed for every client. | Ask the platform team to turn on authorization for the view. See [authorization](/develop/docs/dev-event-broker-troubleshooting-index#authorization-isnt-enforced-until-its-turned-on). |
| `TOPIC_AUTHORIZATION_FAILED`, `GROUP_AUTHORIZATION_FAILED` or `CLUSTER_AUTHORIZATION_FAILED` | Authorization is on, and no identity-policy statement allows the action for your user, or a statement denies it. | Ask the platform team to allow the action. A consumer needs `KafkaRead` and `KafkaDescribe` on the topic and `KafkaRead` on its consumer group. A producer needs `KafkaWrite` and `KafkaDescribe` on the topic. `describe_configs()` needs `KafkaDescribeConfigs` on the cluster and `KafkaCreate` on the topic. See [security](/develop/docs/dev-event-broker-operations-security#actions). |
| A topic you know exists is missing from `list_topics()`, and consuming it fails with `Subscribed topic not available: <topic>: Broker: Topic authorization failed` | Authorization is on and your user can't describe the topic. The broker leaves out topics you can't describe. | Ask the platform team to allow `KafkaDescribe` on the topic. |
| Saving an identity policy fails with `Resource VRN=… requires data engine or kafka prefix` | A Kafka statement's resource doesn't start with `vast:kafka:`. | Use the resource formats in [security](/develop/docs/dev-event-broker-operations-security#resources). |

### Authorization isn't enforced until it's turned on

Identity-policy statements for Kafka actions (`KafkaRead`, `KafkaWrite`, `KafkaCreate`, `KafkaDelete` and the others in [security](/develop/docs/dev-event-broker-operations-security#actions)) take effect only when the platform team turns on authorization for the Kafka view. Until then, the broker allows every operation, for every client, authenticated or not.

When authorization is on, the broker allows an operation only if a statement in the user's identity policy allows it. A request that matches no statement is denied, and so is a request from a connection that hasn't authenticated. That's why the view must also require SASL/PLAIN or mTLS on at least one connection type before authorization can be turned on.

## Producing

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| The producer fails to initialize, or fails on its first send, with an error about idempotence or `InitProducerId`: in `confluent-kafka`, a fatal `_UNSUPPORTED_FEATURE`, `Idempotent producer not supported by any of the 1 connected broker(s)`; in kafka-python, `IncompatibleBrokerVersion: Kafka broker does not support the 'InitProducerIdRequest' Kafka protocol.` | Idempotence is on. VAST Event Broker doesn't support idempotent producers. | Set `enable.idempotence=false`. See [below](/develop/docs/dev-event-broker-troubleshooting-index#the-producer-fails-to-initialize-with-idempotence). |
| Every produce fails with `_MSG_TIMED_OUT`; client debug logs show `Broker: Invalid message` (`CORRUPT_MESSAGE`) | The producer compresses batches. Over-the-wire compression isn't supported, and the client retries the rejected batch until it times out. | Set `compression.type=none`. Check frameworks that set compression for you. VAST compresses stored data itself. |
| Produce to a topic fails with `_MSG_TIMED_OUT` (`confluent-kafka`) or `UNKNOWN_TOPIC_OR_PARTITION` | The topic doesn't exist. Topics aren't created automatically, and `confluent-kafka` waits for the topic to appear until the message times out. | Create the topic first. See [topics](/develop/docs/dev-event-broker-guides-topics). |
| `MESSAGE_TOO_LARGE`; `confluent-kafka` reports `MSG_SIZE_TOO_LARGE` ("Broker: Message size too large") | The record batch is bigger than the topic's `max.message.bytes`, about 1 MB by default, or a record is bigger than the broker allows. | Lower the producer's `batch.size`, send smaller records, or ask for a larger `max.message.bytes`. See [limits](/develop/docs/dev-event-broker-reference-limits#messages). |
| Records are rejected with `INVALID_TIMESTAMP` ("Broker: Invalid timestamp") | The topic uses `CreateTime` and the record's timestamp is outside the topic's `message.timestamp.before.max.ms` / `after.max.ms` window. | Fix the producer's clock or timestamps, or widen the window. See [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration#honoured-configs). |
| `init_transactions()` fails with `_TIMED_OUT` | The producer has a `transactional.id`. Transactions aren't supported. | Remove `transactional.id`. |
| `TRANSACTIONAL_ID_AUTHORIZATION_FAILED` or `INVALID_PRODUCER_ID_MAPPING` | The producer sends transactional batches. Transactions aren't supported. | Remove `transactional.id`. |
| `CORRUPT_MESSAGE` | The client uses an old message format (before v2). | Upgrade the client. |

### The producer fails to initialize with idempotence

An idempotent producer starts by asking the broker for a producer ID with `InitProducerId`. VAST Event Broker doesn't support that API, so the producer can't start. Depending on the client, you get an error at start-up, an error on the first send, or the broker closes the connection. The Java producer turns idempotence on by default from Apache Kafka 3.0, and so does kafka-python 3.x. `confluent-kafka` leaves it off by default. Whatever your client's default, set it off explicitly.

With kafka-python 3.x, pass `enable_idempotence=False` to `KafkaProducer`. With `confluent-kafka`:

```
import os

producer_config = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "enable.idempotence": False,
    "compression.type": "none",
}
```

Without idempotence, a producer that retries after a timeout can write the same record twice. Make consumers tolerate duplicates. See [producing](/develop/docs/dev-event-broker-guides-producing).

## Topics and configs

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Deleting a topic fails with `Unknown broker error` (`UNKNOWN`) | The topic has a blob expansion. | Drop the expansion first, then delete the topic. See [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion#remove-the-expansion). |
| `create_partitions()` fails with `_UNSUPPORTED_FEATURE` (Confluent client), or the broker closes the connection (clients that don't check the broker's API versions) | You can't add partitions. The partition count is fixed when the topic is created. | Create a new topic with the partition count you need, and move producers and consumers to it. |
| `incremental_alter_configs()` fails with `_UNSUPPORTED_FEATURE` (`IncrementalAlterConfigs (KIP-339) not supported by broker`), or the broker closes the connection | Only the non-incremental `AlterConfigs` call is supported. | Use `alter_configs()`. On VAST it changes only the configs you send; the others keep their values. |
| `INVALID_CONFIG` ("Broker: Configuration is invalid") when you create a topic with `retention.bytes` | There's no size-based retention. The broker rejects `retention.bytes`. | Leave it out, and use `retention.ms`. Check tools that add it for you. |
| `INVALID_CONFIG` for another config | The config name isn't one the broker knows, its value has the wrong type or range, or it combines `compact` with a `max.message.bytes` above 5 MB. | Check [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration). |
| `INVALID_REQUEST` when you change a broker config | There are no broker configs to change. | Change topic configs instead. |
| `INVALID_PARTITIONS` ("Broker: Invalid number of partitions") | The partition count is missing (`-1`, the default in `confluent-kafka`'s `NewTopic`), less than 1, or above the maximum. | Pass an explicit count up to the [limit](/develop/docs/dev-event-broker-reference-limits#topics-and-partitions). |
| `INVALID_REPLICA_ASSIGNMENT` | You passed explicit replica assignments. | Leave them out. The replication factor is ignored, so any value works. |
| `INVALID_TOPIC_EXCEPTION`, which `confluent-kafka` reports as `TOPIC_EXCEPTION` ("Broker: Invalid topic") | The topic name has characters other than letters, digits, `.`, `_` and `-`, is `.` or `..`, or is too long. | Choose a valid name. |
| Old records are still there after the retention period | Retention runs in the background, so deletion lags. Retention is measured from the time the broker received each record, and a partition's last record is never deleted. Retention shorter than 6 hours is accepted but not applied: a topic with 5-minute retention can keep its records for close to an hour. | Nothing, unless records stay long after the retention period. Don't set retention below 6 hours. See [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration#notes-on-the-honoured-configs). |
| Duplicate keys in a compacted topic | Compaction runs in the background, so duplicates can exist at any given time. A compacted topic can take about ten minutes after the writes to keep only the latest record per key. | Nothing. Treat the latest record for each key as current. |

## Consuming

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Lag reads zero, or the end offset doesn't move, just after producing; a Spark `availableNow` run processes nothing | The end offset the broker reports trails the newest events by tens of seconds. Consumers can already read the events. | Wait, and check again; read until a poll returns nothing instead of comparing with the end offset. See [consuming](/develop/docs/dev-event-broker-guides-consuming#check-the-committed-offsets). |
| A Flink job with an `exactly-once` sink shows RUNNING but writes nothing | Exactly-once needs transactions, and the producer waits for a transaction coordinator Event Broker doesn't provide. | Use `at-least-once`. See [stream processing](/develop/docs/dev-event-broker-integrations-stream-processing). |
| You can't create a new consumer group, or `list_consumer_groups()` stops at 256 | A Kafka view supports up to 256 consumer groups (a view can accept and list more, but don't rely on it). | Delete groups you no longer use with `delete_consumer_groups()`. Committed offsets of a group are kept until the group or the topic is deleted. |
| `delete_consumer_groups()` fails with `NON_EMPTY_GROUP` ("Broker: The group is not empty") | The group still has active members. | Stop its consumers, wait for them to leave the group, then try again. |
| `JoinGroup failed: Broker: Invalid session timeout` (`INVALID_SESSION_TIMEOUT`) | `session.timeout.ms` is outside the range the broker allows (1 second and 1 hour are both refused). | Use a value close to the client's default. |
| Consumers rebalance or behave unexpectedly with `group.instance.id` or the `cooperative-sticky` assignor | Static membership is accepted but ignored: the consumer joins as an ordinary member. Cooperative rebalancing isn't supported; `cooperative-sticky` may appear to work, but don't depend on it. | Use `range` or `roundrobin`. `group.instance.id` can stay, but it has no effect. |
| A consumer configured with `group.protocol=consumer` fails with `_FATAL`: `Required feature not supported by broker: ConsumerGroupHeartbeat` | The new consumer group protocol (KIP-848) isn't supported. | Use `group.protocol=classic`. |
| The consumer jumps to the start or end of the topic | Its committed offset is no longer valid, usually because retention deleted those records. The broker returns `OFFSET_OUT_OF_RANGE` and the client applies `auto.offset.reset`. With `auto.offset.reset=error`, `confluent-kafka` reports `_AUTO_OFFSET_RESET`: `fetch failed due to requested offset not available on the broker: Broker: Offset out of range`. | Set `auto.offset.reset` to what you want, and keep consumers within the retention period. |
| After a failover, consumers start again from `auto.offset.reset` | Committed offsets aren't replicated. | Plan for reprocessing after failover. |
| An idle consumer sends many fetch requests | The broker answers an empty fetch straight away instead of waiting for data, whatever `fetch.wait.max.ms` and `fetch.min.bytes` say (over 13,000 requests in 10 seconds). | Nothing. It's expected. |
| `offsets_for_times()` returns `-1` for a partition | The time is after the partition's newest event. For an earlier time it returns the first offset whose timestamp is at or after it. | Ask for an earlier time, or seek to the end. |

## Querying topic tables

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Writing to a topic table with the VAST DataBase SDK or SQL fails with `InvalidArgument`; the SDK raises `vastdb.errors.BadRequest` | Topic tables are read-only through the VAST DataBase API: inserts, deletes and column changes are rejected. Only blob expansion is allowed. | Write events with a Kafka producer. To add columns, use [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion) into a separate table. |
| A topic disappeared after you dropped its table | Dropping a topic table deletes the Kafka topic and its committed offsets. | Don't drop topic tables unless you mean to delete the topic. |
| Database queries on a topic fail; the SDK raises `QueryDataInternalError` ("Query failed due to server error: error_code=20") | The topic's maximum message size is above 5 MB. Database queries aren't supported on such topics. | Lower the maximum message size, if the data allows. See [limits](/develop/docs/dev-event-broker-reference-limits#messages). |
| `vastdb.errors.MissingBlobExpansion` | The topic has no blob expansion on that source column. `Table.blob_expansion()` raises it when there's nothing to return. | Check the topic name and that `source_column_name` is `"value"`. Create the expansion with `create_blob_expansion()`. See [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion). |
| `create_blob_expansion()` fails with `BadRequest`: `blob expansion can only be created on Kafka topics, but source table is not a topic` | The source table isn't a topic table in the `kafka_topics` schema. | Point the expansion at the topic table. |
| `create_blob_expansion()` fails with `NotFound`: `failed to get target table for blob expansion` | The target table doesn't exist. | Create the target table first. |

See [querying topics](/develop/docs/dev-event-broker-guides-querying-topics).

## Before you ask for help

Collect these for the platform team:

- The client library and its version, for example `confluent-kafka` 2.8.
- The client configuration, without passwords: `bootstrap.servers`, `security.protocol`, `sasl.mechanism`, `enable.idempotence`, `compression.type`, `group.protocol`.
- The Kafka view's bucket, the topic and the consumer group.
- The full error, and the time it happened.

## See also

- [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility)
- [Topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration)
- [Limits](/develop/docs/dev-event-broker-reference-limits)
- [Security](/develop/docs/dev-event-broker-operations-security)
- [Producing](/develop/docs/dev-event-broker-guides-producing) and [consuming](/develop/docs/dev-event-broker-guides-consuming)
- VAST DataBase [troubleshooting](/develop/docs/dev-database-troubleshooting-index) and [error reference](/develop/docs/dev-database-troubleshooting-errors)