---
title: "Kafka compatibility"
slug: "dev-event-broker-reference-kafka-compatibility"
description: "VAST Event Broker supports Kafka clients, detailing APIs, features, and limitations for seamless integration and efficient event management."
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-event-broker-reference-kafka-compatibility"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka compatibility

*Applies to VAST Cluster 5.5.*

VAST Event Broker speaks the Kafka wire protocol, so Kafka clients connect to it as they would to any Kafka broker. It implements part of the protocol. This page lists the clients VAST supports, the Kafka APIs and versions the broker accepts, the Kafka features it doesn't support, and what your client sees when it uses one of them.

## Supported clients

| Client | Status |
| --- | --- |
| Confluent Kafka Python client (`confluent-kafka`) 2.4 to 2.8 | Supported. See [Publishing events to VAST Event Broker](/docs-5-5/docs/publishing-events-to-vast-event-broker). |
| Confluent Kafka clients for Java, .NET, C (librdkafka) and Rust | Supported |
| Sarama (Go) | Supported |
| franz-go | Supported. On TLS (port 9093), set `kgo.MaxVersions(kversion.V2_8_0())`: without it, connections to the brokers other than the first one fail with `broker closed the connection immediately during api versions negotiation`. |
| aiokafka 0.13 and later | Works with its default settings. |
| aiokafka 0.10 to 0.12 | Set `api_version="1.1.0"` on the producer, consumer and admin client. Without it, consuming in a group times out. |
| kafka-python 2.3.2 | Works. |
| kafka-python 3.0.12 and later | Producers need `enable_idempotence=False`. |
| kafka-python 3.0.0 to 3.0.11 | Upgrade to 3.0.12, or apply the patch in [kafka-python 3.x](/develop/docs/dev-event-broker-reference-kafka-compatibility#kafka-python-3x). Producers need `enable_idempotence=False`. |

The examples in these docs use `confluent-kafka`. Configure any client as this page describes: turn off idempotence and compression, and use an eager assignor. Other clients that speak the Kafka protocol may work too, but aren't on this list.

### kafka-python 3.x

A kafka-python 3.0.0 to 3.0.11 consumer finds no starting offset on VAST Event Broker, so it reads no records, and `offsets_for_times()` returns `None` for every partition. The consumer marks its offset lookups (`ListOffsets` requests) as coming from broker 0, where a consumer should send `-1`, and the broker doesn't return offsets for that request. kafka-python 3.0.12 fixes this ([kafka-python issue 3173](https://github.com/dpkp/kafka-python/issues/3173)), and 2.3.2 isn't affected.

Upgrade to kafka-python 3.0.12 or later. If you can't upgrade yet, apply this patch once, before you create a consumer. It makes the consumer send `-1`:

```
import kafka.consumer.fetcher as fetcher
_orig = fetcher.ListOffsetsRequest

def _list_offsets(*args, **kwargs):
    kwargs.setdefault("replica_id", -1)
    return _orig(*args, **kwargs)

for name in ("min_version_for_isolation_level", "min_version_for_timestamp", "ListOffsetsTopic"):
    setattr(_list_offsets, name, getattr(_orig, name))
fetcher.ListOffsetsRequest = _list_offsets
```

kafka-python 3.0 producers also turn on idempotence by default, so pass `enable_idempotence=False` to `KafkaProducer` (see [idempotent producer](/develop/docs/dev-event-broker-reference-kafka-compatibility#unsupported-features)).

## Listeners

Each virtual IP (VIP) in the Kafka view's VIP pool accepts Kafka connections on two ports:

| Port | Connection |
| --- | --- |
| 9092 | Unencrypted |
| 9093 | TLS |

The platform team chooses which of the two the view allows, and which authentication each one requires. See [security](/develop/docs/dev-event-broker-operations-security).

When a client asks for cluster metadata, the broker lists every VIP in the pool as a separate broker, with the VIP's IP address as its host and the port the client connected on. It doesn't advertise host names. Your client must be able to reach every VIP in the pool, not only the ones in its bootstrap list.

## Supported Kafka APIs

VAST Event Broker supports the Producer API, the Consumer API, consumer groups, and these Admin API operations: create topics, get topic configuration (`describeConfigs`), update topic configuration (`alterConfigs`), delete topics and delete consumer groups. Topic compaction and SSL are also supported.

The table below lists the API keys the broker advertises in its `ApiVersions` response and the version range of each. The ranges can differ between releases. Clients pick a version inside the range automatically when they connect.

| Key | API | Versions | Notes |
| --- | --- | --- | --- |
| 0 | Produce | 0–9 | The broker accepts only record batches in message format v2, which clients send with Produce v3 and later. Older formats are rejected with `CORRUPT_MESSAGE`. |
| 1 | Fetch | 4–6 | No fetch sessions (every fetch is a full fetch), no topic IDs, no leader epochs. |
| 2 | ListOffsets | 1–4 |  |
| 3 | Metadata | 0–12 |  |
| 8 | OffsetCommit | 0–6 | `group.instance.id` (static membership) is ignored. |
| 9 | OffsetFetch | 0–6 | No "require stable" flag. |
| 10 | FindCoordinator | 0–3 | Group coordinators only. A transaction-coordinator lookup gets `COORDINATOR_NOT_AVAILABLE`. |
| 11 | JoinGroup | 0–4 | Classic group protocol. Static membership is ignored. |
| 12 | Heartbeat | 0–2 |  |
| 13 | LeaveGroup | 0–2 |  |
| 14 | SyncGroup | 0–2 |  |
| 15 | DescribeGroups | 0–3 |  |
| 16 | ListGroups | 0–3 | No filtering by group state. |
| 17 | SaslHandshake | 0–1 | `PLAIN` is the only mechanism. |
| 18 | ApiVersions | 0–3 |  |
| 19 | CreateTopics | 0–4 |  |
| 20 | DeleteTopics | 0–3 |  |
| 29 | DescribeAcls | 0–1 | Always returns `SECURITY_DISABLED`. There are no Kafka ACLs; access is controlled by identity policies. |
| 32 | DescribeConfigs | 0–2 |  |
| 33 | AlterConfigs | 0–2 | The non-incremental call only. |
| 36 | SaslAuthenticate | 0 | No re-authentication. |
| 42 | DeleteGroups | 0–1 |  |

Every other API key is unsupported. That includes `InitProducerId` (22), the transaction APIs (24 to 28), `DeleteRecords` (21), `OffsetForLeaderEpoch` (23), `CreateAcls` and `DeleteAcls` (30, 31), `CreatePartitions` (37), `IncrementalAlterConfigs` (44), `OffsetDelete` (47), the client-quota and SCRAM-credential APIs (48 to 51), `DescribeCluster` (60), and the new consumer group protocol (`ConsumerGroupHeartbeat`, 68).

### What your client sees for an unsupported API

Most clients read the broker's `ApiVersions` response first and refuse, on the client side, to call an API the broker doesn't list. `confluent-kafka` raises `_UNSUPPORTED_FEATURE`, for example `CreatePartitions (KIP-195) not supported by broker, requires broker version >= 1.0.0`. Some calls have a fallback: `describe_cluster()` works, because the client uses `Metadata` instead of `DescribeCluster`.

If a request with an unsupported API key, or with a version outside the advertised range, does reach the broker, the broker closes the TCP connection without sending an error code. This is how Apache Kafka brokers behave too. The client usually reports a disconnect and may retry.

An `ApiVersions` request with a version the broker doesn't support gets `UNSUPPORTED_VERSION` and the broker's supported `ApiVersions` range, so the client can retry with a lower version.

## Unsupported features

| Feature | Status | What the client sees | What to do |
| --- | --- | --- | --- |
| Over-the-wire compression | Not supported. VAST compresses the stored data itself. | The broker rejects a compressed batch with `CORRUPT_MESSAGE` ("Invalid message"). The Confluent client retries it until `message.timeout.ms`, then reports `_MSG_TIMED_OUT`. Setting `compression.type` on the topic doesn't change this. | Set `compression.type` to `none` on producers. |
| Idempotent producer | Not supported | The broker doesn't offer `InitProducerId`. An idempotent `confluent-kafka` producer raises a fatal `_UNSUPPORTED_FEATURE`: `Idempotent producer not supported by any of the 1 connected broker(s): requires Apache Kafka broker version >= 0.11.0`. | Set `enable.idempotence` to `false` explicitly. The Java producer turns it on by default from Apache Kafka 3.0. |
| Transactions and exactly-once | Not supported | Same as the idempotent producer: there's no `InitProducerId` and no transaction coordinator. `init_transactions()` fails with `_TIMED_OUT`. | Don't set `transactional.id`. Make consumers tolerate duplicates. |
| Automatic topic creation | Not supported | A metadata request for a topic that doesn't exist gets `UNKNOWN_TOPIC_OR_PARTITION`, even when it asks for auto-creation, and no topic is created. A `confluent-kafka` producer keeps waiting for the topic and reports `_MSG_TIMED_OUT`. | Create topics first. See [topics](/develop/docs/dev-event-broker-guides-topics). |
| Adding partitions to a topic | Not supported: the partition count is fixed when the topic is created. | `CreatePartitions` isn't offered, so `confluent-kafka`'s `create_partitions()` fails with `_UNSUPPORTED_FEATURE` before sending anything. | Choose the partition count when you create the topic. To change it, create a new topic. |
| Incremental config changes | Not supported | `IncrementalAlterConfigs` isn't offered, so `incremental_alter_configs()` fails with `_UNSUPPORTED_FEATURE`. | Use the non-incremental `AlterConfigs` call (`AdminClient.alter_configs()` in `confluent-kafka`), or ask the platform team to change the topic. See [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration). |
| Deleting records or committed offsets | Not supported | `DeleteRecords` and `OffsetDelete` aren't offered. `delete_records()` fails with `_UNSUPPORTED_FEATURE`. | Use retention to remove old records. Delete the whole consumer group to remove its offsets. |
| Topic access by UUID | Not supported | Clients fall back to topic names, because the broker's Fetch versions predate topic IDs. | Nothing. |
| Kafka ACL API | Not supported | `DescribeAcls` returns `SECURITY_DISABLED`, which `confluent-kafka`'s `describe_acls()` returns as an empty list. `CreateAcls` and `DeleteAcls` aren't offered: `create_acls()` and `delete_acls()` fail with `_UNSUPPORTED_FEATURE`. | Grant access with identity policies. See [security](/develop/docs/dev-event-broker-operations-security). |
| SASL mechanisms other than `PLAIN` | Not supported. VAST documents SASL/PLAIN and mTLS only. | The handshake answer lists `PLAIN` as the only mechanism, and the broker then closes the connection. `confluent-kafka` logs `SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure`. | Use `sasl.mechanism=PLAIN` with a VAST local user, or mTLS. |
| SASL re-authentication | Not supported | The broker gives no session lifetime, so clients don't re-authenticate. | Nothing. |
| Static group membership (`group.instance.id`) | Accepted but ignored | The broker's `JoinGroup` versions predate static membership, so the client joins as an ordinary dynamic member: a consumer with `group.instance.id` set joined and consumed normally. | Nothing, but don't rely on static membership to avoid rebalances. |
| Cooperative rebalancing | Not supported | Nothing at once: with two consumers using `cooperative-sticky`, the second join moved one partition and left the others assigned, without errors. | Use an eager assignor such as `range` or `roundrobin`, not `cooperative-sticky`, because VAST doesn't support cooperative rebalancing. |
| New consumer group protocol (KIP-848) | Not supported | `ConsumerGroupHeartbeat` isn't offered. A consumer with `group.protocol=consumer` fails with `_FATAL`: `Required feature not supported by broker: ConsumerGroupHeartbeat`. | Use the classic protocol (`group.protocol=classic`, the default in `confluent-kafka`). |
| `READ_UNCOMMITTED` isolation | Not supported | The setting is accepted, but the consumer only ever gets committed records, as with `read_committed`. | Nothing. All records a consumer sees are committed. |
| Fetch sessions and delayed fetch parameters | Not supported: only full fetches are applied | Every fetch is a full fetch. The broker ignores `fetch.min.bytes` and answers an empty fetch at once instead of waiting `fetch.wait.max.ms`: an idle consumer with `fetch.wait.max.ms=5000` and `fetch.min.bytes=100000` sent over 13,000 fetch requests in 10 seconds. | Nothing. An idle consumer polls far more often than it would against Apache Kafka. |
| Client rack awareness | Not supported | `client.rack` is accepted. The broker never names a preferred read replica, so it has no effect. | Nothing. |
| Replicas | There's one copy of each partition as far as Kafka is concerned. Durability comes from VAST storage. | `CreateTopics` ignores the replication factor: a topic created with `replication_factor=3` has one replica. Explicit replica assignments fail with `INVALID_REPLICA_ASSIGNMENT`. Metadata shows each partition's leader as its only replica and in-sync replica. | Pass any replication factor, for example `1`. |
| Broker default partition count | Not supported | `CreateTopics` with `num_partitions` of `-1` fails with `INVALID_PARTITIONS` ("Broker: Invalid number of partitions"). `-1` is the default in `confluent-kafka`'s `NewTopic`, so `NewTopic("name")` without a count fails. | Always pass the partition count. |

For the `acks` setting: `acks=0`, `acks=1` and `acks=all` all work. `acks=1` and `acks=all` behave the same way: the broker replies after the produce request's database transaction commits. With `acks=0` the broker sends no reply, so the client never learns the offset. See [producing](/develop/docs/dev-event-broker-guides-producing).

## Other limitations

- Event publishing and consuming, and topic management, aren't subject to VAST protocol auditing or quality of service (QoS).
- VAST replication of consumer groups isn't supported.
- Event queries based on the topic partition aren't supported.
- The end offset that the broker reports (through `get_watermark_offsets()`, or a `ListOffsets` request for the latest offset) can trail the newest events by tens of seconds: it can catch up 10 to 55 seconds after a burst of writes, while a consumer can already read the new events. Tools that plan their work from end offsets, such as lag monitors and Spark Structured Streaming, see new events later than a consumer does.

For message sizes, partition counts and consumer group counts, see [limits](/develop/docs/dev-event-broker-reference-limits).

## See also

- [Topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration): which topic configs are honoured.
- [Limits](/develop/docs/dev-event-broker-reference-limits)
- [Producing](/develop/docs/dev-event-broker-guides-producing) and [consuming](/develop/docs/dev-event-broker-guides-consuming): client settings.
- [Security](/develop/docs/dev-event-broker-operations-security): authentication modes and identity-policy actions.
- [Troubleshooting](/develop/docs/dev-event-broker-troubleshooting-index)
- [Publishing events to VAST Event Broker](/docs-5-5/docs/publishing-events-to-vast-event-broker), "Kafka Protocol Support"