Kafka compatibility

Applies to VAST Cluster 5.5.

VAST Event Broker speaks the Kafka wire protocol, so Kafka clients connect to it as they would to any Kafka broker. It implements part of the protocol. This page lists the clients VAST supports, the Kafka APIs and versions the broker accepts, the Kafka features it doesn't support, and what your client sees when it uses one of them.

Supported clients

Client	Status
Confluent Kafka Python client (`confluent-kafka`) 2.4 to 2.8	Supported. See Publishing events to VAST Event Broker.
Confluent Kafka clients for Java, .NET, C (librdkafka) and Rust	Supported
Sarama (Go)	Supported
franz-go	Supported. On TLS (port 9093), set `kgo.MaxVersions(kversion.V2_8_0())`: without it, connections to the brokers other than the first one fail with `broker closed the connection immediately during api versions negotiation`.
aiokafka 0.13 and later	Works with its default settings.
aiokafka 0.10 to 0.12	Set `api_version="1.1.0"` on the producer, consumer and admin client. Without it, consuming in a group times out.
kafka-python 2.3.2	Works.
kafka-python 3.0.12 and later	Producers need `enable_idempotence=False`.
kafka-python 3.0.0 to 3.0.11	Upgrade to 3.0.12, or apply the patch in kafka-python 3.x. Producers need `enable_idempotence=False`.

The examples in these docs use confluent-kafka. Configure any client as this page describes: turn off idempotence and compression, and use an eager assignor. Other clients that speak the Kafka protocol may work too, but aren't on this list.

kafka-python 3.x

A kafka-python 3.0.0 to 3.0.11 consumer finds no starting offset on VAST Event Broker, so it reads no records, and offsets_for_times() returns None for every partition. The consumer marks its offset lookups (ListOffsets requests) as coming from broker 0, where a consumer should send -1, and the broker doesn't return offsets for that request. kafka-python 3.0.12 fixes this (kafka-python issue 3173), and 2.3.2 isn't affected.

Upgrade to kafka-python 3.0.12 or later. If you can't upgrade yet, apply this patch once, before you create a consumer. It makes the consumer send -1:

import kafka.consumer.fetcher as fetcher
_orig = fetcher.ListOffsetsRequest

def _list_offsets(*args, **kwargs):
    kwargs.setdefault("replica_id", -1)
    return _orig(*args, **kwargs)

for name in ("min_version_for_isolation_level", "min_version_for_timestamp", "ListOffsetsTopic"):
    setattr(_list_offsets, name, getattr(_orig, name))
fetcher.ListOffsetsRequest = _list_offsets

kafka-python 3.0 producers also turn on idempotence by default, so pass enable_idempotence=False to KafkaProducer (see idempotent producer).

Listeners

Each virtual IP (VIP) in the Kafka view's VIP pool accepts Kafka connections on two ports:

Port	Connection
9092	Unencrypted
9093	TLS

The platform team chooses which of the two the view allows, and which authentication each one requires. See security.

When a client asks for cluster metadata, the broker lists every VIP in the pool as a separate broker, with the VIP's IP address as its host and the port the client connected on. It doesn't advertise host names. Your client must be able to reach every VIP in the pool, not only the ones in its bootstrap list.

Supported Kafka APIs

VAST Event Broker supports the Producer API, the Consumer API, consumer groups, and these Admin API operations: create topics, get topic configuration (describeConfigs), update topic configuration (alterConfigs), delete topics and delete consumer groups. Topic compaction and SSL are also supported.

The table below lists the API keys the broker advertises in its ApiVersions response and the version range of each. The ranges can differ between releases. Clients pick a version inside the range automatically when they connect.

Key	API	Versions	Notes
0	Produce	0–9	The broker accepts only record batches in message format v2, which clients send with Produce v3 and later. Older formats are rejected with `CORRUPT_MESSAGE`.
1	Fetch	4–6	No fetch sessions (every fetch is a full fetch), no topic IDs, no leader epochs.
2	ListOffsets	1–4
3	Metadata	0–12
8	OffsetCommit	0–6	`group.instance.id` (static membership) is ignored.
9	OffsetFetch	0–6	No "require stable" flag.
10	FindCoordinator	0–3	Group coordinators only. A transaction-coordinator lookup gets `COORDINATOR_NOT_AVAILABLE`.
11	JoinGroup	0–4	Classic group protocol. Static membership is ignored.
12	Heartbeat	0–2
13	LeaveGroup	0–2
14	SyncGroup	0–2
15	DescribeGroups	0–3
16	ListGroups	0–3	No filtering by group state.
17	SaslHandshake	0–1	`PLAIN` is the only mechanism.
18	ApiVersions	0–3
19	CreateTopics	0–4
20	DeleteTopics	0–3
29	DescribeAcls	0–1	Always returns `SECURITY_DISABLED`. There are no Kafka ACLs; access is controlled by identity policies.
32	DescribeConfigs	0–2
33	AlterConfigs	0–2	The non-incremental call only.
36	SaslAuthenticate	0	No re-authentication.
42	DeleteGroups	0–1

Every other API key is unsupported. That includes InitProducerId (22), the transaction APIs (24 to 28), DeleteRecords (21), OffsetForLeaderEpoch (23), CreateAcls and DeleteAcls (30, 31), CreatePartitions (37), IncrementalAlterConfigs (44), OffsetDelete (47), the client-quota and SCRAM-credential APIs (48 to 51), DescribeCluster (60), and the new consumer group protocol (ConsumerGroupHeartbeat, 68).

What your client sees for an unsupported API

Most clients read the broker's ApiVersions response first and refuse, on the client side, to call an API the broker doesn't list. confluent-kafka raises _UNSUPPORTED_FEATURE, for example CreatePartitions (KIP-195) not supported by broker, requires broker version >= 1.0.0. Some calls have a fallback: describe_cluster() works, because the client uses Metadata instead of DescribeCluster.

If a request with an unsupported API key, or with a version outside the advertised range, does reach the broker, the broker closes the TCP connection without sending an error code. This is how Apache Kafka brokers behave too. The client usually reports a disconnect and may retry.

An ApiVersions request with a version the broker doesn't support gets UNSUPPORTED_VERSION and the broker's supported ApiVersions range, so the client can retry with a lower version.

Unsupported features

Feature	Status	What the client sees	What to do
Over-the-wire compression	Not supported. VAST compresses the stored data itself.	The broker rejects a compressed batch with `CORRUPT_MESSAGE` ("Invalid message"). The Confluent client retries it until `message.timeout.ms`, then reports `_MSG_TIMED_OUT`. Setting `compression.type` on the topic doesn't change this.	Set `compression.type` to `none` on producers.
Idempotent producer	Not supported	The broker doesn't offer `InitProducerId`. An idempotent `confluent-kafka` producer raises a fatal `_UNSUPPORTED_FEATURE`: `Idempotent producer not supported by any of the 1 connected broker(s): requires Apache Kafka broker version >= 0.11.0`.	Set `enable.idempotence` to `false` explicitly. The Java producer turns it on by default from Apache Kafka 3.0.
Transactions and exactly-once	Not supported	Same as the idempotent producer: there's no `InitProducerId` and no transaction coordinator. `init_transactions()` fails with `_TIMED_OUT`.	Don't set `transactional.id`. Make consumers tolerate duplicates.
Automatic topic creation	Not supported	A metadata request for a topic that doesn't exist gets `UNKNOWN_TOPIC_OR_PARTITION`, even when it asks for auto-creation, and no topic is created. A `confluent-kafka` producer keeps waiting for the topic and reports `_MSG_TIMED_OUT`.	Create topics first. See topics.
Adding partitions to a topic	Not supported: the partition count is fixed when the topic is created.	`CreatePartitions` isn't offered, so `confluent-kafka`'s `create_partitions()` fails with `_UNSUPPORTED_FEATURE` before sending anything.	Choose the partition count when you create the topic. To change it, create a new topic.
Incremental config changes	Not supported	`IncrementalAlterConfigs` isn't offered, so `incremental_alter_configs()` fails with `_UNSUPPORTED_FEATURE`.	Use the non-incremental `AlterConfigs` call (`AdminClient.alter_configs()` in `confluent-kafka`), or ask the platform team to change the topic. See topic configuration.
Deleting records or committed offsets	Not supported	`DeleteRecords` and `OffsetDelete` aren't offered. `delete_records()` fails with `_UNSUPPORTED_FEATURE`.	Use retention to remove old records. Delete the whole consumer group to remove its offsets.
Topic access by UUID	Not supported	Clients fall back to topic names, because the broker's Fetch versions predate topic IDs.	Nothing.
Kafka ACL API	Not supported	`DescribeAcls` returns `SECURITY_DISABLED`, which `confluent-kafka`'s `describe_acls()` returns as an empty list. `CreateAcls` and `DeleteAcls` aren't offered: `create_acls()` and `delete_acls()` fail with `_UNSUPPORTED_FEATURE`.	Grant access with identity policies. See security.
SASL mechanisms other than `PLAIN`	Not supported. VAST documents SASL/PLAIN and mTLS only.	The handshake answer lists `PLAIN` as the only mechanism, and the broker then closes the connection. `confluent-kafka` logs `SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure`.	Use `sasl.mechanism=PLAIN` with a VAST local user, or mTLS.
SASL re-authentication	Not supported	The broker gives no session lifetime, so clients don't re-authenticate.	Nothing.
Static group membership (`group.instance.id`)	Accepted but ignored	The broker's `JoinGroup` versions predate static membership, so the client joins as an ordinary dynamic member: a consumer with `group.instance.id` set joined and consumed normally.	Nothing, but don't rely on static membership to avoid rebalances.
Cooperative rebalancing	Not supported	Nothing at once: with two consumers using `cooperative-sticky`, the second join moved one partition and left the others assigned, without errors.	Use an eager assignor such as `range` or `roundrobin`, not `cooperative-sticky`, because VAST doesn't support cooperative rebalancing.
New consumer group protocol (KIP-848)	Not supported	`ConsumerGroupHeartbeat` isn't offered. A consumer with `group.protocol=consumer` fails with `_FATAL`: `Required feature not supported by broker: ConsumerGroupHeartbeat`.	Use the classic protocol (`group.protocol=classic`, the default in `confluent-kafka`).
`READ_UNCOMMITTED` isolation	Not supported	The setting is accepted, but the consumer only ever gets committed records, as with `read_committed`.	Nothing. All records a consumer sees are committed.
Fetch sessions and delayed fetch parameters	Not supported: only full fetches are applied	Every fetch is a full fetch. The broker ignores `fetch.min.bytes` and answers an empty fetch at once instead of waiting `fetch.wait.max.ms`: an idle consumer with `fetch.wait.max.ms=5000` and `fetch.min.bytes=100000` sent over 13,000 fetch requests in 10 seconds.	Nothing. An idle consumer polls far more often than it would against Apache Kafka.
Client rack awareness	Not supported	`client.rack` is accepted. The broker never names a preferred read replica, so it has no effect.	Nothing.
Replicas	There's one copy of each partition as far as Kafka is concerned. Durability comes from VAST storage.	`CreateTopics` ignores the replication factor: a topic created with `replication_factor=3` has one replica. Explicit replica assignments fail with `INVALID_REPLICA_ASSIGNMENT`. Metadata shows each partition's leader as its only replica and in-sync replica.	Pass any replication factor, for example `1`.
Broker default partition count	Not supported	`CreateTopics` with `num_partitions` of `-1` fails with `INVALID_PARTITIONS` ("Broker: Invalid number of partitions"). `-1` is the default in `confluent-kafka`'s `NewTopic`, so `NewTopic("name")` without a count fails.	Always pass the partition count.

For the acks setting: acks=0, acks=1 and acks=all all work. acks=1 and acks=all behave the same way: the broker replies after the produce request's database transaction commits. With acks=0 the broker sends no reply, so the client never learns the offset. See producing.

Other limitations

Event publishing and consuming, and topic management, aren't subject to VAST protocol auditing or quality of service (QoS).
VAST replication of consumer groups isn't supported.
Event queries based on the topic partition aren't supported.
The end offset that the broker reports (through get_watermark_offsets(), or a ListOffsets request for the latest offset) can trail the newest events by tens of seconds: it can catch up 10 to 55 seconds after a burst of writes, while a consumer can already read the new events. Tools that plan their work from end offsets, such as lag monitors and Spark Structured Streaming, see new events later than a consumer does.

For message sizes, partition counts and consumer group counts, see limits.