Applies to VAST Cluster 5.5.
Kafka tools such as Kafka Connect, MirrorMaker 2, Schema Registry, Flink and Spark talk to VAST Event Broker through the same Kafka protocol as your own producers and consumers. So they're bound by the same limits: Event Broker implements part of the protocol, and a tool that relies on a missing part fails. This page gives the checklist every tool must pass, the status of each tool, and the VAST features that can replace a tool altogether.
Not every tool is covered for VAST Cluster 5.5, and the covered ones only in the setups the Coverage column lists. Where a tool or setup isn't covered, the settings on its page are deductions: what the tool's own documentation says it uses, checked against what Event Broker supports. Treat them as a starting point, and test before you rely on a tool.
Every Kafka tool connects with the same settings, or you skip the tool and query the topic table.
Checklist for any Kafka tool
Before you point a tool at Event Broker, find out how it uses each of these, in the tool's own documentation. Most tools embed the Apache Kafka Java client or librdkafka, and pass client settings through with a prefix.
| Check | What the tool must do | Why |
|---|---|---|
| Idempotent producer | Set enable.idempotence=false on every producer the tool creates. |
Event Broker doesn't support idempotent producing: it doesn't offer InitProducerId. The Apache Kafka Java producer turns idempotence on by default (from 3.0), so a tool that leaves it alone fails to produce. See Kafka compatibility. |
| Transactions and exactly-once | Leave the tool's exactly-once or transactional mode off, and don't set transactional.id. |
Transactions aren't supported, and there's no transaction coordinator. Design for at-least-once delivery: downstream readers must tolerate duplicates. |
| Compression | Set compression.type=none on every producer. |
The broker rejects compressed batches with UNSUPPORTED_COMPRESSION_TYPE. The cluster compresses stored data itself. Setting compression.type on a topic has no effect, so it doesn't let a tool compress. |
| Consumer groups | Use the classic group protocol with an eager assignor (range or roundrobin). group.instance.id has no effect. |
Cooperative rebalancing isn't supported, static membership is accepted but ignored, and the new consumer group protocol (KIP-848) isn't offered. The Java consumer defaults to the classic protocol, and its default assignor list uses the range assignor. Kafka Connect workers are different: they coordinate through the same classic JoinGroup requests but with their own rebalance protocol, run on the client side. Check how your tool's version behaves upstream, and test it. See Kafka Connect. |
| Kafka ACLs | Don't rely on the tool to read or manage Kafka ACLs. | There are no Kafka ACLs. DescribeAcls returns SECURITY_DISABLED, and the calls that create or delete ACLs aren't offered. Access is controlled by identity policies. See Security. |
| Adding partitions | Create topics with the partition count you need. The tool mustn't expect to add partitions. | The partition count is fixed at creation, and CreatePartitions isn't offered. |
| Changing topic configs | The tool must use the non-incremental AlterConfigs, or not change topic configs. |
IncrementalAlterConfigs isn't offered. Newer admin tools use it by default. See topic configuration. |
| Topic creation | The tool must create its topics through the Admin API, or you create them first. A replication factor is required by many tools: any value is accepted. | There's no automatic topic creation: producing to a missing topic fails with UNKNOWN_TOPIC_OR_PARTITION. The broker ignores the replication factor. |
| Network | Every machine that runs the tool must reach every VIP in the Kafka view's VIP pool, on the port you use. | The broker advertises each VIP as a separate broker, by IP address. A tool that reaches only its bootstrap addresses can't reach partition leaders. See Kafka compatibility. |
| Authentication | Configure the view's authentication mode: none, SASL/PLAIN with a VAST local user, or mTLS (5.5). Port 9092 is unencrypted, 9093 is TLS. | Other SASL mechanisms (SCRAM, GSSAPI, OAUTHBEARER) aren't supported. Each client the tool creates (producer, consumer, admin) needs the settings. See Security. |
If a tool sends a request the broker doesn't support at all, the broker closes the connection without an error code. The tool then reports a disconnect or a timeout, not a clear error. See troubleshooting.
Status
| Tool | What it's for with Event Broker | Coverage | Page |
|---|---|---|---|
| Kafka Connect | Run source and sink connectors that move data between Event Broker topics and other systems | One worker, unencrypted; security modes and sinks not covered | Kafka Connect |
| Debezium | Capture row changes from a database (change data capture) into topics, as Kafka Connect source connectors | PostgreSQL, Debezium 3.7 | Kafka Connect: Debezium example |
| MirrorMaker 2 | Copy topics between Apache Kafka clusters and Event Broker | Kafka 4.1.0 to Event Broker; reverse direction and security modes not covered | MirrorMaker 2 |
| Confluent Schema Registry and serializers | Store Avro, Protobuf or JSON Schema schemas, and serialize event values against them | Registry 8.0.0: serializers work; the registry's own storage must be on Apache Kafka | Schema Registry |
| Apache Flink | Stream processing with Event Broker topics as Kafka sources and sinks | Flink 2.2, Flink SQL, at-least-once; exactly-once doesn't work | Stream processing |
| Spark Structured Streaming | Stream processing with Event Broker topics as Kafka sources and sinks | Spark 3.5.1 | Stream processing |
| Kafka Streams / ksqlDB | Stream processing inside Kafka client applications | Not covered | None |
| kcat and Kafka UI tools | Inspect topics, consume and produce by hand, browse consumer groups | Not covered | None. Screens that show or manage ACLs fail, because DescribeAcls returns SECURITY_DISABLED (Kafka compatibility). Screens that edit topic configs need the non-incremental AlterConfigs. |
Alternatives on VAST
Some jobs that need a Kafka tool elsewhere are built into VAST:
- Query the topic table instead of running a sink connector. Every topic is already a VAST DataBase table in the
kafka_topicsschema of the Kafka view's bucket. You can read it with the VAST DataBase SDK, the Query Engine, Trino or Spark, without copying events anywhere. See Query topics as tables. - Use blob expansion instead of a parse job. Blob expansion (5.5) parses each JSON event as it's produced and writes the fields you choose into a typed table, in the same transaction as the event. You don't need a stream job that reads JSON from a topic and writes columns to a table. See Blob expansion.
See also
- Kafka compatibility: supported APIs and what a client sees for the rest.
- Topic configuration: which topic configs a tool can set.
- Producing and consuming: the client settings these tools need.
- Security: authentication modes and identity-policy actions.
- Kafka Connect, MirrorMaker 2, Schema Registry, Stream processing