---
title: "Overview"
slug: "dev-event-broker-overview"
description: "VAST Event Broker offers a Kafka-compatible solution, integrating topics as VAST DataBase tables for seamless event production and consumption."
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-event-broker-overview"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

*Applies to VAST Cluster 5.5.*

VAST Event Broker is a Kafka-compatible event broker built into VAST Cluster. Kafka clients produce to it and consume from it over the Kafka protocol. Each topic is stored as a VAST DataBase table, so you can also read the same events with the VAST DataBase SDK, the VAST Query Engine, Trino or Spark, without copying them anywhere first.

## What it is

- **A protocol on the cluster, not a separate system.** The Kafka protocol is served by the cluster's CNodes, on the virtual IPs of a VIP pool. There are no broker machines to run.
- **A broker is a view.** Your platform team enables the Kafka protocol on a view, and that view is your broker. Clients connect to the virtual IPs of the VIP pool assigned to it ([KB](/docs-5-5/docs/publishing-events-to-vast-event-broker)).
- **A topic is a table.** Topics live in a schema called `kafka_topics` in the view's bucket. Each topic shows up as a table, so the topic name is the table name.
- **Kafka-compatible.** Producer, consumer, consumer-group and Kafka Admin APIs work, with the gaps listed in [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility). Supported clients include the Confluent Kafka clients (Python, Java, .NET, C and Rust), Sarama and franz-go; see [supported clients](/develop/docs/dev-event-broker-reference-kafka-compatibility#supported-clients).

See [concepts](/develop/docs/dev-event-broker-concepts-index) for the objects, and [architecture](/develop/docs/dev-event-broker-concepts-architecture) for how the cluster serves them.

## How it differs from running Apache Kafka

VAST Event Broker runs on the cluster's [DASE architecture](/develop/docs/dev-database-concepts-architecture-cluster-architecture), and every topic is a VAST DataBase table. That removes much of the work of running Kafka yourself:

![Two panels. Apache Kafka: brokers each store their partitions, and replicas of other brokers' partitions, on local disks, coordinated by a controller quorum, and a connector job copies topics into a separate table for analytics. VAST Event Broker: Kafka clients connect to the virtual IPs of a Kafka view; CNodes serve the protocol; every topic is a table on the cluster's shared storage, which SQL engines read directly. There are no replicas to copy and no connector.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/kafka-vs-event-broker.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A18Z&se=2026-11-08T09%3A55%3A18Z&sr=c&sp=r&sig=LZDvqhayV2jUzxxPt2gX0Es6J%2BM9V2Y8NtcD6nYL22A%3D)

*With VAST Event Broker, the topic is the table: no replicas to manage and no connector to copy events into a lake.*

| With Apache Kafka you... | With VAST Event Broker |
| --- | --- |
| Run broker servers, each storing its partitions on local disks | There are no broker servers. The cluster's CNodes serve the Kafka protocol, and topics are stored on the cluster's shared storage. |
| Set a replication factor, and watch for under-replicated partitions | `replication_factor` is accepted and ignored. Durability comes from the cluster's storage, which is protected by erasure coding, not from copies on several brokers. |
| Reassign partitions, copying their data, when you add brokers or one fails | No partition data belongs to a broker. A partition's leader is only the virtual IP clients send to, and a virtual IP moves to another CNode if its CNode goes offline. |
| Size each broker's disks for its share of retention | Topics use the cluster's capacity, like any other data in the view's bucket. Capacity and processing grow separately. |
| Run ZooKeeper or a KRaft controller quorum | There's no separate Kafka cluster, so there's no controller quorum to run. |
| Run Kafka Connect, or a stream job, to copy topics into a lake or warehouse table | A topic is a table. Query it with the VAST DataBase SDK, the VAST Query Engine, Trino or Spark as soon as the produce commits. |
| Run a stream-processing job to parse JSON events into columns | [Blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion) parses JSON into a table in the same transaction as the produce. |

The trade-off is Kafka compatibility: some client features aren't supported, such as idempotent and transactional producers and compressed batches. See [use cases and considerations](/develop/docs/dev-event-broker-overview#use-cases-and-considerations) and [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility).

## How it fits with VAST DataBase

![Kafka producers send events to a Kafka view, which is your broker, through the virtual IPs of its VIP pool; the cluster's CNodes serve the Kafka protocol. The cluster can also publish S3 bucket events into a topic. Each topic is a table, with partitions, in the kafka_topics schema of the view's bucket, and the cluster keeps committed offsets per consumer group. Kafka consumer groups consume the events in order, as they arrive. The VAST DataBase SDK, the VAST Query Engine, Trino or Spark query the same topic as a table, and blob expansion parses JSON event values into typed columns of a separate table, in the same transaction as the produce. Other Kafka views, for example on other tenants, are separate brokers with their own topics. Topic tables are read-only through the VAST DataBase API.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/eb-end-to-end.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A18Z&se=2026-11-08T09%3A55%3A18Z&sr=c&sp=r&sig=LZDvqhayV2jUzxxPt2gX0Es6J%2BM9V2Y8NtcD6nYL22A%3D)

*Producers write to a Kafka view; consumer groups, SQL engines and blob expansion all use the same topic tables.*

Events you produce are rows in a VAST DataBase table as soon as the produce request commits. That gives you two ways to read the same data:

| You want to | Use |
| --- | --- |
| Process events in order, as they arrive, with committed offsets per consumer group | A Kafka consumer. See [consuming](/develop/docs/dev-event-broker-guides-consuming). |
| Query events with filters, joins and aggregations, over any time range still retained | The VAST DataBase SDK, the VAST Query Engine, Trino or Spark. See [querying topics](/develop/docs/dev-event-broker-guides-querying-topics). |
| Turn JSON event payloads into typed columns of an ordinary table, as they arrive | Blob expansion (new in 5.5). See [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion). |

Topic tables are read-only through the VAST DataBase API: you write events with a Kafka producer, not with SQL or the SDK.

The cluster can also publish S3 bucket notifications (object created, removed or tagged) into a topic. See [S3 bucket events](/develop/docs/dev-event-broker-guides-s3-bucket-events).

## Use cases and considerations

Example use cases:

- **Queryable Kafka events:** you already produce or consume with Kafka clients and want the events queryable on the same cluster as your other data.
- **Analytics on a stream** without a connector job that copies topics into tables.
- **S3 object events** from a bucket on the cluster as a Kafka stream.
- **JSON events parsed into a typed table** in the same transaction as the write.

Important to note:

These aren't supported. For the details, see [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility) and [limits](/develop/docs/dev-event-broker-reference-limits).

- **Transactions and idempotent producers.** If your pipeline needs exactly-once delivery from the broker, VAST Event Broker doesn't provide it. Producers must turn idempotence off.
- **Over-the-wire compression.** Producers must send uncompressed batches. (The cluster compresses data itself.)
- **Automatic topic creation.** Create topics explicitly.
- **Changing the partition count.** The number of partitions is fixed when you create the topic.
- **Some consumer features:** cooperative rebalancing, READ UNCOMMITTED, fetch sessions and rack awareness. Static group membership (`group.instance.id`) is accepted but ignored.
- **More than 256 consumer groups per broker view.**
- **Protocol auditing and QoS.** Producing, consuming and topic management aren't covered by VAST protocol auditing or Quality of Service.
- **Replicating consumer groups.** Topics can be replicated, but committed consumer-group offsets aren't.

## What's new in 5.5

- **mTLS authentication.** Clients can authenticate with a client certificate over TLS. See [security](/develop/docs/dev-event-broker-operations-security).
- **Blob expansion.** JSON in the event value can be parsed into columns of a separate VAST DataBase table, in the same transaction as the event. See [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion).
- **More partitions.** Up to 20,000 partitions per topic, and 200,000 per broker view.
- **More topic settings.** Cleanup policy (`delete`, `compact`), minimum compaction lag, tombstone retention and a maximum message size per topic. See [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration).
- **Larger records.** Records of up to 950 MB, on a topic whose maximum message size is set to allow them. See [limits](/develop/docs/dev-event-broker-reference-limits).
- **Shared VIP pools with mTLS.** A VIP pool can be shared by broker views of several tenants when every one of them uses mTLS (platform team setting).

## What's in this section

- **Start:** this overview and the [quickstart](/develop/docs/dev-event-broker-quickstart): produce and consume with `confluent-kafka`, then read the topic as a table.
- **Concepts:** [objects and lifecycle](/develop/docs/dev-event-broker-concepts-index) and [architecture](/develop/docs/dev-event-broker-concepts-architecture).
- **Guides:** [topics](/develop/docs/dev-event-broker-guides-topics), [producing](/develop/docs/dev-event-broker-guides-producing), [consuming](/develop/docs/dev-event-broker-guides-consuming), [querying topics](/develop/docs/dev-event-broker-guides-querying-topics), [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion) and [S3 bucket events](/develop/docs/dev-event-broker-guides-s3-bucket-events).
- **Reference:** [Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility), [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration) and [limits](/develop/docs/dev-event-broker-reference-limits).
- **Operations:** [setup](/develop/docs/dev-event-broker-operations-setup), [security](/develop/docs/dev-event-broker-operations-security), [monitoring](/develop/docs/dev-event-broker-operations-monitoring) and [CI/CD and automation](/develop/docs/dev-event-broker-operations-ci-cd).
- **Troubleshooting:** [symptom, cause and fix](/develop/docs/dev-event-broker-troubleshooting-index).

## Where to start

1. Get a broker from your platform team: [setup](/develop/docs/dev-event-broker-operations-setup) lists what to ask for.
2. Run the [quickstart](/develop/docs/dev-event-broker-quickstart).
3. Read [concepts](/develop/docs/dev-event-broker-concepts-index) before you design topics and consumer groups.

## See also

- [VAST DataBase overview](/develop/docs/dev-database-overview): the database your topics are stored in.
- [Querying VAST DataBase](/develop/docs/dev-database-guides-querying): how queries on tables, including topic tables, run.
- [Publishing events to VAST Event Broker](/docs-5-5/docs/publishing-events-to-vast-event-broker): the admin guide.