Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Overview

Prev Next

Applies to VAST Cluster 5.5.

VAST Event Broker is a Kafka-compatible event broker built into VAST Cluster. Kafka clients produce to it and consume from it over the Kafka protocol. Each topic is stored as a VAST DataBase table, so you can also read the same events with the VAST DataBase SDK, the VAST Query Engine, Trino or Spark, without copying them anywhere first.

What it is

  • A protocol on the cluster, not a separate system. The Kafka protocol is served by the cluster's CNodes, on the virtual IPs of a VIP pool. There are no broker machines to run.
  • A broker is a view. Your platform team enables the Kafka protocol on a view, and that view is your broker. Clients connect to the virtual IPs of the VIP pool assigned to it (KB).
  • A topic is a table. Topics live in a schema called kafka_topics in the view's bucket. Each topic shows up as a table, so the topic name is the table name.
  • Kafka-compatible. Producer, consumer, consumer-group and Kafka Admin APIs work, with the gaps listed in Kafka compatibility. Supported clients include the Confluent Kafka clients (Python, Java, .NET, C and Rust), Sarama and franz-go; see supported clients.

See concepts for the objects, and architecture for how the cluster serves them.

How it differs from running Apache Kafka

VAST Event Broker runs on the cluster's DASE architecture, and every topic is a VAST DataBase table. That removes much of the work of running Kafka yourself:

Two panels. Apache Kafka: brokers each store their partitions, and replicas of other brokers' partitions, on local disks, coordinated by a controller quorum, and a connector job copies topics into a separate table for analytics. VAST Event Broker: Kafka clients connect to the virtual IPs of a Kafka view; CNodes serve the protocol; every topic is a table on the cluster's shared storage, which SQL engines read directly. There are no replicas to copy and no connector.

With VAST Event Broker, the topic is the table: no replicas to manage and no connector to copy events into a lake.

With Apache Kafka you... With VAST Event Broker
Run broker servers, each storing its partitions on local disks There are no broker servers. The cluster's CNodes serve the Kafka protocol, and topics are stored on the cluster's shared storage.
Set a replication factor, and watch for under-replicated partitions replication_factor is accepted and ignored. Durability comes from the cluster's storage, which is protected by erasure coding, not from copies on several brokers.
Reassign partitions, copying their data, when you add brokers or one fails No partition data belongs to a broker. A partition's leader is only the virtual IP clients send to, and a virtual IP moves to another CNode if its CNode goes offline.
Size each broker's disks for its share of retention Topics use the cluster's capacity, like any other data in the view's bucket. Capacity and processing grow separately.
Run ZooKeeper or a KRaft controller quorum There's no separate Kafka cluster, so there's no controller quorum to run.
Run Kafka Connect, or a stream job, to copy topics into a lake or warehouse table A topic is a table. Query it with the VAST DataBase SDK, the VAST Query Engine, Trino or Spark as soon as the produce commits.
Run a stream-processing job to parse JSON events into columns Blob expansion parses JSON into a table in the same transaction as the produce.

The trade-off is Kafka compatibility: some client features aren't supported, such as idempotent and transactional producers and compressed batches. See use cases and considerations and Kafka compatibility.

How it fits with VAST DataBase

Kafka producers send events to a Kafka view, which is your broker, through the virtual IPs of its VIP pool; the cluster's CNodes serve the Kafka protocol. The cluster can also publish S3 bucket events into a topic. Each topic is a table, with partitions, in the kafka_topics schema of the view's bucket, and the cluster keeps committed offsets per consumer group. Kafka consumer groups consume the events in order, as they arrive. The VAST DataBase SDK, the VAST Query Engine, Trino or Spark query the same topic as a table, and blob expansion parses JSON event values into typed columns of a separate table, in the same transaction as the produce. Other Kafka views, for example on other tenants, are separate brokers with their own topics. Topic tables are read-only through the VAST DataBase API.

Producers write to a Kafka view; consumer groups, SQL engines and blob expansion all use the same topic tables.

Events you produce are rows in a VAST DataBase table as soon as the produce request commits. That gives you two ways to read the same data:

You want to Use
Process events in order, as they arrive, with committed offsets per consumer group A Kafka consumer. See consuming.
Query events with filters, joins and aggregations, over any time range still retained The VAST DataBase SDK, the VAST Query Engine, Trino or Spark. See querying topics.
Turn JSON event payloads into typed columns of an ordinary table, as they arrive Blob expansion (new in 5.5). See blob expansion.

Topic tables are read-only through the VAST DataBase API: you write events with a Kafka producer, not with SQL or the SDK.

The cluster can also publish S3 bucket notifications (object created, removed or tagged) into a topic. See S3 bucket events.

Use cases and considerations

Example use cases:

  • Queryable Kafka events: you already produce or consume with Kafka clients and want the events queryable on the same cluster as your other data.
  • Analytics on a stream without a connector job that copies topics into tables.
  • S3 object events from a bucket on the cluster as a Kafka stream.
  • JSON events parsed into a typed table in the same transaction as the write.

Important to note:

These aren't supported. For the details, see Kafka compatibility and limits.

  • Transactions and idempotent producers. If your pipeline needs exactly-once delivery from the broker, VAST Event Broker doesn't provide it. Producers must turn idempotence off.
  • Over-the-wire compression. Producers must send uncompressed batches. (The cluster compresses data itself.)
  • Automatic topic creation. Create topics explicitly.
  • Changing the partition count. The number of partitions is fixed when you create the topic.
  • Some consumer features: cooperative rebalancing, READ UNCOMMITTED, fetch sessions and rack awareness. Static group membership (group.instance.id) is accepted but ignored.
  • More than 256 consumer groups per broker view.
  • Protocol auditing and QoS. Producing, consuming and topic management aren't covered by VAST protocol auditing or Quality of Service.
  • Replicating consumer groups. Topics can be replicated, but committed consumer-group offsets aren't.

What's new in 5.5

  • mTLS authentication. Clients can authenticate with a client certificate over TLS. See security.
  • Blob expansion. JSON in the event value can be parsed into columns of a separate VAST DataBase table, in the same transaction as the event. See blob expansion.
  • More partitions. Up to 20,000 partitions per topic, and 200,000 per broker view.
  • More topic settings. Cleanup policy (delete, compact), minimum compaction lag, tombstone retention and a maximum message size per topic. See topic configuration.
  • Larger records. Records of up to 950 MB, on a topic whose maximum message size is set to allow them. See limits.
  • Shared VIP pools with mTLS. A VIP pool can be shared by broker views of several tenants when every one of them uses mTLS (platform team setting).

What's in this section

Where to start

  1. Get a broker from your platform team: setup lists what to ask for.
  2. Run the quickstart.
  3. Read concepts before you design topics and consumer groups.

See also

Related articles