Architecture

Applies to VAST Cluster 5.5.

This page explains how VAST Event Broker is built on the cluster: where the Kafka protocol is served, how clients find partitions and group coordinators, and what happens when you produce and consume. It explains behaviour you'll see from your clients. You don't need it to get started, but read it before you tune producers and consumers or debug client errors.

Built on DASE

VAST Event Broker isn't a Kafka cluster deployed on VAST. It's the Kafka protocol, served by the cluster's CNodes, over topics stored as VAST DataBase tables on the cluster's shared storage. The cluster's DASE architecture is why:

Each partition is led by a virtual IP. A partition's leader is one of the view's virtual IPs, the Kafka equivalent of a broker. Each virtual IP is on one CNode at a time, so exactly one CNode reads and writes a partition at any moment. When a virtual IP moves, another CNode takes the partition over, without copying data, because every CNode reaches every SSD. Queries that read the topic as a table can read it from several CNodes at once.
No replicas. Durability comes from the cluster's erasure-coded storage, so there are no follower replicas, no in-sync replica set and no replication factor to tune.
Failures move addresses, not data. If a CNode goes offline, its virtual IPs move to other CNodes, and clients carry on against the same addresses. No partition data has to be copied.
Throughput and capacity scale separately. More CNodes serve more connections and requests; more storage enclosures hold more retained events.

Every virtual IP is a broker, and every CNode reaches every partition.

Where the protocol runs

The Kafka protocol is served by the cluster's CNodes, in the same process that serves the cluster's other data protocols. There's no separate broker process or JVM.

Every virtual IP in a Kafka view's VIP pool listens for Kafka clients on two ports: 9092 for unencrypted connections and 9093 for TLS. Which of the two a view accepts, and how clients authenticate on each, is set per view by your platform team. See security.

When a connection arrives, the cluster finds the Kafka view from the VIP pool the address belongs to. That's how one cluster runs several independent brokers: each Kafka view has its own VIP pool.

Every VIP is a broker

Clients bootstrap from a few of the pool's virtual IPs (three are recommended). The metadata the cluster returns lists every virtual IP in the pool as a broker, with its IP address (not a host name) and the port the client connected on. So:

Your clients must be able to reach every virtual IP in the pool, not only the bootstrap addresses.
Clients see IP addresses in metadata, and TLS certificates for the broker carry the virtual IPs, not host names.

Partition leaders and group coordinators

Each partition has a leader, and each consumer group has a coordinator, like in Apache Kafka. The cluster picks them by hashing the topic name and partition number (or the group name) onto the sorted list of virtual IPs in the pool. The result is stable while the pool doesn't change, and spreads partitions and groups across the virtual IPs.

There are no follower replicas. Metadata reports the leader as the only replica and the only in-sync replica. All CNodes read and write the same shared storage, so leadership only tells clients which address to use. It doesn't mean a CNode owns the partition's data.

If a client sends a fetch to a virtual IP that isn't the partition's leader, it gets NOT_LEADER_OR_FOLLOWER. A group request to the wrong virtual IP gets NOT_COORDINATOR. Kafka clients handle both by refreshing metadata and retrying. You may also see them briefly while your platform team changes the VIP pool.

The replication_factor you pass when you create a topic is accepted and ignored. Durability comes from the cluster's storage, not from copies on several brokers.

Topics are tables

A topic is a table in the kafka_topics schema of the Kafka view's bucket. Each partition is stored separately inside it, and each event is one row. The cluster uses the row's ID in its partition as the event's offset. See concepts for the columns.

Because a topic is a table, you can read it with the VAST DataBase SDK, the VAST Query Engine, Trino or Spark. Those reads are read-only. The Query Engine reads a topic partition by partition, in parallel. See querying topics.

Producing: one transaction per request

A Kafka producer sends a produce request that can hold batches for several partitions and topics. The cluster writes the whole request in one VAST DataBase transaction, across all its partitions.

One produce request, one transaction: all of its partitions commit or none do.

If any batch fails to write, the transaction rolls back, and every partition in the response gets the error. Errors found while checking a single batch (an unknown topic, a refused permission, an oversized or compressed batch) are reported for that partition only.
With acks=1 and acks=all the response is sent after the transaction commits. The two behave the same, because there are no follower replicas to wait for. With acks=0 no response is sent.
Within a partition, events get increasing offsets in the order they're written. Offsets can have gaps: a rolled-back request uses up offsets, and compaction removes events.
If the topic has a blob expansion, the parsed rows are written to its target table in the same transaction. They commit or roll back with the events.

The producer must send uncompressed batches, and must have idempotence turned off. See producing.

Consuming: committed events only

Consumers only ever see committed events. A fetch waits for produce requests that started before it to finish, and then reads up to the last committed event (the last stable offset). The high watermark a consumer sees is the same offset. An isolation level of READ UNCOMMITTED is accepted but treated as READ COMMITTED.

Things that differ from Apache Kafka:

Every fetch is a full fetch. The cluster doesn't create fetch sessions, so clients send the full partition list each time. Clients handle this automatically.
An empty fetch returns at once. The cluster doesn't hold a fetch open to wait for new data, and ignores the minimum-bytes setting. A consumer that's caught up polls more often than it would against Apache Kafka. Check monitoring if that matters for your load.

Consumer groups

Consumer groups use the classic group protocol: consumers join through the group's coordinator, the group leader assigns partitions in the client, and the coordinator distributes the assignment. Cooperative (incremental) rebalancing isn't supported. Static membership is accepted but ignored: a consumer with group.instance.id joins as an ordinary member. The newer consumer protocol (KIP-848) isn't supported.

The cluster stores the offsets that consumers commit. See consuming.

Background work

Two kinds of work run in the background, per partition:

Retention deletes events older than the topic's retention period, measured from the cluster's timestamp. It runs as scheduled jobs, so events stay readable for a while after they expire.
Compaction, on topics with cleanup.policy=compact, keeps the latest event per key and removes tombstones after their retention period.

Changing a topic's retention or compaction settings reschedules this work straight away.