---
title: "Concepts"
slug: "dev-event-broker-concepts-index"
description: "Explore VAST Event Broker's objects, Kafka mapping, topic lifecycle, and consumer groups to optimize your data management and event processing strategies."
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-event-broker-concepts-index"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Concepts

*Applies to VAST Cluster 5.5.*

This page names the objects you work with in VAST Event Broker, says how each maps to Kafka and to VAST DataBase, and walks through the life of a topic. Read it before you design topics and consumer groups.

## Objects

| Object | What it is | Kafka equivalent |
| --- | --- | --- |
| Kafka view | A view on which your platform team has enabled the Kafka protocol. It's the broker: it holds your topics and consumer groups. Enabling Kafka also enables the S3 and Database protocols on the view, and the view has an S3 bucket. | A Kafka cluster |
| VIP pool | The virtual IPs that clients connect to. A Kafka view has exactly one VIP pool, with at least one virtual IP per CNode. Every virtual IP in the pool acts as a broker. | The brokers |
| Bucket and `kafka_topics` schema | The view's bucket is a VAST DataBase database. Topics are tables in its `kafka_topics` schema. | — |
| Topic | A named stream of events, stored as one VAST DataBase table. | A topic |
| Partition | A slice of a topic. Events in one partition have an order; the partition count is fixed when you create the topic, up to 20,000. Each partition is stored separately inside the topic's table. | A partition |
| Event | One Kafka record: key, value, headers and timestamps. It's one row of the topic table. | A record or message |
| Offset | An event's position in its partition. The cluster uses the row's ID in the partition as its offset. | An offset |
| Consumer group | A named set of consumers that share the partitions of the topics they read, and the offsets they've committed. Up to 256 per Kafka view. | A consumer group |
| Retention | How long events are kept, per topic. The default is seven days, the minimum is 6 hours, and `-1` keeps events forever. | `retention.ms` |
| Compaction | Per topic, keeps only the latest event for each key. Runs in the background. | `cleanup.policy=compact` |
| Blob expansion target table | An ordinary VAST DataBase table that receives the fields of JSON event values, parsed as each event is written (5.5). | — (no Kafka equivalent) |

### Topic table columns

Each event becomes one row with these columns:

| Column | Type | Holds |
| --- | --- | --- |
| `key` | binary | The record key, or null |
| `value` | binary | The record value, or null |
| `timestamp_broker` | timestamp (ms) | When the cluster received the event |
| `timestamp_producer` | timestamp (ms) | The timestamp the producer set |
| `headers` | list of (`key` string, `value` binary) | The record headers |
| `external_offset`, `compaction_process_seq_num`, `extra_metadata` | int64, int64, binary | Used by the cluster. Empty (null) for events produced by Kafka clients. |

There's no partition or offset column you can query. An event's offset is its row ID in its partition. Key and value are bytes: decode them yourself, or use [blob expansion](/develop/docs/dev-event-broker-guides-blob-expansion) to get typed columns. See [querying topics](/develop/docs/dev-event-broker-guides-querying-topics).

### Timestamps

Every event keeps both timestamps. The topic's timestamp type decides which one consumers get: the producer's (`CreateTime`, the default) or the cluster's (`LogAppendTime`). With `CreateTime`, a topic can also reject events whose timestamp is too far before or after the cluster's time. See [topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration).

## How the objects relate

![A Kafka view is the broker. It has one VIP pool, which clients connect to, and one bucket, which is a VAST DataBase database. Topics are tables in the bucket's kafka_topics schema. A topic has a fixed number of partitions, and each partition holds events in offset order; each event is one row. Consumer groups belong to the view, and the cluster keeps each group's committed offset per partition. A blob expansion writes the JSON fields of a topic's events into a target table, an ordinary table in a schema of your own.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/eb-object-model.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A37Z&se=2026-11-08T09%3A55%3A37Z&sr=c&sp=r&sig=mpYzqDlZXKKfZNXtlxdyokF%2BToyZPAuivGpZBl0xugQ%3D)

*A Kafka view holds its topics, as tables, and its consumer groups.*

- One Kafka view has one VIP pool, one bucket and one `kafka_topics` schema.
- A topic belongs to one Kafka view. Two Kafka views are two separate brokers, with separate topics and consumer groups.
- A consumer group belongs to one Kafka view. Its committed offsets are kept by the cluster, for that view only.
- A topic table can have one blob expansion per source column (only `value` is supported). A target table can be the target of only one expansion.

## Lifecycle of a topic

1. **Create.** With the Kafka Admin API (`CreateTopics`), or by your platform team in the VAST Web UI, CLI or REST API. Give an explicit partition count. You can't create a topic through the VAST DataBase SDK, and producing to a missing topic doesn't create it.  See [topics](/develop/docs/dev-event-broker-guides-topics).
2. **(Optional) Add a blob expansion**, before you produce the events you want parsed. Events already in the topic aren't expanded.
3. **Produce.** Each produce request is written in one VAST DataBase transaction. When the producer gets its acknowledgement, the events are committed.  See [producing](/develop/docs/dev-event-broker-guides-producing).
4. **Consume and query.** Consumers and queries see committed events only. See [consuming](/develop/docs/dev-event-broker-guides-consuming) and [querying topics](/develop/docs/dev-event-broker-guides-querying-topics).
5. **Expire and compact.** In the background, the cluster deletes events older than the retention period and, on compacted topics, older events for each key.
6. **Change settings.** Retention, compaction and timestamp settings can change; the partition count can't.
7. **Delete.** With the Kafka Admin API (`DeleteTopics`) or by your platform team. Dropping the topic's table through the VAST DataBase API also deletes the topic. Deleting a topic also deletes the offsets that consumer groups committed for it.

### Retention

Retention is time-based only: there's no size limit per topic. The age of an event is measured from the cluster's timestamp (`timestamp_broker`), not the producer's.

Deletion is approximate. Expired events stay readable for a while after they pass the retention period, until a background job removes them. The newest committed event in a partition isn't removed by retention. Don't use retention as a precise cut-off: filter by timestamp when you query.

### Compaction

With compaction on, the cluster keeps only the latest event for each key in a partition, and deletes older versions. Compaction runs in the background, so duplicate keys can exist for a while. An event with a null value is a tombstone: it marks the key as deleted, and is itself removed after the topic's tombstone retention period.

Offsets don't change when compaction removes events, so a compacted partition has gaps in its offsets. A topic whose maximum message size is set above 5 MB isn't compacted, and can't be queried as a table.

### Consumer groups

- A group comes into being when a consumer joins it, or when a consumer commits an offset to it.
- Committed offsets don't expire while the group is in use. They're removed when you delete the group (Kafka Admin API `DeleteGroups`, which needs the group to be empty), when you delete the topic, or when the cluster deletes the group for inactivity (below).
- The cluster cleans up the membership of a group that has had no members for a long time, and deletes groups that have been inactive for several days, because some clients never delete their groups. Delete groups you no longer need yourself.
- Committed offsets aren't replicated. After a failover to a replica, consumer groups have no committed offsets, and consumers start from their `auto.offset.reset` position.

See [consuming](/develop/docs/dev-event-broker-guides-consuming).

## See also

- [Architecture](/develop/docs/dev-event-broker-concepts-architecture): how the cluster serves topics, partitions and groups.
- [Topics](/develop/docs/dev-event-broker-guides-topics): create, configure and delete topics.
- [Limits](/develop/docs/dev-event-broker-reference-limits): partitions, groups and message sizes.
- [VAST DataBase concepts](/develop/docs/dev-database-concepts-index): buckets, schemas, tables and transactions.