Concepts

Applies to VAST Cluster 5.5.

This page names the objects you work with in VAST Event Broker, says how each maps to Kafka and to VAST DataBase, and walks through the life of a topic. Read it before you design topics and consumer groups.

Objects

Object	What it is	Kafka equivalent
Kafka view	A view on which your platform team has enabled the Kafka protocol. It's the broker: it holds your topics and consumer groups. Enabling Kafka also enables the S3 and Database protocols on the view, and the view has an S3 bucket.	A Kafka cluster
VIP pool	The virtual IPs that clients connect to. A Kafka view has exactly one VIP pool, with at least one virtual IP per CNode. Every virtual IP in the pool acts as a broker.	The brokers
Bucket and `kafka_topics` schema	The view's bucket is a VAST DataBase database. Topics are tables in its `kafka_topics` schema.	—
Topic	A named stream of events, stored as one VAST DataBase table.	A topic
Partition	A slice of a topic. Events in one partition have an order; the partition count is fixed when you create the topic, up to 20,000. Each partition is stored separately inside the topic's table.	A partition
Event	One Kafka record: key, value, headers and timestamps. It's one row of the topic table.	A record or message
Offset	An event's position in its partition. The cluster uses the row's ID in the partition as its offset.	An offset
Consumer group	A named set of consumers that share the partitions of the topics they read, and the offsets they've committed. Up to 256 per Kafka view.	A consumer group
Retention	How long events are kept, per topic. The default is seven days, the minimum is 6 hours, and `-1` keeps events forever.	`retention.ms`
Compaction	Per topic, keeps only the latest event for each key. Runs in the background.	`cleanup.policy=compact`
Blob expansion target table	An ordinary VAST DataBase table that receives the fields of JSON event values, parsed as each event is written (5.5).	— (no Kafka equivalent)

Topic table columns

Each event becomes one row with these columns:

Column	Type	Holds
`key`	binary	The record key, or null
`value`	binary	The record value, or null
`timestamp_broker`	timestamp (ms)	When the cluster received the event
`timestamp_producer`	timestamp (ms)	The timestamp the producer set
`headers`	list of (`key` string, `value` binary)	The record headers
`external_offset`, `compaction_process_seq_num`, `extra_metadata`	int64, int64, binary	Used by the cluster. Empty (null) for events produced by Kafka clients.

There's no partition or offset column you can query. An event's offset is its row ID in its partition. Key and value are bytes: decode them yourself, or use blob expansion to get typed columns. See querying topics.

Timestamps

Every event keeps both timestamps. The topic's timestamp type decides which one consumers get: the producer's (CreateTime, the default) or the cluster's (LogAppendTime). With CreateTime, a topic can also reject events whose timestamp is too far before or after the cluster's time. See topic configuration.

How the objects relate

A Kafka view holds its topics, as tables, and its consumer groups.

One Kafka view has one VIP pool, one bucket and one kafka_topics schema.
A topic belongs to one Kafka view. Two Kafka views are two separate brokers, with separate topics and consumer groups.
A consumer group belongs to one Kafka view. Its committed offsets are kept by the cluster, for that view only.
A topic table can have one blob expansion per source column (only value is supported). A target table can be the target of only one expansion.

Lifecycle of a topic

Create. With the Kafka Admin API (CreateTopics), or by your platform team in the VAST Web UI, CLI or REST API. Give an explicit partition count. You can't create a topic through the VAST DataBase SDK, and producing to a missing topic doesn't create it. See topics.
(Optional) Add a blob expansion, before you produce the events you want parsed. Events already in the topic aren't expanded.
Produce. Each produce request is written in one VAST DataBase transaction. When the producer gets its acknowledgement, the events are committed. See producing.
Consume and query. Consumers and queries see committed events only. See consuming and querying topics.
Expire and compact. In the background, the cluster deletes events older than the retention period and, on compacted topics, older events for each key.
Change settings. Retention, compaction and timestamp settings can change; the partition count can't.
Delete. With the Kafka Admin API (DeleteTopics) or by your platform team. Dropping the topic's table through the VAST DataBase API also deletes the topic. Deleting a topic also deletes the offsets that consumer groups committed for it.

Retention

Retention is time-based only: there's no size limit per topic. The age of an event is measured from the cluster's timestamp (timestamp_broker), not the producer's.

Deletion is approximate. Expired events stay readable for a while after they pass the retention period, until a background job removes them. The newest committed event in a partition isn't removed by retention. Don't use retention as a precise cut-off: filter by timestamp when you query.

Compaction

With compaction on, the cluster keeps only the latest event for each key in a partition, and deletes older versions. Compaction runs in the background, so duplicate keys can exist for a while. An event with a null value is a tombstone: it marks the key as deleted, and is itself removed after the topic's tombstone retention period.

Offsets don't change when compaction removes events, so a compacted partition has gaps in its offsets. A topic whose maximum message size is set above 5 MB isn't compacted, and can't be queried as a table.

Consumer groups

A group comes into being when a consumer joins it, or when a consumer commits an offset to it.
Committed offsets don't expire while the group is in use. They're removed when you delete the group (Kafka Admin API DeleteGroups, which needs the group to be empty), when you delete the topic, or when the cluster deletes the group for inactivity (below).
The cluster cleans up the membership of a group that has had no members for a long time, and deletes groups that have been inactive for several days, because some clients never delete their groups. Delete groups you no longer need yourself.
Committed offsets aren't replicated. After a failover to a replica, consumer groups have no committed offsets, and consumers start from their auto.offset.reset position.

See consuming.