--- title: "VAST Event Broker" slug: "dev-event-broker" updated: 2026-11-05T22:16:11Z published: 2026-11-05T22:16:11Z canonical: "docs.vastdata.com/dev-event-broker" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # VAST Event Broker VAST Event Broker is a Kafka-compatible event broker built into VAST Cluster. Start with the Overview and the Quickstart, which create, produce, consume and query one topic in about 10 minutes. Read Concepts before you design topics and consumer groups, then follow the guide for your task: Manage topics, Produce events, Consume events or Query topics as tables. Check Kafka compatibility before you bring an existing Kafka client or tool. ## Overview and concepts Where to begin: the overview and quickstart, then the concepts and architecture the rest of the section relies on, and the release notes. - **[Overview](/develop/docs/dev-event-broker-overview)**: Kafka-compatible event broker built into VAST Cluster for developers using Kafka clients, with each topic stored as a queryable VAST DataBase table. - **[Quickstart](/develop/docs/dev-event-broker-quickstart)**: Create a topic, produce and consume JSON events with confluent-kafka, then read the same topic as a table with the Python SDK. - **[Concepts](/develop/docs/dev-event-broker-concepts-index)**: The Event Broker objects, how each maps to Kafka and to VAST DataBase, and the lifecycle of a topic. - **[Architecture](/develop/docs/dev-event-broker-concepts-architecture)**: How CNodes serve the Kafka protocol, why every virtual IP acts as a broker, and what happens when you produce and consume. ## Guides Step-by-step instructions for the work you'll do most. Start with Manage topics, Produce events and Consume events; read Query topics as tables, Expand JSON events into columns and Publish S3 bucket events to a topic when you use those features. - **[Manage topics](/develop/docs/dev-event-broker-guides-topics)**: Create, inspect, change and delete topics from a Kafka admin client, including partitions, retention and compaction, and what the broker rejects. - **[Produce events](/develop/docs/dev-event-broker-guides-producing)**: Configure a confluent-kafka producer for Event Broker: required settings, acks, key partitioning, headers, batching and the errors it returns. - **[Consume events](/develop/docs/dev-event-broker-guides-consuming)**: Read topics with confluent-kafka consumers: consumer groups, offset commits, rebalancing, direct partition reads, seeking, isolation and group deletion. - **[Query topics as tables](/develop/docs/dev-event-broker-guides-querying-topics)**: Read Kafka-produced events as VAST DataBase tables with the Python SDK, the Query Engine, Trino or Spark, without a consumer. - **[Expand JSON events into columns](/develop/docs/dev-event-broker-guides-blob-expansion)**: Parse JSON event values into typed table columns as they're produced, then change or remove the expansion with the Python SDK. - **[Publish S3 bucket events to a topic](/develop/docs/dev-event-broker-guides-s3-bucket-events)**: Send S3 object create, tag and delete events to a topic with a bucket notification, and consume them with at-least-once delivery. ## Integrations Settings for Kafka tools such as Kafka Connect, MirrorMaker 2, Schema Registry, Flink and Spark. Start with Integrations for the checklist every tool must pass and each tool's status, then read the page for your tool. - **[Integrations](/develop/docs/dev-event-broker-integrations-index)**: The checklist any Kafka tool must pass against Event Broker, each tool's status, and VAST features that replace a tool. - **[Kafka Connect](/develop/docs/dev-event-broker-integrations-kafka-connect)**: Distributed-mode worker and security settings for Kafka Connect on Event Broker, with a Debezium PostgreSQL change-capture example. - **[MirrorMaker 2](/develop/docs/dev-event-broker-integrations-mirrormaker)**: MirrorMaker 2 settings for migrating topics from Apache Kafka into Event Broker, or copying them from Event Broker to Kafka. - **[Schema Registry](/develop/docs/dev-event-broker-integrations-schema-registry)**: Use Avro, Protobuf and JSON Schema serializers with Event Broker, where to store the registry itself, and how encoded values affect queries. - **[Stream processing](/develop/docs/dev-event-broker-integrations-stream-processing)**: Source and sink settings for Flink and Spark Structured Streaming on Event Broker, without idempotence or exactly-once, plus reading topic tables directly. ## Reference Exact broker behaviour to look up. Check Kafka compatibility before you bring an existing client or tool; use Topic configuration and Limits of VAST Event Broker when you set configs or plan topics. - **[Kafka compatibility](/develop/docs/dev-event-broker-reference-kafka-compatibility)**: Supported Kafka clients, listeners, APIs and versions, the Kafka features Event Broker lacks, and what clients see when they use one. - **[Topic configuration](/develop/docs/dev-event-broker-reference-topic-configuration)**: Which Kafka topic configs Event Broker honours, which it stores but ignores, which it rejects, and how partition count works. - **[Limits of VAST Event Broker](/develop/docs/dev-event-broker-reference-limits)**: Topic, partition, message, consumer group, Kafka view and blob expansion limits for Event Broker in VAST Cluster 5.5. ## Operations What to arrange with your platform team, and how to run Event Broker in production. Start with Setup to get a Kafka view and Security for your client settings; read Monitoring and CI/CD and automation as you move toward production. - **[Setup](/develop/docs/dev-event-broker-operations-setup)**: What to request from your platform team for a Kafka view, and a check that Kafka clients and the SDK reach it. - **[Security](/develop/docs/dev-event-broker-operations-security)**: Client settings for unencrypted, TLS, SASL/PLAIN and mTLS connections to a Kafka view, and the permissions to ask for. - **[Monitoring](/develop/docs/dev-event-broker-operations-monitoring)**: Kafka reports your platform team can open in VAST Analytics, and how to check consumer lag yourself from a Kafka client. - **[CI/CD and automation](/develop/docs/dev-event-broker-operations-ci-cd)**: Manage Kafka views, topics and blob expansions from a pipeline with the VMS REST API, vastpy, Terraform and the admin MCP server. ## Troubleshooting Help when a client can't connect, produce or consume. Start with Troubleshooting VAST Event Broker and find your symptom or Kafka error in its tables. - **[Troubleshooting VAST Event Broker](/develop/docs/dev-event-broker-troubleshooting-index)**: Symptoms, causes and fixes for connecting, authentication, producing, topic configs, consuming and querying topic tables in Event Broker.