MirrorMaker 2

Applies to VAST Cluster 5.5.

MirrorMaker 2 (MM2) is the Apache Kafka tool that copies topics from one Kafka cluster to another. It runs on the Kafka Connect framework. With VAST Event Broker its main use is migration: you mirror topics from an Apache Kafka cluster into Event Broker, move consumers and then producers across, and switch the old cluster off. It can also keep a copy running in the other direction, from Event Broker to an Apache Kafka cluster. This page gives the settings MM2 needs for each direction, the parts of MM2 that can't work against Event Broker, and the VAST feature to use instead between two VAST clusters.

The examples use MirrorMaker 2 from Apache Kafka 4.1.0 (dedicated mode, connect-mirror-maker.sh), Kafka to Event Broker, unencrypted. They cover topic creation, records with keys, values and headers, the internal topics, consumer group offset sync, running without the idempotence override, and adding a partition on the source. Event Broker to Kafka, SASL/PLAIN and mTLS, and Apache Kafka 3.9 are described from the MM2 documentation and source.

What MM2 needs from the broker

MM2 runs three connectors for each replication flow: one copies the records, one copies consumer group offsets (checkpoints), and one sends heartbeats. They use producers, consumers and admin clients on both clusters. On the Event Broker side, these are the settings that matter:

Setting	Value	Why
`<vast>.producer.enable.idempotence`	`false`	Event Broker doesn't support idempotent producing. Kafka Connect leaves idempotence to the Java client's default for connector producers, and that default is on from Apache Kafka 3.0. See Kafka compatibility. With MM2 4.1.0, the producers that write to Event Broker already have idempotence off without this line, and only the producer that writes to the source cluster has it on. Keep the line anyway, so a different version or default can't turn it on.
`<vast>.producer.compression.type`	`none`	The broker rejects compressed batches with `UNSUPPORTED_COMPRESSION_TYPE`.
`<vast>.exactly.once.source.support`	leave unset (`disabled`)	Exactly-once in MM2 uses transactions, which Event Broker doesn't support.
`sync.topic.acls.enabled`	`false`	There are no Kafka ACLs. `DescribeAcls` returns `SECURITY_DISABLED`, and the calls that create ACLs aren't offered. Access is set with identity policies. See Security.
`sync.topic.configs.enabled`	`false` with Apache Kafka 4.x. With 3.x, see Topic configs.	MM2 4.x changes target topic configs with `IncrementalAlterConfigs`, which Event Broker doesn't offer.
Target topics	Create them on Event Broker first, with their final partition count	The partition count is fixed at creation, and `CreatePartitions` isn't offered, so MM2 can't add partitions to follow the source. See Partitions.

Here <vast> is the alias you give the Event Broker cluster in the MM2 configuration. MM2 applies <alias>.producer.* settings to every producer that writes to that cluster.

All three connectors must also be able to reach every VIP in the Kafka view's VIP pool, and use the view's authentication. See the integrations checklist.

Kafka to Event Broker

This configuration mirrors two topics from an Apache Kafka cluster (alias kafka) into Event Broker (alias vast), keeps their names, and copies the committed offsets of consumer groups. Run it with bin/connect-mirror-maker.sh from an Apache Kafka 3.x or 4.x distribution:

# connect-mirror-maker.properties
clusters = kafka, vast
kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092
vast.bootstrap.servers = <vip-1>:9092,<vip-2>:9092

kafka->vast.enabled = true
vast->kafka.enabled = false
kafka->vast.topics = orders, clicks

# Keep the topic names (the default prefixes them with "kafka.")
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

# Every producer that writes to Event Broker
vast.producer.enable.idempotence = false
vast.producer.compression.type = none

# Features Event Broker doesn't support
sync.topic.acls.enabled = false
sync.topic.configs.enabled = false

# Copy committed offsets of groups that aren't active on Event Broker
sync.group.offsets.enabled = true

For SASL/PLAIN over TLS, add the client settings for the vast alias, for example vast.security.protocol = SASL_SSL, vast.sasl.mechanism = PLAIN, vast.sasl.jaas.config, and a truststore with the CA of the cluster's Kafka certificate. Use port 9093. See Security for what each mode needs.

The IdentityReplicationPolicy keeps names, which is what you want for a migration. Don't use it for a flow in both directions between the same topics: nothing then stops records from being copied back and forth.

Topics that MM2 creates on Event Broker

With Event Broker as the target, MM2 creates these topics on it, through the Admin API:

MM2 creates all of these on Event Broker itself, with CreateTopics, and creates the mirrored topics with the source's partition counts; a compacted source topic arrives compacted. All the records arrive, with their keys, values and headers.

Topic	What it holds
`mm2-configs.kafka.internal`, `mm2-offsets.kafka.internal`, `mm2-status.kafka.internal`	The Kafka Connect configuration, source offsets and status of the MM2 workers for flows from `kafka`. Compacted.
`kafka.checkpoints.internal`	Translated consumer group offsets (checkpoints). Compacted, one partition.
`heartbeats`	Heartbeats, used to check that the flow is alive. Compacted, one partition.

Event Broker supports compaction and ignores the replication factor, so these should work as MM2 creates them. If authorization is on for the view, MM2's user needs KafkaCreate, KafkaWrite, KafkaRead and KafkaDescribe on these topics and on the mirrored ones, and KafkaRead on the MM2 worker group (kafka-mm2 by default). See Security.

Every topic is also a table in kafka_topics. Names with . or - need quoting in SQL.

The MM2 workers coordinate through a group on the target cluster, with the Kafka Connect rebalance protocol. See Kafka Connect for what that means on Event Broker.

Partitions

When a source topic has no counterpart on the target yet, MM2 creates it with the source's partition count. When the source topic later gets more partitions, MM2 asks the target to add them with CreatePartitions. Event Broker doesn't offer that request, so the target keeps its count, and records from the new source partitions have nowhere to go.

So:

Before you start MM2, create each target topic on Event Broker with the partition count it should end up with, and at least as many partitions as the source topic. See Topics. MM2 doesn't create a topic that already exists.
Don't add partitions to a source topic while you mirror it. If you must, create a new topic on both sides.

If a source topic does gain a partition, MM2 can't follow it, and it only warns: WARN Could not create topic-partitions for <topic>. Events written to the new partition are not mirrored: they never reach Event Broker, and the task keeps running. Watch for that warning.

Pre-creating also avoids a second problem: when MM2 creates a topic, it copies the source topic's non-default configs. Event Broker rejects some Apache Kafka configs, such as retention.bytes, with INVALID_CONFIG, and then the topic isn't created. See Topic configuration.

Topic configs

MM2 can keep the target topics' configs in step with the source (sync.topic.configs.enabled, on by default). How it changes them depends on the Apache Kafka version:

MM2 version	How it changes target configs	On Event Broker
4.x	Always with `IncrementalAlterConfigs`. If the target doesn't support it, the connector fails.	Set `sync.topic.configs.enabled = false`.
3.x	`use.incremental.alter.configs`: `requested` (the default) tries `IncrementalAlterConfigs` and falls back to `AlterConfigs`; `required` fails; `never` always uses `AlterConfigs`. The setting is deprecated.	`never` or `requested` should use the `AlterConfigs` call that Event Broker supports. Each sync still fails for a topic whose source configs include one that Event Broker rejects. The safe choice is `sync.topic.configs.enabled = false`.

With config sync off, set each target topic's retention and compaction yourself when you create it. Only some configs take effect on Event Broker anyway. See Topic configuration.

Consumer group offsets

MM2 records, in an offset-syncs topic, how source offsets map to target offsets. The checkpoint connector uses that mapping to translate each group's committed offsets, and writes them to kafka.checkpoints.internal on Event Broker.

With sync.group.offsets.enabled = true (off by default), it also commits the translated offsets to the group on Event Broker, every 60 seconds by default (sync.group.offsets.interval.seconds). It does this only while the group has no active members on Event Broker. So, to move a consumer group:

Let MM2 run with offset sync on until the group's offsets have been copied.
Stop the consumers on the Apache Kafka cluster.
Start them against Event Broker, with the same group.id. They resume from the translated offsets.

MM2 translates offsets only at the points it has recorded, so consumers can re-read some events after the move. Make them tolerate duplicates. For example, a group that had committed offset 318 on one partition and 182 on another came over as 318 and 1: the second partition's offset was translated down to the last recorded point, so the group would re-read those events. A partition the group had never committed stayed uncommitted.

Event Broker allows 256 consumer groups per Kafka view. See Limits. Use groups and groups.exclude to copy only the groups you're moving.

Event Broker to Kafka

To mirror from Event Broker into an Apache Kafka cluster, enable the other flow. The Apache Kafka side is a normal target, so config sync and partition creation work there. What changes is the Event Broker side:

clusters = kafka, vast
kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092
vast.bootstrap.servers = <vip-1>:9092,<vip-2>:9092

vast->kafka.enabled = true
kafka->vast.enabled = false
vast->kafka.topics = orders

# Keep the offset-syncs topic on the Kafka side, so MM2 doesn't write to Event Broker
offset-syncs.topic.location = target

# If MM2 writes anything to Event Broker, these still apply
vast.producer.enable.idempotence = false
vast.producer.compression.type = none

sync.topic.acls.enabled = false

Offset-syncs topic. By default (offset-syncs.topic.location = source), MM2 creates mm2-offset-syncs.kafka.internal (compacted, one partition) on the source cluster, here Event Broker, and produces to it. Set the location to target to keep it on the Apache Kafka side. If you keep it on Event Broker, MM2's user needs permission to create and write to it there.
ACL sync. MM2 reads ACLs from the source. Event Broker answers SECURITY_DISABLED, which MM2 treats as "no ACLs" and logs. Turn the sync off to skip the call.
Topic configs. MM2 reads the source topic configs with DescribeConfigs, which Event Broker supports. Event Broker returns configs it stores but ignores, such as segment.bytes, filled in with defaults. Check which configs arrive on the Apache Kafka topics, and exclude any you don't want with config.properties.exclude.
Partitions. Event Broker topics can't gain partitions, so there's nothing to follow.
Consumer groups. The checkpoint connector lists the groups on Event Broker and reads their committed offsets. A group listing on Event Broker returns at most 256 groups. See Limits.

What won't work

Exactly-once mirroring (exactly.once.source.support = enabled): it needs transactions. Mirroring is at-least-once, so a target topic can hold duplicates after a restart.
Following partition increases on the source: CreatePartitions isn't offered.
ACL sync: there are no Kafka ACLs on Event Broker.
Topic config sync to Event Broker with MM2 4.x: IncrementalAlterConfigs isn't offered.
Bidirectional flows that keep topic names: with IdentityReplicationPolicy, nothing prevents loops.

Between two VAST clusters: use VAST replication

To copy topics from one VAST cluster to another, use VAST asynchronous replication instead of MM2. Topics can be replicated to a replication peer, and protected with snapshots. The replicated topics are read-only at the destination. After a failover, the destination is where events are produced (Data Protection Capabilities for Event Topics).

Know what isn't copied:

Committed consumer group offsets aren't replicated. After a failover, consumer groups have no committed offsets on the new primary, so they start from their auto.offset.reset setting. Record your consumers' positions yourself if they must resume exactly. See Consuming.
The platform team must create the Kafka view at the destination by hand, with a VIP pool of the same name as at the source.

What to ask your platform team: whether the Kafka view's path is replicated, to which peer, and what happens to producers and consumers on failover. The procedures are in the KB, linked from the section above.