Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Topic configuration

Prev Next

Applies to VAST Cluster 5.5.

A topic has Kafka topic configs, as in Apache Kafka. VAST Event Broker honours some of them. It accepts and stores others but ignores them, and it rejects the rest. This page lists each config and what the broker does with it. For how to create topics and set configs, see topics.

How the broker treats a config

Status Meaning
Honoured The broker applies the config.
Stored and ignored The broker checks the value's type, stores it and returns it from describe_configs(), but doesn't change its behaviour. A value of the wrong type is rejected with INVALID_CONFIG.
Rejected CreateTopics or AlterConfigs fails with INVALID_CONFIG.

Any config name the broker doesn't know is rejected.

Honoured configs

Config What it does Values Default
retention.ms Time-based retention. Records older than this are deleted in the background. Milliseconds, or -1 for no time limit. Don't rely on less than 6 hours: shorter values are accepted, but see the notes below. 7 days (604800000)
cleanup.policy delete removes records by age. compact keeps only the latest record for each key in a partition. delete, compact, delete,compact, compact,delete (the VAST CLI accepts them too) delete
min.compaction.lag.ms The minimum time a record stays uncompacted. Milliseconds, 0 or more 0
delete.retention.ms How long a tombstone (a record with a key and a null value) is kept after compaction. Milliseconds, 0 or more 1 day (86400000)
max.message.bytes The largest record batch a producer can send to the topic. A larger batch fails with MESSAGE_TOO_LARGE, which confluent-kafka reports as MSG_SIZE_TOO_LARGE ("Broker: Message size too large"). Bytes. In the VAST Web UI this is "Max size", in megabytes. 1048588 (about 1 MB)
message.timestamp.type Which timestamp consumers get with each record. CreateTime returns the producer's timestamp; LogAppendTime returns the time the broker received the record. CreateTime, LogAppendTime CreateTime
message.timestamp.before.max.ms With CreateTime, how far before the broker's time a record's timestamp may be. A record outside the window is rejected with INVALID_TIMESTAMP ("Broker: Invalid timestamp"). Milliseconds, 0 or more No limit (9223372036854775807)
message.timestamp.after.max.ms With CreateTime, how far after the broker's time a record's timestamp may be. A record outside the window is rejected with INVALID_TIMESTAMP. Milliseconds, 0 or more No limit (9223372036854775807)

The defaults are what describe_configs() returns for a new topic on VAST Cluster 5.5.0-SP6. You can see a topic's current values the same way.

Notes on the honoured configs

  • Retention is approximate. A background job deletes expired records, so they stay readable for a while after they pass retention.ms. Don't rely on retention shorter than 6 hours: on a topic with retention.ms of 5 minutes, all its records were still readable 54 minutes after they were written.
  • Retention is based on the broker's timestamp, the time the broker received the record, not the producer's CreateTime.
  • Retention never removes a partition's last record.
  • Compaction runs in the background. Duplicate keys may exist at any given time. With min.compaction.lag.ms and delete.retention.ms of 0, a compacted topic can keep only the latest record for each key about ten minutes after the writes, with a tombstoned key gone altogether.
  • Large messages and compaction. If Max size is set above 5 MB, the topic isn't compacted and database queries on the topic aren't supported. The broker refuses to combine compact with a max.message.bytes above 5 MB, at topic creation or in alter_configs(), with INVALID_CONFIG: 6 MiB and 10 MiB were refused, 5 MiB accepted. It doesn't check the records already in the topic: a topic holding a 15 MB record could be switched to compact with a 4 MiB maximum.
  • Changes apply at once. When you change retention or compaction with alter_configs(), the broker reschedules its background jobs for every partition straight away.
  • validate_only is honoured by CreateTopics and AlterConfigs: nothing is changed. It checks config names and values, but not everything a real request checks: a partition count above the maximum, and compact with a max.message.bytes of 10 MiB, both pass validate_only and fail for real.

Where you can set them

You can set all the honoured configs with the Kafka Admin API (create_topics() and alter_configs()). The platform team can also set them in the VAST Web UI, the VAST CLI (topic create and topic modify) or the VMS REST API, except that the CLI has no option for the maximum message size. Use the non-incremental alter_configs(): the broker doesn't support incremental_alter_configs(), which fails with _UNSUPPORTED_FEATURE. See Kafka compatibility.

Unlike Apache Kafka, alter_configs() changes only the configs you send: the others keep their values. For example, setting only retention.ms on a compacted topic leaves cleanup.policy=compact in place.

The VAST CLI offers only the honoured configs, and so do the Web UI and REST API. The ignored configs below can be set only through the Kafka Admin API.

Stored and ignored configs

The broker accepts these configs, type-checks their values and returns what you set from describe_configs(), but they have no effect.

Config Why it has no effect
compression.type, compression.gzip.level, compression.lz4.level, compression.zstd.level The broker rejects compressed batches from producers, whatever the topic says. VAST compresses the stored data itself. Setting compression.type on the topic doesn't let producers compress.
min.insync.replicas, unclean.leader.election.enable, follower.replication.throttled.replicas, leader.replication.throttled.replicas There's no Kafka-level replication. Durability comes from VAST storage.
segment.bytes, segment.ms, segment.jitter.ms, segment.index.bytes, index.interval.bytes, preallocate, file.delete.delay.ms, flush.messages, flush.ms Topics are VAST DataBase tables, not log segment files.
min.cleanable.dirty.ratio Compaction is scheduled by min.compaction.lag.ms.
max.compaction.lag.ms Accepted. Whether it affects compaction isn't confirmed.
message.timestamp.difference.max.ms Use message.timestamp.before.max.ms and message.timestamp.after.max.ms.
message.format.version, message.downconversion.enable The broker uses message format v2 only. message.format.version accepts only 3.0-IV1, the value describe_configs() returns; other values such as 3.0 or 2.8 are rejected with INVALID_CONFIG.
local.retention.ms, local.retention.bytes, remote.storage.enable, remote.log.copy.disable, remote.log.delete.on.disable There's no tiered storage.

describe_configs() returns every config in this table along with the honoured ones, 35 in all, filled in with defaults, and it reports none of them as read-only. Don't read a returned value as a sign that the broker uses it.

Rejected configs

Config What happens What to do
retention.bytes INVALID_CONFIG ("Broker: Configuration is invalid"), in create_topics() and in alter_configs(). There's no size-based retention. Leave it out. Use retention.ms.
Any other config name INVALID_CONFIG Leave it out.
Broker configs (resource type BROKER) describe_configs() returns none, and a change is rejected with INVALID_REQUEST. None. There are no broker configs to change.

Some tools and frameworks set retention.bytes or other configs when they create topics. If topic creation fails with INVALID_CONFIG, check which configs the tool sends.

The partition count

The partition count isn't a config, but it's set at the same time. Pass it explicitly when you create a topic: a count of -1 (the broker default, and the default in confluent-kafka's NewTopic) or 0 is rejected with INVALID_PARTITIONS. You can't change it after the topic is created. For the maximum, see limits.

See also

Related articles