Security

Applies to VAST Cluster 5.5.

Security in VAST Event Broker is set per Kafka view by your platform team: which connections the view accepts (unencrypted, TLS or both), how clients authenticate on each (none, SASL/PLAIN or mTLS), and whether identity policies decide what each user may do. This page explains what each choice means for your clients, gives the client settings for each mode, and lists the permissions to ask for.

Connections and authentication

A Kafka view can accept two kinds of connection, each on its own port, and sets an authentication method for each:

Connection	Port	Authentication methods
Unencrypted	9092	None, or SASL/PLAIN
TLS	9093	None, SASL/PLAIN, or mTLS (5.5)

A view can allow one kind or both. Each kind has exactly one method, so on TLS a client uses SASL/PLAIN or mTLS, not both. If a client uses a kind the view doesn't allow, the broker accepts the TCP connection and closes it at the first request, before it answers even the version request.

If your platform team sets nothing, a new Kafka view accepts unencrypted connections without authentication.

TLS

TLS uses one Kafka certificate for the whole cluster. The cluster gives clients IP addresses, not host names, so the certificate lists every virtual IP of every Kafka view's VIP pool as an IP address in its Subject Alternative Names. It's issued by an internal CA, because public CAs don't issue certificates for private IP addresses. Your client needs that CA's certificate: without it, the TLS handshake fails. Hostname verification, which confluent-kafka does by default, passes against the IP-address entries.

The broker negotiates TLS 1.2. A client that allows only TLS 1.3 fails the handshake.

To look at the certificate the broker presents:

openssl s_client -connect <vip>:9093 -showcerts

If your platform team adds another Kafka view later, they reissue the certificate with its virtual IPs. Clients keep working as long as they trust the same CA.

SASL/PLAIN

SASL/PLAIN works on both unencrypted and TLS connections. The user name and password are those of a VAST local user in the view's tenant. Active Directory and LDAP users aren't supported. S3 access keys don't work here.

Use SASL/PLAIN over TLS: on an unencrypted connection, the password crosses the network in clear text.

Things to know:

PLAIN is the only SASL mechanism. The broker answers a request for SCRAM, GSSAPI (Kerberos) or OAUTHBEARER by listing PLAIN as its only mechanism, then closes the connection. confluent-kafka logs SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure.
When the view requires SASL/PLAIN, the broker answers only the version and SASL handshake requests before authentication. Any other request closes the connection, with no error code. confluent-kafka logs Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication.
When the view doesn't use SASL on that connection kind, a SASL handshake gets ILLEGAL_SASL_STATE. confluent-kafka reports it as a broker transport failure.
A wrong password, or a user name that isn't a local user, gets SASL_AUTHENTICATION_FAILED with no message. confluent-kafka logs SASL authentication error: followed by nothing, and requests fail with _TRANSPORT.
If the client sets a SASL authorization identity, it must be the user name. Otherwise the broker answers Authentication failed: SASL PLAIN authorization identity (authzid) must match authentication identity (authcid).
There's no re-authentication: a connection stays authenticated until it closes.

mTLS

With mTLS (5.5), the client authenticates with its own certificate on a TLS connection. Your platform team decides which CA to trust: one for the whole cluster, or up to two per tenant (a third is refused with "We cannot have more than 2 per tenant TLS certificates for protocol KAFKA.").

The certificate's common name (CN) is looked up as a VAST user name, and that user is who the connection acts as for permissions.

What happens to a connection without a usable certificate:

Client certificate	What happens
Signed by a CA the cluster doesn't trust	The TLS handshake fails with `unknown ca`.
None	The TLS handshake completes, and the broker closes the connection at the first request.
CN isn't a VAST user	The same: the broker closes the connection at the first request.
Revoked in a revocation list your platform team uploaded for the CA	The TLS handshake fails with `certificate revoked`.

With no certificate, or a CN that isn't a user, confluent-kafka logs Disconnected while requesting ApiVersion: might be caused by broker version < 0.10.

Revocation through CRL distribution points in certificates isn't supported. The platform team uploads a revocation list instead.

mTLS uses TLS 1.2, like the other TLS connections, so leave TLS 1.2 enabled in your client.

With mTLS, the broker views of several tenants can share one VIP pool. The cluster tells the tenants apart by the client certificate.

Client settings

These are confluent-kafka (librdkafka) settings. Add them to the configuration of every producer, consumer and admin client. Bootstrap addresses use the port of the connection kind: 9092 or 9093. Each configuration below was used to produce, consume and run admin calls against a view in that mode.

No authentication, unencrypted:

conf = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],   # "<vip>:9092,..."
    "security.protocol": "PLAINTEXT",
}

SASL/PLAIN over TLS (recommended for SASL):

conf = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],   # "<vip>:9093,..."
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "PLAIN",
    "sasl.username": os.environ["KAFKA_USERNAME"],
    "sasl.password": os.environ["KAFKA_PASSWORD"],
    "ssl.ca.location": "/path/to/kafka-ca.pem",
}

SASL/PLAIN, unencrypted: as above, with "security.protocol": "SASL_PLAINTEXT", port 9092 and no ssl.ca.location.

TLS without client authentication:

conf = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],   # "<vip>:9093,..."
    "security.protocol": "SSL",
    "ssl.ca.location": "/path/to/kafka-ca.pem",
}

mTLS:

conf = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],   # "<vip>:9093,..."
    "security.protocol": "SSL",
    "ssl.ca.location": "/path/to/kafka-ca.pem",           # CA of the cluster's Kafka certificate
    "ssl.certificate.location": "/path/to/client.pem",    # your certificate, CN = your VAST user
    "ssl.key.location": "/path/to/client.key",
}

Permissions

When permissions are checked

Permissions are checked only when your platform team turns on authorization for the Kafka view. Authorization needs an authentication method on at least one connection kind, so that the cluster knows who the user is. The VAST CLI refuses it otherwise, with At least one of the authentication mechanisms must be allowed in order to enable authorization for the KAFKA protocol.

Authorization off: every client that can connect can do everything: produce, consume, and create or delete topics and groups. This includes users with no identity policy at all.
Authorization on: only operations that the user's identity policy explicitly allows are performed. An operation that no statement matches is refused, and so is every operation on a connection that hasn't authenticated, for example an unencrypted connection without authentication on a view that uses mTLS on TLS. A Deny statement overrides an Allow.

The cluster evaluates the identity policies attached to the user and its groups. Bucket policies on the Kafka view's bucket don't apply to Kafka operations.

Actions

Kafka actions are written without a prefix, for example "Action": "KafkaRead". Each row below was checked with authorization on, by granting and removing the action:

Action	Allows
`KafkaWrite` on a topic	Producing to the topic
`KafkaRead` on a topic	Fetching from the topic
`KafkaRead` on a consumer group	Joining the group, heartbeats, and committing and fetching its offsets
`KafkaDescribe` on a topic	Seeing the topic in metadata, and looking up its offsets. Without it the topic is left out of metadata, so clients can't produce to it or consume from it: they get `TOPIC_AUTHORIZATION_FAILED`.
`KafkaDescribe` on a consumer group	Listing and describing the group. Without it the group is left out of group lists.
`KafkaCreate` on the cluster and on the new topic	Creating the topic. Both are needed.
`KafkaDelete` on a topic or group	Deleting the topic, or the consumer group
`KafkaDescribeConfigs` on the cluster, and `KafkaCreate` on the topic	Reading the topic's configuration (`describe_configs()`). Without `KafkaDescribeConfigs` you get `CLUSTER_AUTHORIZATION_FAILED`; without `KafkaCreate` on the topic, `TOPIC_AUTHORIZATION_FAILED`.
`KafkaAlterConfigs` on the cluster and on the topic	Changing the topic's configuration (`alter_configs()`). Both are needed.

So a consumer typically needs KafkaRead and KafkaDescribe on its topics, and KafkaRead on its group. A producer needs KafkaWrite and KafkaDescribe on its topics.

Kafka* matches every Kafka action. s3:Kafka* matches none of them.

Resources

The resources are:

Resource	Format
The Kafka view (cluster)	`vast:kafka:cluster:<bucket>`
A topic	`vast:kafka:topic:<bucket>:<topic>`
A consumer group	`vast:kafka:group:<bucket>:<group>`

<bucket> is the Kafka view's bucket. "*" matches every resource, and * inside a name works as a wildcard: vast:kafka:topic:<bucket>:orders* matches every topic whose name starts with orders. A resource that doesn't start with vast:kafka: (or the DataEngine prefix) is refused when the policy is saved, with Resource VRN=… requires data engine or kafka prefix.

An example identity policy for a consumer of one topic, with its own group. With it, the user consumed and committed offsets, and producing was refused:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOrdersTopic",
      "Effect": "Allow",
      "Action": ["KafkaRead", "KafkaDescribe"],
      "Resource": "vast:kafka:topic:<bucket>:orders"
    },
    {
      "Sid": "UseOwnGroup",
      "Effect": "Allow",
      "Action": "KafkaRead",
      "Resource": "vast:kafka:group:<bucket>:orders-reporting"
    }
  ]
}

What a refused operation looks like

Refused	Client sees
Produce or fetch on a topic	`TOPIC_AUTHORIZATION_FAILED` for that topic. A consumer reports `Subscribed topic not available: <topic>: Broker: Topic authorization failed`.
A consumer group operation	`GROUP_AUTHORIZATION_FAILED`, for example `JoinGroup failed: Broker: Group authorization failed`
Creating a topic, or reading or changing configurations	`CLUSTER_AUTHORIZATION_FAILED` when the cluster action is missing, `TOPIC_AUTHORIZATION_FAILED` when the topic action is
Deleting a topic	`TOPIC_AUTHORIZATION_FAILED`

No Kafka ACLs

The Kafka ACL API isn't supported: identity policies take its place. A DescribeAcls request gets SECURITY_DISABLED ("No Authorizer is configured on the broker"), which confluent-kafka's describe_acls() returns as an empty list. create_acls() and delete_acls() fail in the client with _UNSUPPORTED_FEATURE, and a client that sends those requests anyway has its connection closed. Tools that manage Kafka ACLs (such as kafka-acls) don't work.

Reading topics as tables

Reading a topic as a table through the VAST DataBase SDK, the Query Engine, Trino or Spark is a VAST DataBase request, not a Kafka one. It uses S3 access keys and the usual s3:Tabular* permissions on the Kafka view's bucket, not the Kafka actions. Blob expansion needs s3:TabularCreateBlobExpansion, s3:TabularAlterBlobExpansion, s3:TabularDropBlobExpansion and s3:TabularGetBlobExpansion. See VAST DataBase security and setup prerequisites.

So one VAST user can need two credentials: a password (SASL/PLAIN) or certificate (mTLS) for Kafka, and an S3 key pair for SQL and the SDK.

Auditing

Producing, consuming and topic management aren't recorded by VAST protocol auditing, and aren't subject to Quality of Service limits. If you need an audit trail of who read or wrote a topic, keep it in your application.

What to ask your platform team

Which connection kinds the view allows, and the port of each (9092, 9093).
The authentication method on each kind.
For TLS: the CA certificate that signed the cluster's Kafka certificate.
For SASL/PLAIN: a VAST local user and its password.
For mTLS: a client certificate with your VAST user name as its CN, its key, and confirmation that the CA is trusted for your tenant.
Whether authorization is on, and an identity policy with the actions above for your topics and groups.
For SQL and SDK reads: an S3 key pair and s3:Tabular* read access to the Kafka view's bucket.

The procedures are in the KB: user authentication, mTLS authentication, authorization and Kafka permissions in an identity policy.