Connect DataEngine to the Kubernetes Cluster and Registry

ℹ️ Info

Outcome: DataEngine is enabled on the tenant and linked to the Kafka broker, RKE2 cluster, namespaces, and container registry.

Before you begin: Complete the VAST setup, RKE2 cluster, registry, and DataEngine prerequisite procedures.

Previous: Install DataEngine Prerequisites on the Kubernetes Cluster  |  Next: Install the VAST DataEngine CLI Tools

ℹ️ Info

This procedure uses the Cluster Admin path to the enablement wizard. Tenant Admin users open the same wizard from the tenant context.

Enable DataEngine on the tenant

Navigate to Element Store → Tenants

Right-click the tenant (for example de-lab-tenant) and select Enable DataEngine

Create the Kafka broker

In the Enable DataEngine Window, select the dropdown under Default Broker and choose “+Add new broker”. The “Add View” window will be presented.

General

1. “Tenant” will already be configured.

2. Fill in the path for the Broker Bucket (for example /de-lab-broker)

3. Check the “Create Directory” box.

4. In “Policy Name”, choose the S3 default policy we created earlier for DataEngine (for example de-lab-s3-policy)

5. “Protocols” will be auto-populated

6. In “S3 Bucket Name” fill in the name of the Broker Bucket (for example de-lab-broker)

S3

1. Choose “User”

2. In “Bucket Owner” choose the Bucket Owner User we created earlier (for example de-lab-user)

Kafka

1. In “VIP Pool” choose the DataEngine VIP pool we created earlier (for example de-lab-vippool)

2. Uncheck “Encrypted connections (port 9093)”

3. Check “Unencrypted connections (port 9092)”

4. Select “Create”. This will bring you back to the “Enable DataEngine” window.

Create the Kafka topics

1. Select the dropdown under Default Topic and choose “+Add new topic”. The “Create Topic” window will be presented.

2. In “Create Topic”

   1. Fill in “Topic Name” (for example de-default-topic)

   2. Fill in “Number of Partitions”. A good starting point is the number of CNodes (for example 3)

   3. Select “Create”

3. Repeat steps 1 and 2 for the Default Deadletter Topic

   1. (Topic Name must be unique. for example de-deadletter-topic)

Select “Next” to continue the wizard.

Link the Kubernetes cluster

1. In “Name” give the RKE2 cluster a name (for example de-lab-rke2)

2. In “URL API server” fill it out in this format “https://<Primary IP address of the control-plane node>:6443” (for example https://10.144.19.2:6443)

3. In Certificates, you will need the values present in the kubeconfig file (for example ~/.kubeconfig/rke2.yaml) These values are already presented in base64 encoding as required.

   1. CA Certificate = the value of certificate-authority-data:

   2. Client Certificate = the value of client-certificate-data:

   3. Private Key = the value of client-key-data:

Select “Next” to continue the wizard

Select the namespaces

In “Select Namespaces”

1. Choose vast-pipelines

2. Click on the Right Arrow

Select “Next” to continue the wizard

Link the container registry

In “Link Container Registry”

1. For “Primary Kubernetes Cluster” select the cluster we created previously (for example de-lab-rke2)

2. For “Name” give the registry a name (for example de-lab-registry)

3. For “Base URL” fill out the URL of the registry (for example https://10.144.19.8:5000)

4. For “Authentication Method” choose None

Select “Finish” to complete the wizard

You may need to refresh your VMS window, but “VAST DataEngine” should now be available by clicking on “VAST Admin” and selecting it in the drop down

Verify the DataEngine connection

1. Refresh the VMS page.

2. Open the VAST Admin menu and select VAST DataEngine.

3. Confirm that the Kubernetes cluster and container registry appear in the DataEngine configuration.