Install an RKE2 Cluster

ℹ️ Info

Outcome: An RKE2 cluster is running and every node reports Ready in kubectl.

Before you begin: Provision the Ubuntu 24.04 virtual machines and confirm access to both the management and VAST storage networks.

Previous: Set Up VAST Tenants and VIP Pools  |  Next: Create a Podman Registry for the RKE2 Cluster

Prepare the virtual machines

Configure passwordless SSH and clush to run commands across the hosts. Use the first control-plane node as the manager for the installation.

VM List

Set Up Passwordless SSH (Prerequisite)

ClusterShell requires seamless SSH access to the managed VMs. On your manager VM, ensure you can SSH into each VM without a password.

Log into your Manager VM via SSH:

ssh vastdata@10.144.19.2

If you haven't done this yet, generate a key and copy it to all VMs:

Generate SSH key

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Copy the key to all 7 VMs

(Replace IPs with your IP range and vastdata with your actual admin username).

for ip in 10.144.19.{2..8}; do
    ssh-copy-id vastdata@$ip
done

Answer “yes” and enter your password for each node.

Install ClusterShell (recommended)

Clustershell (clush) is a huge timesaver during the initial setup of the VMs but will not be used very much afterward, therefore it is optional. This guide uses it for many commands.

Run the following on your designated management host to install the package from the Ubuntu Universe repository:

sudo apt update
sudo apt install clustershell -y

Configure Host Groups

Open the config file

sudo vi /etc/clustershell/groups.d/local.cfg

Delete the sample entries

Paste into file:

(Replace IPs with your IP ranges)

control: 10.144.19.[2-4]
worker: 10.144.19.[5-7]
repo: 10.144.19.8
all: @control @worker @repo

Test Configuration

nodeset -ll

Example Output

@all 10.144.19.[2-8]
@control 10.144.19.[2-4]
@repo 10.144.19.8
@worker 10.144.19.[5-7]

clush -a hostname

Example Output

10.144.19.2: ubuntu-01
10.144.19.3: ubuntu-02
10.144.19.4: ubuntu-03
10.144.19.5: ubuntu-04
10.144.19.6: ubuntu-05
10.144.19.7: ubuntu-06
10.144.19.8: ubuntu-07

Now that clush is configured it will be simpler to copy files and run commands on multiple VMs

Test connectivity to your VAST VIP Pool

Choose an IP on the de-lab-csi-vippool(For example 172.200.205.19). Use clush to ping the VIPPool from all VMs

clush -a 'ping -c 4 172.200.205.19 | grep "packet loss"'

Example Output

10.144.19.2: 4 packets transmitted, 4 received, 0% packet loss, time 3067ms
10.144.19.7: 4 packets transmitted, 4 received, 0% packet loss, time 3063ms
10.144.19.4: 4 packets transmitted, 4 received, 0% packet loss, time 3111ms
10.144.19.6: 4 packets transmitted, 4 received, 0% packet loss, time 3059ms
10.144.19.3: 4 packets transmitted, 4 received, 0% packet loss, time 3092ms
10.144.19.5: 4 packets transmitted, 4 received, 0% packet loss, time 3080ms
10.144.19.8: 4 packets transmitted, 4 received, 0% packet loss, time 3095ms

Update your VMs (recommended)

clush -a -S "sudo apt update"
clush -a -S "sudo apt upgrade -y"

Install nfs-common package on all VMs

clush -a -S 'sudo apt install -y nfs-common'

Reboot All VMs

Replace the IP with the IP of your manager VM, in order to ensure the command completes on all other VMs.

clush -a -S -x 10.144.19.2 'sudo reboot'

After that command completes, reboot the manager VM

sudo reboot

Once your manager Node has rebooted, log back in

Use clush to verify all VMs came back

clush -a hostname

Example Output

10.144.19.2: ubuntu-01
10.144.19.3: ubuntu-02
10.144.19.4: ubuntu-03
10.144.19.5: ubuntu-04
10.144.19.6: ubuntu-05
10.144.19.7: ubuntu-06
10.144.19.8: ubuntu-07

Install RKE2

This is a script that will speed up installation and configuration of RKE2 in your lab. It is currently configured to install version 1.35.

Create and Copy the Install Script

Edit the VERSION variable below to select yor RKE2 version.

Create this script on your manager node as install_server.sh

#!/bin/sh
#

TYPE=$1
SERVER=$2
VERSION=v1.35
TMP_CONFIG=/tmp/config.yaml

help() {
   echo "install_server.sh [first|next|agent] [server]"
   echo " first - Install the first server"
   echo " next  - Install additional server"
   echo " agent - Install an agent server"
   echo " server - API server IP/hostname. Only needed for 'next' or 'agent' installs"
}

add_ips() {
   for i in `ip -br a|awk '{ print $3 }'|grep -vE '^127|^f'|awk -F"/" '{ print $1 }'`
   do
	echo " - $i" >> $TMP_CONFIG
   done
}

add_common() {
cat >> $TMP_CONFIG << EOF
token: RKE2CLUSTER
EOF
 if [ "x$TYPE" != "xagent" ]
 then
cat >> $TMP_CONFIG << EOF
write-kubeconfig-mode: "0644"
tls-san:
EOF
 fi
}

add_install() {
FILE="/tmp/install_$TYPE.sh"
rm -f $FILE
cat > $FILE <<EOF 
#!/bin/sh
mkdir -p /etc/rancher/rke2/ /var/lib/rancher/rke2/server/manifests/
cp $TMP_CONFIG /etc/rancher/rke2/
curl -sfL https://get.rke2.io --output /tmp/install.sh
chmod +x /tmp/install.sh
cp $TMP_CONFIG /etc/rancher/rke2/
EOF
if [ "x$TYPE" = "xfirst" ]
then
cat >> $FILE <<EOF 
INSTALL_RKE2_CHANNEL=$VERSION /tmp/install.sh
EOF
elif  [ "x$TYPE" = "xnext" ]
then
cat >> $FILE <<EOF 
INSTALL_RKE2_CHANNEL=$VERSION /tmp/install.sh
EOF
elif  [ "x$TYPE" = "xagent" ]
then
cat >> $FILE <<EOF 
INSTALL_RKE2_CHANNEL=$VERSION INSTALL_RKE2_TYPE="agent" /tmp/install.sh
EOF
fi
if [ "x$TYPE" = "xagent" ]
then
cat >> $FILE <<EOF
systemctl enable --now rke2-agent.service
EOF
else
cat >> $FILE <<EOF
systemctl enable --now rke2-server.service
ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
EOF
fi
}

add_agent_labels() {
 if [ "x$TYPE" = "xagent" ]
 then
cat >> $TMP_CONFIG << EOF
EOF
 fi
}

add_extras() {
 if [ "x$TYPE" != "xfirst" ]
 then
  echo "server: https://$SERVER:9345" >> $TMP_CONFIG
 fi
}

create_config_file() {
 rm -f $TMP_CONFIG
 add_common 
 if [ "x$TYPE" != "xagent" ]
 then
  add_ips
 fi
 add_agent_labels
 add_extras 
 add_install
 echo "Next: cd /tmp; chmod 555 /tmp/install_$TYPE.sh; sudo sh /tmp/install_$TYPE.sh"
}

check_input() {
 if [ "x$TYPE" = "x" ]
 then
    help        # was: help()
    exit 0
 elif [ $TYPE != "first" ] && [ $TYPE != "next" ] && [ $TYPE != "agent" ]
 then
    help        # was: help()
    exit 1    
 fi

 echo "Installing ... $TYPE system"
 if [ "x$TYPE" != "xfirst" ]
 then
    if [ "x$SERVER" = "x" ]
    then
      echo "Error: Server is no bueno"
      help
      exit 1
    else 
      echo "Server is $SERVER"
    fi
 fi
}

check_input
create_config_file

Copy it to all of the control and worker VMs. (Not the repo VM)

clush -g control,worker -c ~/install_server.sh --dest ~/

Verify it was copied

clush -g worker,control "ls -l ~/install_server.sh"

Example Output

10.144.19.2: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh
10.144.19.5: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh
10.144.19.6: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh
10.144.19.4: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh
10.144.19.3: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh
10.144.19.7: -rw-rw-r-- 1 vastdata vastdata 2475 Aug 11 13:51 /home/vastdata/install_server.sh

Make it executable

clush -g worker,control "chmod +x ~/install_server.sh"

Run the script on the manager node

The script will run and generate a set of commands for you to run

./install_server.sh first

Example Output

Installing ... first system
Next: cd /tmp; chmod 555 /tmp/install_first.sh; sudo sh /tmp/install_first.sh

Run the generated commands

cd /tmp; chmod 555 /tmp/install_first.sh; sudo sh /tmp/install_first.sh

Run the script on the second and third control plane VMs

SSH to the second control-plane node on its management IP. On that host, join using the storage (backend) IP of the first control-plane node — not the IP you SSH’d to, and not this node’s own address.

install_server.sh next writes server: https://<JOIN_IP>:9345. Cluster membership traffic should stay on the data/backend network when the VMs have one. Use a management IP only if there is no separate storage network.

This lab’s first control-plane storage IP is 172.200.14.146:

./install_server.sh next 172.200.14.146

Example Output

Installing ... next system
Server is 172.200.14.160
Next: cd /tmp; chmod 555 /tmp/install_next.sh; sudo sh /tmp/install_next.sh

Run the Generated Commands

cd /tmp; chmod 555 /tmp/install_next.sh; sudo sh /tmp/install_next.sh

repeat steps for the third control plane node

Run the Script on the worker VMs

ssh to the first worker node and run

(change IP to match the IP of your first control-plane node)

./install_server.sh agent 172.200.14.146

Example Output

Installing ... agent system
Server is 172.200.16.160
Next: cd /tmp; chmod 555 /tmp/install_agent.sh; sudo sh /tmp/install_agent.sh

Run the Generated Commands

cd /tmp; chmod 555 /tmp/install_agent.sh; sudo sh /tmp/install_agent.sh

repeat steps for the second and third worker VMs

Setup kubectl

On the manager node

cd ~
mkdir ~/.kubeconfig
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kubeconfig/.
sudo chown vastdata:vastdata ~/.kubeconfig/rke2.yaml

Add the path to the KUBECONFIG environment variable. Add KUBECONFIG to ~/.profile, then source the file. The optional k alias shortens interactive kubectl commands; the guide continues to show full commands.

echo "export KUBECONFIG=${HOME}/.kubeconfig/rke2.yaml" >> ~/.profile
echo "alias k='kubectl'" >> ~/.profile
. ~/.profile

ℹ️ Info

This guide provides the full kubectl commands for clarity. Feel free to use the k alias instead as you follow along.

Verify that the cluster is up and running

kubectl cluster-info

Example Output

Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/rke2-coredns-rke2-coredns:udp-53/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Get the nodes in the cluster.

kubectl get node

Example Output

NAME      STATUS   ROLES                AGE     VERSION
ubuntu-01   Ready    control-plane,etcd   22h     v1.35.6+rke2r1
ubuntu-02   Ready    control-plane,etcd   21h     v1.35.6+rke2r1
ubuntu-03   Ready    control-plane,etcd   21h     v1.35.6+rke2r1
ubuntu-04   Ready    <none>               18h     v1.35.6+rke2r1
ubuntu-05   Ready    <none>               5h24m   v1.35.6+rke2r1
ubuntu-06   Ready    <none>               60m     v1.35.6+rke2r1

Label the worker nodes

Run this command for each worker node

kubectl label node <node-name> node-role.kubernetes.io/worker=""

Get the nodes in the cluster.

kubectl get node

Example Output

NAME      STATUS   ROLES                AGE     VERSION
ubuntu-01   Ready    control-plane,etcd   22h     v1.35.6+rke2r1
ubuntu-02   Ready    control-plane,etcd   21h     v1.35.6+rke2r1
ubuntu-03   Ready    control-plane,etcd   21h     v1.35.6+rke2r1
ubuntu-04   Ready    worker               18h     v1.35.6+rke2r1
ubuntu-05   Ready    worker               5h30m   v1.35.6+rke2r1
ubuntu-06   Ready    worker               66m     v1.35.6+rke2r1

Follow this guide to continue: Create a Podman Registry for the RKE2 Cluster