Create a Podman Registry for the RKE2 Cluster (Self-Signed Cert)

ℹ️ Info
Outcome: A self-signed TLS-enabled Podman registry is reachable and trusted by every RKE2 node.
Before you begin: Install the RKE2 cluster, mount the registry view, and record the registry IP and port.
Previous: Install an RKE2 Cluster | Next: Installing the VAST CSI Driver

ℹ️ Info
These instructions are for a lab environment with no DNS. Use management IPs for SSH, scp, and clush. Use the registry IP in the certificate SAN, Podman, registries.yaml, and image names. Do not use hostnames unless you have added them to /etc/hosts on every node.

📝 Note
Image pulls are done by each node's kubelet/containerd. Installing the CA only on the manager, in a pod, or only under Podman certs.d does not fix cluster-wide pulls.

Set the registry variables

Replace the example values below with the IP addresses and view paths from your lab worksheet.

ℹ️ Info
Set these in the shell of each node before running commands:

export REGISTRY_HOST=10.144.19.8
export REGISTRY_PORT=5000
export MASTER_IP=10.144.19.2
export SSH_USER=vastdata
export NFS_EXPORT=172.200.205.19:/de-lab-docker/images
export MOUNTPOINT=/mnt/nfs/docker-registry
export CERT_DAYS=3650
export CA_FILE=/etc/rancher/rke2/certs/registry-ca.crt

Variable	Meaning
`REGISTRY_HOST`	Registry IP (must be in the cert SAN)
`REGISTRY_PORT`	Registry listen port
`MASTER_IP`	Manager management IP (has `clush`)
`SSH_USER`	Account used for `scp`/`ssh`/`clush`
`NFS_EXPORT`	Persistent image store
`MOUNTPOINT`	Local NFS mount for image data and certs
`CERT_DAYS`	Self-signed certificate validity
`CA_FILE`	Cluster trust path on every RKE2 node

If you later add DNS for the registry, put that name in the cert SAN under [alt_names] as well (DNS.1 = ...) and keep IP.1. Until then, connect only by IP.

Distribute Variables to ALL VMs

On the Manager Node, create the variables file setup_vars

cd ~
vi setup_vars

Paste the export commands above into the file, modify the values for your environment and save it.

Copy to all nodes

clush -a -c ~/setup_vars ~/setup_vars

Now you can use source to add the variables to the shell on each machine.

source setup_vars

You can validate the variables by using the export command.

Set up the registry VM

Host: registry VM 10.144.19.8.

1. Install Podman and OpenSSL

ℹ️ Info
nfs-common should have been installed in a previous step

sudo apt update
sudo apt install -y podman openssl

2. Create persistent storage and mount NFS

source your environment variables from above

source setup_vars

Run the following commands to create a directory and mount the NFS view

sudo mkdir -p "${MOUNTPOINT}"
sudo mount -t nfs "${NFS_EXPORT}" "${MOUNTPOINT}"
echo "${NFS_EXPORT} ${MOUNTPOINT} nfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab

3. Create a self-signed certificate

The registry image expects a cert whose SAN matches how clients connect. For an IP-only registry, put the IP in subjectAltName.

sudo mkdir -p "${MOUNTPOINT}/certs"

cat << EOF | sudo tee "${MOUNTPOINT}/certs/openssl.cnf"
[req]
default_bits       = 4096
prompt             = no
default_md         = sha256
distinguished_name = dn
x509_extensions    = v3_req

[dn]
C  = US
ST = Lab
L  = Lab
O  = VAST Lab
CN = ${REGISTRY_HOST}

[v3_req]
subjectAltName = @alt_names
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
IP.1 = ${REGISTRY_HOST}
EOF

sudo openssl req -x509 -nodes -days "${CERT_DAYS}" \
  -newkey rsa:4096 \
  -keyout "${MOUNTPOINT}/certs/domain.key" \
  -out "${MOUNTPOINT}/certs/domain.crt" \
  -config "${MOUNTPOINT}/certs/openssl.cnf"

sudo chmod 600 "${MOUNTPOINT}/certs/domain.key"
sudo chmod 644 "${MOUNTPOINT}/certs/domain.crt"
# registry:2 runs as uid 1000 and cannot read a root-owned 600 key
sudo chown 1000:1000 "${MOUNTPOINT}/certs/domain.key" "${MOUNTPOINT}/certs/domain.crt"

domain.crt is both the server certificate and the CA clients will trust.

ℹ️ Info
If the container still logs open /certs/domain.key: permission denied, the NFS export is likely root-squashing chown. In that case create the files as a uid-1000 user, or set the export to no_root_squash for this share.

4. Trust the cert on the registry VM (Podman client)

Podman looks for a CA at /etc/containers/certs.d/<host:port>/ca.crt. Do not mark the registry insecure; serve HTTPS and trust the cert.

sudo mkdir -p "/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}"
sudo cp "${MOUNTPOINT}/certs/domain.crt" \
  "/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt"

5. Run the TLS registry container

podman run -d \
  --name local-registry \
  -p ${REGISTRY_PORT}:5000 \
  -v "${MOUNTPOINT}:/var/lib/registry:Z" \
  -v "${MOUNTPOINT}/certs:/certs:Z" \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  docker.io/library/registry:2

6. Auto-start on boot

mkdir -p ~/.config/systemd/user/
cd ~/.config/systemd/user/
podman generate systemd --name local-registry --files --new
systemctl --user daemon-reload
systemctl --user enable --now container-local-registry.service
sudo loginctl enable-linger "$USER"

7. Confirm TLS is working

Plain HTTP should fail (connection error or empty TLS handshake). HTTPS without the CA should fail verification:

curl -sS "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"
# expect SSL certificate problem

Example Output

curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

HTTPS with the CA should succeed:

curl -sS --cacert "${MOUNTPOINT}/certs/domain.crt" \
  "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"

Expected output on a new registry:

{"repositories":[]}

Configure RKE2 nodes to trust the certificate

Use @control and @worker (10.144.19.2–10.144.19.7). Do not use -a / @repo. The registry VM does not run RKE2 and does not have clush.

Trust paths

Client	Path	Why
RKE2 containerd / kubelet	`${CA_FILE}` plus `tls.ca_file` in `/etc/rancher/rke2/registries.yaml`	This is what cluster pulls use
Podman	`/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt`	Already installed on the registry VM in setup step 4

Troubleshooting

If the cert SAN does not match REGISTRY_HOST, reissue the cert.

If a pod is already failing, check the image in kubectl describe before changing files:

x509 on 10.144.19.8:5000 — CA or SAN
127.0.0.1:31999 / manifest unknown — Zarf rewrite, not TLS (see Testing)
401 Unauthorized — TLS worked; credentials are a separate problem

1. Copy the CA from the registry VM to the manager

On the registry VM, confirm SAN and fingerprint, then copy:

openssl x509 -in "${MOUNTPOINT}/certs/domain.crt" -noout -subject -dates -fingerprint -sha256 -ext subjectAltName

scp "${MOUNTPOINT}/certs/domain.crt" "${SSH_USER}@${MASTER_IP}:/tmp/registry-ca.crt"

The SAN must include IP Address:${REGISTRY_HOST}. Record the SHA-256 fingerprint.

SSH to the manager and continue from there:

ssh "${SSH_USER}@${MASTER_IP}"

source your environment variables from above

source setup_vars

Run the following command to fingerprint the cert.

openssl x509 -in /tmp/registry-ca.crt -noout -fingerprint -sha256

openssl s_client \
  -connect "${REGISTRY_HOST}:${REGISTRY_PORT}" \
  -servername "${REGISTRY_HOST}" \
  -CAfile /tmp/registry-ca.crt \
  -verify_return_error </dev/null

Do not distribute the file if the fingerprint or verify step fails.

2. Install the CA on all RKE2 nodes

On the manager, install locally before clush. Do not rm /tmp/registry-ca.crt until fingerprints match on every node.

sudo install -d -m 0755 /etc/rancher/rke2/certs
sudo install -m 0644 /tmp/registry-ca.crt "${CA_FILE}"
sudo openssl x509 -in "${CA_FILE}" -noout -fingerprint -sha256

Then copy to @control and @worker (includes the manager):

clush -g control,worker "sudo install -d -m 0755 /etc/rancher/rke2/certs"
clush -g control,worker -c /tmp/registry-ca.crt --dest /tmp/registry-ca.crt
clush -g control,worker -S "sudo install -m 0644 /tmp/registry-ca.crt ${CA_FILE}"
clush -b -g control,worker -S "hostname; sudo openssl x509 -in ${CA_FILE} -noout -fingerprint -sha256"

All six fingerprints must match. Then remove the temp copy:

clush -g control,worker "rm -f /tmp/registry-ca.crt"

3. Backup and write `registries.yaml`

Still on the manager. Backup first on every RKE2 node:

clush -g control,worker -S 'sudo mkdir -p /etc/rancher/rke2
if [ -f /etc/rancher/rke2/registries.yaml ]; then
  sudo cp /etc/rancher/rke2/registries.yaml /etc/rancher/rke2/registries.yaml.pre-ca-trust
fi'

Write the lab file. This replaces /etc/rancher/rke2/registries.yaml. If you already had other mirrors, copy those entries from registries.yaml.pre-ca-trust into this file before distributing.

cat << EOF > /tmp/registries.yaml
mirrors:
  "${REGISTRY_HOST}:${REGISTRY_PORT}":
    endpoint:
      - "https://${REGISTRY_HOST}:${REGISTRY_PORT}"
configs:
  "${REGISTRY_HOST}:${REGISTRY_PORT}":
    tls:
      ca_file: ${CA_FILE}
      insecure_skip_verify: true
EOF

cat /tmp/registries.yaml

4. Distribute `registries.yaml` with clush

clush -g control,worker -c /tmp/registries.yaml --dest /tmp/registries.yaml
clush -g control,worker -S "sudo install -m 0644 /tmp/registries.yaml /etc/rancher/rke2/registries.yaml && rm -f /tmp/registries.yaml"
clush -g control,worker "sudo ls -l /etc/rancher/rke2/registries.yaml /etc/rancher/rke2/registries.yaml.pre-ca-trust"

You may receive errors on the ls commands for the backup files /etc/rancher/rke2/registries.yaml.pre-ca-trust these can be safely ignored.

10.144.19.7: ls: cannot access '/etc/rancher/rke2/registries.yaml.pre-ca-trust': No such file or directory

5. Restart RKE2 so nodes re-read registry config

From the manager, restart the control-plane group, then the workers:

clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"

These commands take a few moments to return.

6. Verify that all nodes have `hosts.toml` file

⚠️ Warning
IMPORATANT! Check that containerd received the CA

clush -g control,worker "sudo awk '1' /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml"

If any hosts are missing the file, copy it from the master node; otherwise, skip to https://vastdata.atlassian.net/wiki/spaces/ALTA/pages/8001191941/Create+a+Podman+Registry+for+the+RKE2+Cluster#Testing :

export HOSTS_DIR="/var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}"
export HOSTS_FILE="${HOSTS_DIR}/hosts.toml"

# Stage the local source because it is root-readable.
sudo install -m 0644 "$HOSTS_FILE" /tmp/rke2-hosts.toml.seed
# Copy to every node's temporary directory.
clush -g control,worker --copy \
  /tmp/rke2-hosts.toml.seed \
  --dest /tmp/rke2-hosts.toml.seed
  
# Install only when missing.
clush -g control,worker "
  if sudo test -f '${HOSTS_FILE}'; then
    echo 'hosts.toml already exists'
  else
    sudo install -d -m 0700 '${HOSTS_DIR}' &&
    sudo install -m 0600 /tmp/rke2-hosts.toml.seed '${HOSTS_FILE}' &&
    echo 'hosts.toml installed'
  fi
  rm -f /tmp/rke2-hosts.toml.seed
"
rm -f /tmp/rke2-hosts.toml.seed

7. Restart RKE2 so nodes re-read registry config

From the manager, restart the control-plane group, then the workers:

clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"

Verify after copy

clush -g control,worker "sudo awk '1' /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml"

After installing VAST Operator

8. Knative Config

Make sure we have a clean secret

kubectl delete secret private-registry -n knative-serving
kubectl create secret docker-registry private-registry --docker-server=dummy --docker-username=dummy --docker-password=dummy -n knative-serving

Patch knative-serving

kubectl patch cm config-deployment -n knative-serving --type merge -p '{"data":{"registriesSkippingTagResolving":"10.144.19.8:5000"}}'

9. Prevent Zarf from Rewriting Images in this Namespace

Tell Zarf to ignore the vast-pipelines namespace so it stops redirecting pulls to its internal registry:

# Label namespace to ignore Zarf mutation
kubectl label namespace vast-pipelines zarf.dev/agent=ignore --overwrite

10. Trust the Insecure Registry on your Build Host

These steps vary based on what platform you are running Docker on.

For Mac with Docker Desktop

Open Docker Desktop App → Settings → Docker Engine

Edit the IP Address to match your registry, then Paste the JSON below in the box

{
  "min-api-version": "1.24",
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "insecure-registries": [
    "10.144.19.8:5000"
  ]
}

Choose “Apply & Restart

For Linux with Docker.io

Edit or create /etc/docker/daemon.json

{
  "min-api-version": "1.24",
  "features": {
    "containerd-snapshotter": false
  },
  "storage-driver": "overlay2",
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "insecure-registries": [
    "10.144.10.6:5000"
  ]
}

Edit the IP Address to match your registry, then Paste the JSON above

Restart Docker

systemctl restart docker.service

Testing

1. Podman push over TLS

On the registry node:

source your environment variables from above

source setup_vars

Run the following commands to pull and push a test image:

podman pull docker.io/library/alpine:latest
podman tag alpine:latest ${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1
podman push ${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1
curl -sS --cacert /etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt \
  "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"

Expected output:

{"repositories":["test-alpine"]}

If push fails with x509: certificate signed by unknown authority, the CA is missing from certs.d. If it fails with certificate is not valid for any names / IP SAN errors, regenerate the cert with IP.1 = ${REGISTRY_HOST}.

2. Node-level pull with `crictl`

On a workerVM:

source your environment variables from above

source setup_vars

use crictl to test a pull to the node

ℹ️ Info
sudo does not use the PATH from source, so call the RKE2 binary by full path.

CRICTL=/var/lib/rancher/rke2/bin/crictl
CRICTL_CONFIG=/var/lib/rancher/rke2/agent/etc/crictl.yaml

sudo "$CRICTL" --config "$CRICTL_CONFIG" pull "${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1"
sudo "$CRICTL" --config "$CRICTL_CONFIG" images | grep test-alpine

If crictl reports a TLS error, confirm ca_file on that node and restart rke2-server or rke2-agent. if an x509 error is reported, verify that /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml exists and matches the manager node.

3. Run a Kubernetes test pod on the Manager VM

On the Managernode:

kubectl run registry-test \
  --image=${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1 \
  --restart=Never \
  --command -- sleep 3600

kubectl get pod registry-test
kubectl describe pod registry-test
kubectl delete pod registry-test

Example deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: 10.144.19.8:5000/my-app:v1
          ports:
            - containerPort: 80

Certificate rotation

Because this registry cert is self-signed, domain.crt is also the trust anchor. Replacing it changes which nodes must trust it.

Generate a new domain.crt / domain.key on the registry VM (same SAN: IP.1 = ${REGISTRY_HOST}).
Build a bundle of old + new certs (concatenate PEM). Install that bundle as ${CA_FILE} on @control,@worker. registries.yaml can stay as-is if ca_file already points at ${CA_FILE}. Then restart:
```
clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"
```
Restart the registry container so it serves the new leaf. Confirm openssl s_client with the bundle.
Repeat the crictl --config pull and the Kubernetes test pod.
After every node pulls successfully, install only the new cert as ${CA_FILE} and restart RKE2 the same way.

If a node fails after a registries.yaml edit, restore the backup and restart the same service:

sudo cp /etc/rancher/rke2/registries.yaml.pre-ca-trust \
  /etc/rancher/rke2/registries.yaml
sudo systemctl restart rke2-server  # or rke2-agent

Restoring a backup that still has insecure_skip_verify is only an emergency rollback. Put ca_file back before leaving the node in service.

Create a Podman Registry for the RKE2 Cluster (Self-Signed Cert)

Set the registry variables

Distribute Variables to ALL VMs

Set up the registry VM

1. Install Podman and OpenSSL

2. Create persistent storage and mount NFS

3. Create a self-signed certificate

4. Trust the cert on the registry VM (Podman client)

5. Run the TLS registry container

6. Auto-start on boot

7. Confirm TLS is working

Configure RKE2 nodes to trust the certificate

Troubleshooting

1. Copy the CA from the registry VM to the manager

2. Install the CA on all RKE2 nodes

3. Backup and write registries.yaml

4. Distribute registries.yaml with clush

5. Restart RKE2 so nodes re-read registry config

6. Verify that all nodes have hosts.toml file

7. Restart RKE2 so nodes re-read registry config

After installing VAST Operator

8. Knative Config

9. Prevent Zarf from Rewriting Images in this Namespace

10. Trust the Insecure Registry on your Build Host

Testing

1. Podman push over TLS

2. Node-level pull with crictl

3. Run a Kubernetes test pod on the Manager VM

Example deployment

Certificate rotation

3. Backup and write `registries.yaml`

4. Distribute `registries.yaml` with clush

6. Verify that all nodes have `hosts.toml` file

2. Node-level pull with `crictl`