---
title: "Create a Podman Registry for the RKE2 Cluster (Self-Signed Cert)"
slug: "create-a-registry-with-podman-self-signed-tls-with-rke2-cluster"
description: "Set up a secure IP-only registry with self-signed certificates, ensuring cluster-wide trust and proper configuration for RKE2 nodes."
tags: ["DataEngine"]
updated: 2026-10-28T02:14:35Z
published: 2026-10-28T02:14:35Z
canonical: "docs.vastdata.com/create-a-registry-with-podman-self-signed-tls-with-rke2-cluster"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create a Podman Registry for the RKE2 Cluster (Self-Signed Cert)

> [!NOTE]
> ℹ️ Info
> 
> **Outcome:** A self-signed TLS-enabled Podman registry is reachable and trusted by every RKE2 node.
> 
> **Before you begin:** Install the RKE2 cluster, mount the registry view, and record the registry IP and port.
> 
> **Previous:** [Install an RKE2 Cluster](/develop/docs/rke2-cluster) | **Next:** [Installing the VAST CSI Driver](/develop/docs/installing-vast-csi-driver)

> [!NOTE]
> ℹ️ Info
> 
> These instructions are for a lab environment with **no DNS**. Use management IPs for SSH, `scp`, and `clush`. Use the registry IP in the certificate SAN, Podman, `registries.yaml`, and image names. Do not use hostnames unless you have added them to `/etc/hosts` on every node.

> [!NOTE]
> 📝 Note
> 
> Image pulls are done by each node's kubelet/containerd. Installing the CA only on the manager, in a pod, or only under Podman `certs.d` does not fix cluster-wide pulls.

## Set the registry variables

Replace the example values below with the IP addresses and view paths from your lab worksheet.

> [!NOTE]
> ℹ️ Info
> 
> Set these in the shell of each node before running commands:

```bash
export REGISTRY_HOST=10.144.19.8
export REGISTRY_PORT=5000
export MASTER_IP=10.144.19.2
export SSH_USER=vastdata
export NFS_EXPORT=172.200.205.19:/de-lab-docker/images
export MOUNTPOINT=/mnt/nfs/docker-registry
export CERT_DAYS=3650
export CA_FILE=/etc/rancher/rke2/certs/registry-ca.crt
```

| Variable | Meaning |
| --- | --- |
| `REGISTRY_HOST` | Registry **IP** (must be in the cert SAN) |
| `REGISTRY_PORT` | Registry listen port |
| `MASTER_IP` | Manager management IP (has `clush`) |
| `SSH_USER` | Account used for `scp`/`ssh`/`clush` |
| `NFS_EXPORT` | Persistent image store |
| `MOUNTPOINT` | Local NFS mount for image data and certs |
| `CERT_DAYS` | Self-signed certificate validity |
| `CA_FILE` | Cluster trust path on every RKE2 node |

If you later add DNS for the registry, put that name in the cert SAN under `[alt_names]` as well (`DNS.1 = ...`) **and** keep `IP.1`. Until then, connect only by IP.

### Distribute Variables to ALL VMs

On the Manager Node, create the variables file `setup_vars`

```bash
cd ~
vi setup_vars
```

Paste the export commands above into the file, modify the values for your environment and save it.

Copy to all nodes

```bash
clush -a -c ~/setup_vars ~/setup_vars
```

Now you can use `source` to add the variables to the shell on each machine.

```bash
source setup_vars
```

You can validate the variables by using the `export` command.

## Set up the registry VM

Host: registry VM `10.144.19.8`.

### 1. Install Podman and OpenSSL

> [!NOTE]
> ℹ️ Info
> 
> `nfs-common` should have been installed in a previous step

```bash
sudo apt update
sudo apt install -y podman openssl
```

### 2. Create persistent storage and mount NFS

`source` your environment variables from above

```bash
source setup_vars
```

Run the following commands to create a directory and mount the NFS view

```bash
sudo mkdir -p "${MOUNTPOINT}"
sudo mount -t nfs "${NFS_EXPORT}" "${MOUNTPOINT}"
echo "${NFS_EXPORT} ${MOUNTPOINT} nfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab
```

### 3. Create a self-signed certificate

The registry image expects a cert whose SAN matches how clients connect. For an IP-only registry, put the IP in `subjectAltName`.

```bash
sudo mkdir -p "${MOUNTPOINT}/certs"

cat << EOF | sudo tee "${MOUNTPOINT}/certs/openssl.cnf"
[req]
default_bits       = 4096
prompt             = no
default_md         = sha256
distinguished_name = dn
x509_extensions    = v3_req

[dn]
C  = US
ST = Lab
L  = Lab
O  = VAST Lab
CN = ${REGISTRY_HOST}

[v3_req]
subjectAltName = @alt_names
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
IP.1 = ${REGISTRY_HOST}
EOF

sudo openssl req -x509 -nodes -days "${CERT_DAYS}" \
  -newkey rsa:4096 \
  -keyout "${MOUNTPOINT}/certs/domain.key" \
  -out "${MOUNTPOINT}/certs/domain.crt" \
  -config "${MOUNTPOINT}/certs/openssl.cnf"

sudo chmod 600 "${MOUNTPOINT}/certs/domain.key"
sudo chmod 644 "${MOUNTPOINT}/certs/domain.crt"
# registry:2 runs as uid 1000 and cannot read a root-owned 600 key
sudo chown 1000:1000 "${MOUNTPOINT}/certs/domain.key" "${MOUNTPOINT}/certs/domain.crt"
```

`domain.crt` is both the server certificate and the CA clients will trust.

> [!NOTE]
> ℹ️ Info
> 
> If the container still logs `open /certs/domain.key: permission denied`, the NFS export is likely root-squashing `chown`. In that case create the files as a uid-1000 user, or set the export to `no_root_squash` for this share.

### 4. Trust the cert on the registry VM (Podman client)

Podman looks for a CA at `/etc/containers/certs.d/&lt;host:port&gt;/ca.crt`. Do **not** mark the registry insecure; serve HTTPS and trust the cert.

```bash
sudo mkdir -p "/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}"
sudo cp "${MOUNTPOINT}/certs/domain.crt" \
  "/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt"
```

### 5. Run the TLS registry container

```bash
podman run -d \
  --name local-registry \
  -p ${REGISTRY_PORT}:5000 \
  -v "${MOUNTPOINT}:/var/lib/registry:Z" \
  -v "${MOUNTPOINT}/certs:/certs:Z" \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  docker.io/library/registry:2
```

### 6. Auto-start on boot

```bash
mkdir -p ~/.config/systemd/user/
cd ~/.config/systemd/user/
podman generate systemd --name local-registry --files --new
systemctl --user daemon-reload
systemctl --user enable --now container-local-registry.service
sudo loginctl enable-linger "$USER"
```

### 7. Confirm TLS is working

Plain HTTP should fail (connection error or empty TLS handshake). HTTPS without the CA should fail verification:

```bash
curl -sS "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"
# expect SSL certificate problem
```

Example Output

```plaintext
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```

HTTPS with the CA should succeed:

```bash
curl -sS --cacert "${MOUNTPOINT}/certs/domain.crt" \
  "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"
```

Expected output on a new registry:

```json
{"repositories":[]}
```

## Configure RKE2 nodes to trust the certificate

Use `@control` and `@worker` (`10.144.19.2`–`10.144.19.7`). Do **not** use `-a` / `@repo`. The registry VM does not run RKE2 and does not have `clush`.

**Trust paths**

| Client | Path | Why |
| --- | --- | --- |
| RKE2 containerd / kubelet | `${CA_FILE}` plus `tls.ca_file` in `/etc/rancher/rke2/registries.yaml` | This is what cluster pulls use |
| Podman | `/etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt` | Already installed on the registry VM in setup step 4 |

### Troubleshooting

If the cert SAN does not match `REGISTRY_HOST`, reissue the cert.

If a pod is already failing, check the image in `kubectl describe` before changing files:

- `x509` on `10.144.19.8:5000` — CA or SAN
- `127.0.0.1:31999` / `manifest unknown` — Zarf rewrite, not TLS (see Testing)
- `401 Unauthorized` — TLS worked; credentials are a separate problem

### 1. Copy the CA from the registry VM to the manager

On the **registry VM**, confirm SAN and fingerprint, then copy:

```bash
openssl x509 -in "${MOUNTPOINT}/certs/domain.crt" -noout -subject -dates -fingerprint -sha256 -ext subjectAltName

scp "${MOUNTPOINT}/certs/domain.crt" "${SSH_USER}@${MASTER_IP}:/tmp/registry-ca.crt"
```

The SAN must include `IP Address:${REGISTRY_HOST}`. Record the SHA-256 fingerprint.

SSH to the manager and continue from there:

```bash
ssh "${SSH_USER}@${MASTER_IP}"
```

`source` your environment variables from above

```bash
source setup_vars
```

Run the following command to fingerprint the cert.

```bash
openssl x509 -in /tmp/registry-ca.crt -noout -fingerprint -sha256

openssl s_client \
  -connect "${REGISTRY_HOST}:${REGISTRY_PORT}" \
  -servername "${REGISTRY_HOST}" \
  -CAfile /tmp/registry-ca.crt \
  -verify_return_error </dev/null
```

Do not distribute the file if the fingerprint or verify step fails.

### 2. Install the CA on all RKE2 nodes

On the **manager**, install locally **before** `clush`. Do not `rm` `/tmp/registry-ca.crt` until fingerprints match on every node.

```bash
sudo install -d -m 0755 /etc/rancher/rke2/certs
sudo install -m 0644 /tmp/registry-ca.crt "${CA_FILE}"
sudo openssl x509 -in "${CA_FILE}" -noout -fingerprint -sha256
```

Then copy to `@control` and `@worker` (includes the manager):

```bash
clush -g control,worker "sudo install -d -m 0755 /etc/rancher/rke2/certs"
clush -g control,worker -c /tmp/registry-ca.crt --dest /tmp/registry-ca.crt
clush -g control,worker -S "sudo install -m 0644 /tmp/registry-ca.crt ${CA_FILE}"
clush -b -g control,worker -S "hostname; sudo openssl x509 -in ${CA_FILE} -noout -fingerprint -sha256"
```

All six fingerprints must match. Then remove the temp copy:

```bash
clush -g control,worker "rm -f /tmp/registry-ca.crt"
```

### 3. Backup and write `registries.yaml`

Still on the **manager**. Backup first on every RKE2 node:

```bash
clush -g control,worker -S 'sudo mkdir -p /etc/rancher/rke2
if [ -f /etc/rancher/rke2/registries.yaml ]; then
  sudo cp /etc/rancher/rke2/registries.yaml /etc/rancher/rke2/registries.yaml.pre-ca-trust
fi'
```

Write the lab file. This replaces `/etc/rancher/rke2/registries.yaml`. If you already had other mirrors, copy those entries from `registries.yaml.pre-ca-trust` into this file before distributing.

```bash
cat << EOF > /tmp/registries.yaml
mirrors:
  "${REGISTRY_HOST}:${REGISTRY_PORT}":
    endpoint:
      - "https://${REGISTRY_HOST}:${REGISTRY_PORT}"
configs:
  "${REGISTRY_HOST}:${REGISTRY_PORT}":
    tls:
      ca_file: ${CA_FILE}
      insecure_skip_verify: true
EOF

cat /tmp/registries.yaml
```

### 4. Distribute `registries.yaml` with clush

```bash
clush -g control,worker -c /tmp/registries.yaml --dest /tmp/registries.yaml
clush -g control,worker -S "sudo install -m 0644 /tmp/registries.yaml /etc/rancher/rke2/registries.yaml && rm -f /tmp/registries.yaml"
clush -g control,worker "sudo ls -l /etc/rancher/rke2/registries.yaml /etc/rancher/rke2/registries.yaml.pre-ca-trust"
```

You may receive errors on the `ls` commands for the backup files `/etc/rancher/rke2/registries.yaml.pre-ca-trust `these can be safely ignored.

```plaintext
10.144.19.7: ls: cannot access '/etc/rancher/rke2/registries.yaml.pre-ca-trust': No such file or directory
```

### 5. Restart RKE2 so nodes re-read registry config

From the **manager**, restart the control-plane group, then the workers:

```bash
clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"
```

These commands take a few moments to return.

### 6. Verify that all nodes have `hosts.toml` file

> [!NOTE]
> ⚠️ Warning
> 
> IMPORATANT! Check that containerd received the CA

```bash
clush -g control,worker "sudo awk '1' /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml"
```

If any hosts are missing the file, copy it from the master node; otherwise, skip to [https://vastdata.atlassian.net/wiki/spaces/ALTA/pages/8001191941/Create+a+Podman+Registry+for+the+RKE2+Cluster#Testing](https://vastdata.atlassian.net/wiki/spaces/ALTA/pages/8001191941/Create+a+Podman+Registry+for+the+RKE2+Cluster#Testing) :

```bash
export HOSTS_DIR="/var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}"
export HOSTS_FILE="${HOSTS_DIR}/hosts.toml"

# Stage the local source because it is root-readable.
sudo install -m 0644 "$HOSTS_FILE" /tmp/rke2-hosts.toml.seed
# Copy to every node's temporary directory.
clush -g control,worker --copy \
  /tmp/rke2-hosts.toml.seed \
  --dest /tmp/rke2-hosts.toml.seed
  
# Install only when missing.
clush -g control,worker "
  if sudo test -f '${HOSTS_FILE}'; then
    echo 'hosts.toml already exists'
  else
    sudo install -d -m 0700 '${HOSTS_DIR}' &&
    sudo install -m 0600 /tmp/rke2-hosts.toml.seed '${HOSTS_FILE}' &&
    echo 'hosts.toml installed'
  fi
  rm -f /tmp/rke2-hosts.toml.seed
"
rm -f /tmp/rke2-hosts.toml.seed
```

### 7. Restart RKE2 so nodes re-read registry config

From the **manager**, restart the control-plane group, then the workers:

```bash
clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"
```

Verify after copy

```bash
clush -g control,worker "sudo awk '1' /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml"
```

### 

### After installing VAST Operator

### 8. Knative Config

Make sure we have a clean secret

```plaintext
kubectl delete secret private-registry -n knative-serving
kubectl create secret docker-registry private-registry --docker-server=dummy --docker-username=dummy --docker-password=dummy -n knative-serving
```

Patch `knative-serving`

```plaintext
kubectl patch cm config-deployment -n knative-serving --type merge -p '{"data":{"registriesSkippingTagResolving":"10.144.19.8:5000"}}'
```

### 9. Prevent Zarf from Rewriting Images in this Namespace

Tell Zarf to ignore the `vast-pipelines` namespace so it stops redirecting pulls to its internal registry:

```plaintext
# Label namespace to ignore Zarf mutation
kubectl label namespace vast-pipelines zarf.dev/agent=ignore --overwrite
```

### 10. Trust the Insecure Registry on your Build Host

These steps vary based on what platform you are running Docker on.

**For Mac with Docker Desktop**

Open Docker Desktop App → Settings → Docker Engine

Edit the IP Address to match your registry, then Paste the JSON below in the box

```json
{
  "min-api-version": "1.24",
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "insecure-registries": [
    "10.144.19.8:5000"
  ]
}
```

Choose “Apply & Restart

**For Linux with Docker.io**

Edit or create `/etc/docker/daemon.json`

```json
{
  "min-api-version": "1.24",
  "features": {
    "containerd-snapshotter": false
  },
  "storage-driver": "overlay2",
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "insecure-registries": [
    "10.144.10.6:5000"
  ]
}
```

Edit the IP Address to match your registry, then Paste the JSON above

Restart Docker

```shell
systemctl restart docker.service
```

## Testing

### 1. Podman push over TLS

On the registry node:

`source` your environment variables from above

```bash
source setup_vars
```

Run the following commands to pull and push a test image:

```bash
podman pull docker.io/library/alpine:latest
podman tag alpine:latest ${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1
podman push ${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1
curl -sS --cacert /etc/containers/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt \
  "https://${REGISTRY_HOST}:${REGISTRY_PORT}/v2/_catalog"
```

Expected output:

```json
{"repositories":["test-alpine"]}
```

If push fails with `x509: certificate signed by unknown authority`, the CA is missing from `certs.d`. If it fails with `certificate is not valid for any names` / IP SAN errors, regenerate the cert with `IP.1 = ${REGISTRY_HOST}`.

### 2. Node-level pull with `crictl`

On a workerVM:

`source` your environment variables from above

```bash
source setup_vars
```

use `crictl` to test a pull to the node

> [!NOTE]
> ℹ️ Info
> 
> `sudo` does not use the PATH from `source`, so call the RKE2 binary by full path.

```bash
CRICTL=/var/lib/rancher/rke2/bin/crictl
CRICTL_CONFIG=/var/lib/rancher/rke2/agent/etc/crictl.yaml

sudo "$CRICTL" --config "$CRICTL_CONFIG" pull "${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1"
sudo "$CRICTL" --config "$CRICTL_CONFIG" images | grep test-alpine
```

If `crictl` reports a TLS error, confirm `ca_file` on that node and restart `rke2-server` or `rke2-agent`. if an x509 error is reported, verify that `/var/lib/rancher/rke2/agent/etc/containerd/certs.d/${REGISTRY_HOST}:${REGISTRY_PORT}/hosts.toml` exists and matches the manager node.

### 3. Run a Kubernetes test pod on the Manager VM

On the Managernode:

```bash
kubectl run registry-test \
  --image=${REGISTRY_HOST}:${REGISTRY_PORT}/test-alpine:v1 \
  --restart=Never \
  --command -- sleep 3600

kubectl get pod registry-test
kubectl describe pod registry-test
kubectl delete pod registry-test
```

## Example deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: 10.144.19.8:5000/my-app:v1
          ports:
            - containerPort: 80
```

## Certificate rotation

Because this registry cert is self-signed, `domain.crt` is also the trust anchor. Replacing it changes which nodes must trust it.

1. Generate a new `domain.crt` / `domain.key` on the registry VM (same SAN: `IP.1 = ${REGISTRY_HOST}`).
2. Build a **bundle** of old + new certs (concatenate PEM). Install that bundle as `${CA_FILE}` on `@control,@worker`. `registries.yaml` can stay as-is if `ca_file` already points at `${CA_FILE}`. Then restart:

```plaintext
clush -g control "sudo systemctl restart rke2-server"
clush -g worker "sudo systemctl restart rke2-agent"
```
3. Restart the registry container so it serves the new leaf. Confirm `openssl s_client` with the bundle.
4. Repeat the `crictl --config` pull and the Kubernetes test pod.
5. After every node pulls successfully, install **only** the new cert as `${CA_FILE}` and restart RKE2 the same way.

If a node fails after a `registries.yaml` edit, restore the backup and restart the same service:

```plaintext
sudo cp /etc/rancher/rke2/registries.yaml.pre-ca-trust \
  /etc/rancher/rke2/registries.yaml
sudo systemctl restart rke2-server  # or rke2-agent
```

Restoring a backup that still has `insecure_skip_verify` is only an emergency rollback. Put `ca_file` back before leaving the node in service.