Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Migrating PVCs Across K8s Clusters Within the Same VAST Cluster

Prev Next

This document outlines the steps to migrate PersistentVolumeClaims (PVCs) between two Kubernetes clusters that share the same VAST backend storage. It also includes steps to validate that the application functions correctly post-migration.

Since the PVCs are provisioned by the VAST CSI driver, it's crucial to ensure the Helm chart configuration (used to deploy the CSI plugin) is identical across both clusters to ensure consistent behavior.
Additionally, the StorageClass definitions and associated secrets (e.g., secret.yaml used by the CSI driver) must also be identical across clusters to avoid binding or mounting issues.

Benefits of This Approach

  • No data copying: Leverages shared backend storage for instant access.

  • Minimal downtime: Simply detaches and reattaches the volume.

  • Reversible: Original data remains intact in VAST if rollback is needed.

Instructions

PVC migration steps:

On the Source Kubernetes Cluster :

  1. Identify the PVC and PV in use:

kubectl get pvc <pvc-name> -n <namespace> -o yaml
kubectl get pv <pv-name> -o yaml

Take Note:

  • Volume ID/Handle (e.g., the backend storage identifier, disk name, volume ID, or NFS path).

  • StorageClass used.

  1. Delete the PVC from the source cluster (without deleting the actual backend volume!):
    Before deleting, ensure the PV has a persistentVolumeReclaimPolicy set to Retain, not Delete.

    If it's not, Retain, edit, and patch the PV first.
    Then, safely delete the PVC:

    kubectl delete pvc <pvc-name> -n <namespace>

  2. Prepare the PV for the Destination Cluster:
    On the source cluster, mark the PV as released by removing any claimRef (necessary step):
    The claimRef field in a PV is what binds a PersistentVolumeClaim (PVC) to a specific PersistentVolume (PV).
    When you try to reuse the PV manifest in another cluster, and it contains a claimRef, Kubernetes will reject it (or consider it already bound and unusable), especially since the target PVC probably doesn’t exist yet in the new cluster.

  3. Export the YAML definitions of the PVC and its associated PV.

kubectl get pvc -n ns1 -o yaml > pvc_backup.yaml
kubectl get pv -n ns1 -o yaml > pv_backup.yaml

Copy the generated YAML to a system with access to the destination cluster.

On the destination cluster

  1. Restore on the Destination Cluster

Import the PV into the Destination Cluster:

kubectl_second apply -f pv_backup.yaml

Recreate PVC in Destination Cluster:

kubectl_second apply -f pvc_backup.yaml

Run the following to make sure everything is in place:

kubectl get pvc -n ns1
kubectl get pv
kubectl_second get pvc -n ns1
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
vast-pvc-1   Bound    pvc-8ef55295-bfca-4d33-816d-3d48238f8a4b   1Gi        RWX            vastdata-filesystem   <unset>                 3d20h
kubectl_second get pv -n ns1
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM            STORAGECLASS          VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-8ef55295-bfca-4d33-816d-3d48238f8a4b   1Gi        RWX            Delete           Bound      ns1/vast-pvc-1   vastdata-filesystem   <unset>                          3d20h

PVC bound successfully, mount the PVC successfully, and verified the existing data was not damaged.

sudo mount -t nfs -o vers=3 -vvv 172.21.241.1:/k8s/pvc-8ef55295-bfca-4d33-816d-3d48238f8a4b /mnt/pvc/
ls -ltrh /mnt/pvc/
total 42M
-rw-r--r-- 1 nobody nogroup 8.3M Apr  7 09:06 test-app-1-1
-rw-r--r-- 1 nobody nogroup 8.3M Apr  7 09:06 test-app-1-4
-rw-r--r-- 1 nobody nogroup 8.3M Apr  7 09:06 test-app-1-0
-rw-r--r-- 1 nobody nogroup 8.3M Apr  7 09:06 test-app-1-3
-rw-r--r-- 1 nobody nogroup 8.3M Apr  7 09:06 test-app-1-2


Stop the application on the source side and enable it on the destination side - the application worked as expected.

The test application writes timestamped log files (e.g., test-app-1-0, test-app-1-1, etc.) to the mounted PVC every second.

  • The volume was mounted correctly after migration

  • Data was preserved during the move

  • The application resumed logging without reconfiguration

kubectl_second apply -f app.yaml -n ns1
kubectl_second get pods -n ns1
NAME                                   READY   STATUS    RESTARTS        AGE
csi-vast-controller-669c57699c-5hvzk   5/5     Running   3 (3d21h ago)   3d21h
csi-vast-node-r59zv                    2/2     Running   1 (3d21h ago)   3d21h
test-app-1-0                           1/1     Running   0               3d20h
test-app-1-1                           1/1     Running   0               3d20h
test-app-1-2                           1/1     Running   0               3d20h
test-app-1-3                           1/1     Running   0               3d20h
test-app-1-4                           1/1     Running   0               3d20h
tail /mnt/pvc/test-app-1-1
2025-04-07T09:05:57+00:00
2025-04-07T09:05:58+00:00
2025-04-07T09:05:59+00:00
2025-04-07T09:06:00+00:00
2025-04-07T09:06:01+00:00
2025-04-07T09:06:02+00:00
2025-04-07T09:06:03+00:00
2025-04-07T09:06:04+00:00
2025-04-07T09:06:05+00:00
2025-04-07T09:06:06+00:00

Additional Resources

Create a Kubernetes Secret with VMS User Credentials for VAST CSI Driver

Related articles