--- title: "KubeVirt and VASTCSI 2.3" slug: "kubevirt-and-vastcsi-23" description: "Discover how Kubevirt and VASTCSI simplify VM cloning and stress testing in your Kubernetes clusters—boost efficiency & reliability!" updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/kubevirt-and-vastcsi-23" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # KubeVirt and VASTCSI 2.3 ## Overview This page delves into the practicalities of cloning virtual machines in a Kubernetes/KubeVirt cluster using the VastCSI driver. Our key objectives include testing the VM's cloning functionality in the KubeVirt setup, subjecting the cluster to stress tests involving simultaneous cloning and PVC/snapshots deletion, and finally, providing a detailed summary of the specific environment and commands used for future reference. Let's kick off with a brief rundown of KubeVirt and the technical components in play Important Note: ## Environment Building Blocks | #### KubeVirt | #### CDI | #### DataVolume | #### VM | | --- | --- | --- | --- | | KubeVirt is an extension for Kubernetes that enables the management and orchestration of virtual machines (VMs) alongside containerized workloads. It allows users to run and manage traditional VMs and Kubernetes workloads in a unified environment, providing flexibility and ease of use for diverse workloads. ```bash +---------------------+ \| KubeVirt \| ~~+---------------------+~~ \| Orchestration (K8s) \| +---------------------+ \| Scheduling (K8s) \| +---------------------+ \| Container Runtime \| ~~+---------------------+~~ \| Operating System \| +---------------------+ \| Virtual(kvm) \| ~~+---------------------+~~ \| Physical \| +---------------------+ ``` | The [Containerized Data Importer](https://github.com/kubevirt/containerized-data-importer) (CDI) project provides facilities for enabling [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVCs) to be used as disks for KubeVirt VMs by way of [DataVolumes](https://github.com/kubevirt/containerized-data-importer/blob/main/doc/datavolumes.md). We will import disk images from a web server (Fedora, CentOS) to DataVolumes. ![Image](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/fc444155-9744-4db4-a37e-2ab824b76e9d.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) | In KubeVirt, a `DataVolume` can be associated with a Persistent Volume Claim (PVC), enhancing the management of persistent storage for virtual machines (VMs). A `DataVolume` that is bound to a PVC establishes a connection between the VM and a specific storage resource. This binding allows the VM to utilize the persistent storage defined by the PVC, ensuring that the data associated with the VM persists across restarts and remains independent of the VM's lifecycle.In summary, DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine's launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user. | A virtual machine managed by KubeVirt within a Kubernetes cluster. It runs as a pod, leveraging Kubernetes infrastructure for orchestration and sharing the same underlying host resources. Can be accessed as a console using either `virtctl` commands or via SSH for management and interaction. | Assuming the environment is ready for functionality and the stress test is complete, please proceed to the next section. If you're planning to install Env from scratch, you might want to start by looking at Appendix A - Setup. ### #1 - Clone an RW PVC from a snapshot → Attach to a new VM ### Flow 1. Import a dataVolume from an image into a "golden image" PVC named Fedora. 1. ReadWriteMany - meaning many VMs can use the same PVC. 2. ReadWriteOnce - PVCs can be accessed by only one VM at a time. → Since this is a golden image, and we want to make some modifications such as SSH keys, root password, and add some packages, we'll use this option. Then clone the PVC N times (per VMs). 2. Clone golden PVC, PVC per vm. 3. Create the VM’s pre PVC - write data to the VM. 4. Create a snapshot from a PVC 5. Restore a writable (ReadWriteMany) from the snapshot to a new PVC 6. Attach a new VM to the newly created RW PVC ## ![The diagram illustrates a Kubernetes infrastructure where Persistent Volume Claims (PVCs) and Virtual Machines (VMs) are connected to a "Golden Image" PVC derived from the Fedora dataVolume, and snapshots of this image are used to create additional daily PVCs and VMs for backup purposes.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/965a289a-db92-4624-a77d-e21e5259d7cc.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) ### Functionality Tests The procedure involves creating a virtual machine (VM) from a Persistent Volume Claim (PVC) associated with a golden image. Subsequently, the data is written internally within the VM. A snapshot of its PVC is created, resulting in a new VM derived from it. The objective is to determine the image's writability and confirm the persistence of previously written data. | **Important Notes:** - > [!NOTE] > VAST Catalog must be disable in order to create writable snapshots - > [!NOTE] > With trash folder enable, PVC are deleted from the storage by the CSI driver. - > [!NOTE] > no_root_squash view policy is required for k8s/kubeVirt control plane | | --- | ### Method First, let’s import a Fedora image from a dataVolume to a PVC using the following YAML, which includes: - kind: DataVolume - storageClassName: vastdata-filesystem (csi storageClass name) - access mode: ReadOnlyMany (keep it immutable; next we'll create RW pvc) - PVC size: 30G. Writable YAML template ```yaml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata:  name: "fedora"  annotations:    cdi.kubevirt.io/storage.bind.immediate.requested: "true" spec:  source:    http:      url: "https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"  pvc:    storageClassName: vastdata-filesystem    accessModes:    - ReadOnlyMany    resources:      requests:        storage: "30Gi" ``` Applying the data volume will create a PVC. A temporary pod with the prefix **importer** will be created; its job is to import the image to a PVC. We can check the status. The new data volume named Fedora isn't ready until the job is done. ```bash root@v55lg2:/opt/LLNL/datavolumes# kubectl apply -f dv_fedora.yaml datavolume.cdi.kubevirt.io/fedora created root@v55lg2:/opt/LLNL/datavolumes# kubectl get pods NAME                                                  READY   STATUS    RESTARTS   AGE importer-prime-68d84758-a49e-4996-88b6-9ac52a36a1ee   1/1     Running   0          3m22s root@v55lg2:/opt/LLNL/datavolumes# kubectl get datavolumes.cdi.kubevirt.io NAME     PHASE              PROGRESS   RESTARTS   AGE fedora   ImportInProgress   66.77%                3m4s ``` Monitor the process by checking the importer pod logs. When complete, the Fedora datavolume and PVC are ready to use. ```plaintext root@v55lg2:/opt/LLNL/datavolumes# kubectl logs -f importer-prime-68d84758-a49e-4996-88b6-9ac52a36a1ee I0122 12:32:46.146124       1 importer.go:103] Starting importer I0122 12:32:46.147086       1 importer.go:176] begin import process I0122 12:32:47.919697       1 data-processor.go:356] Calculating available size I0122 12:32:47.920485       1 data-processor.go:368] Checking out file system volume size. I0122 12:32:47.921197       1 data-processor.go:376] Request image size not empty. I0122 12:32:47.921221       1 data-processor.go:381] Target size 30Gi. I0122 12:32:48.061179       1 data-processor.go:255] New phase: TransferDataFile I0122 12:32:48.063593       1 util.go:194] Writing data... I0122 12:32:49.061893       1 prometheus.go:75] 0.02 I0122 12:32:52.063418       1 prometheus.go:75] 0.75 I0122 12:32:53.064514       1 prometheus.go:75] 1.46 I0122 12:32:55.076411       1 prometheus.go:75] 2.50 ... I0122 12:37:36.362152       1 prometheus.go:75] 99.97 I0122 12:37:37.362659       1 prometheus.go:75] 99.98 I0122 12:37:48.369059       1 prometheus.go:75] 99.99 I0122 12:37:49.370083       1 prometheus.go:75] 100.00 I0122 12:37:52.526490       1 data-processor.go:255] New phase: Resize E0122 12:37:52.714510       1 prlimit.go:155] failed to kill the process; os: process already finished W0122 12:37:52.714615       1 data-processor.go:338] Available space less than requested size, resizing image to available space 30440161280. I0122 12:37:52.714629       1 data-processor.go:349] Expanding image size to: 30440161280 E0122 12:37:52.814279       1 prlimit.go:155] failed to kill the process; os: process already finished I0122 12:37:52.814300       1 data-processor.go:261] Validating image E0122 12:37:52.869491       1 prlimit.go:155] failed to kill the process; os: process already finished I0122 12:37:52.870161       1 data-processor.go:255] New phase: Complete I0122 12:37:52.870447       1 importer.go:223] Import Complete root@v55lg2:/opt/LLNL/datavolumes# kubectl get datavolumes.cdi.kubevirt.io NAME     PHASE       PROGRESS   RESTARTS   AGE fedora   Succeeded   100.0%                6m10s root@v55lg2:/opt/LLNL/datavolumes# kubectl get pvc NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE fedora   Bound    pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699   30Gi       ROX            vastdata-filesystem   6m31s ``` We created this "golden" PVC with RO mode. ### Clone a PVC Now it's time to clone this PVC into a new "golden-pvc", and modify it as we see fit. In this case, I created a VM with the following tweaks: - Filled its belly with 20 GB of data to test some real workload while cloning - Ran the elbencho benchmark @boot to continuously generate some I/O as if it were a VDI, for example - Install some required packages - Modify the SSH config and root password We'll use this PVC to create additional PVCs, one for each VM, but first, let's start by cloning the PVC Fedora → golden-fedora (rwm). Note that now we're using accessMode: ReadWriteMany Clone PVC to PVC YAML: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata:    name: golden-fedora    namespace: default spec:  accessModes:  - ReadWriteMany  storageClassName: vastdata-filesystem  resources:    requests:      storage: 30Gi  dataSource:    kind: PersistentVolumeClaim    name: fedora ``` - **Note**: cloning a PVC does not create a snapshot. Create the VM VMFedora from the PVC golden-pvc. Note the PVC source name, spec, and SSH keys (optional). ```yaml apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata:  creationTimestamp: 2018-07-04T15:03:08Z  generation: 1  labels:    kubevirt.io/os: linux  name: vmfedora spec:  running: true  template:    metadata:      creationTimestamp: null      labels:        kubevirt.io/domain: vm1    spec:      domain:        cpu:          cores: 2        devices:          disks:          - disk:              bus: virtio            name: disk0          - cdrom:              bus: sata              readonly: true            name: cloudinitdisk        machine:          type: q35        resources:          requests:            memory: 1024M      volumes:      - name: disk0        persistentVolumeClaim:          claimName: golden-fedora      - cloudInitNoCloud:          userData: |            #cloud-config            hostname: vmfedora            ssh_pwauth: True            disable_root: false            ssh_authorized_keys:            - ssh-rsa        name: cloudinitdisk ``` We'll create a new VM called Fedora, SSH, and write something to it. ```bash root@v55lg2:/opt/LLNL/vms# kubectl apply -f vm_fedora.yaml virtualmachine.kubevirt.io/vmfedora created root@v55lg2:/opt/LLNL/process# kubectl get vms NAME       AGE   STATUS    READY vmfedora   52s   Running   True root@v55lg2:/opt/LLNL/process# kubectl get vmis NAME       AGE   PHASE     IP          NODENAME   READY vmfedora   56s   Running   10.44.0.8   v55lg3     True root@v55lg2:/opt/LLNL/process# kubectl get pods NAME                           READY   STATUS    RESTARTS   AGE virt-launcher-vmfedora-z9779   2/2     Running   0          96s root@v55lg2:/opt/LLNL/process# ssh 10.44.0.8 [root@vmfedora ~]# date > /home/`date +%d.%m_%H.%M.%S`.txt           [root@vmfedora ~]# cat /home/22.01_14.02.14.txt Mon Jan 22 02:02:14 PM UTC 2024 ``` As explained at the beginning of the section, I'll tweak this VM a little bit, and to make a long story short, here's a set of commands that is very high-level: ```bash ### ssh to the vm (or using virtctl cmd) ### elbencho yum dependencies, make and deploy yum install -y cronie cronie-anacron crontabs boost-devel gcc-c++ git libaio-devel make ncurses-devel numactl-devel rpm-build vim git clone https://github.com/breuner/elbencho.git cd elbencho make rpm yum install /root/elbencho/packaging/RPMS/x86_64/elbencho-3.0.5-1.x86_64.rpm -y ### Optional : ### Create a bash script to generate some I/O with the following elbencho command, and add it to crontab @reboot /usr/bin/elbencho --write --mkdirs --threads 2 --dirs 3 --files 4 --size 1m --block 1m /home/elben_data --infloop ``` Once the VM is ready, we'll shut it down and create a snapshot. Let's name it **snap-fedora-daily** Snapshot YAML: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata:  name: snap-fedora-daily spec:  volumeSnapshotClassName: vastdata-snapshot  source:    persistentVolumeClaimName: golden-fedora ``` The snapshot is ready to clone. ```plaintext root@v55lg2:/opt/LLNL/process# kubectl get volumesnapshot NAME                READYTOUSE   SOURCEPVC       SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT                                    CREATIONTIME   AGE snap-fedora-daily   true         golden-fedora                           0             vastdata-snapshot   snapcontent-da56675d-785e-4220-ae09-48c9a1d86440   7s             8s ``` Now, let's create two writable PVCs from the snapshot, named **fedora-daily01 & fedora-daily02.** The clone-from-snapshot process does not erase the snapshot from storage, so we can reuse it. PVC from snapshot YAML: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata:  name: fedora-daily01 spec:  storageClassName: vastdata-filesystem  dataSource:    name: snap-fedora-daily    kind: VolumeSnapshot    apiGroup: snapshot.storage.k8s.io  accessModes:    - ReadWriteMany  resources:    requests:      storage: 30Gi ``` Monitor global snapshot creation status from the cluster UI : ![The image displays a screenshot from a backup and disaster recovery interface, specifically focusing on "Global Snapshot Clones." It shows details such as snapshot names, source storage systems (CSI), target paths, status, synchronization progress, and associated source snapshots, indicating successful completion with 100% sync progress for both entries listed.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/9db9206f-a676-40b8-8a96-dcb3b2be632f.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) We have x3 PVCs: the imported data volume, the golden PVC, and the new daily PVC we created from the snapshots. In addition, if we examine the mounted folder for the VAST Kubevirt images, the data fills as new PVCs are created. ```bash root@v55lg2:/opt/LLNL/process# kubectl get pvc NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE fedora           Bound    pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699   30Gi       ROX            vastdata-filesystem   171m fedora-daily01   Bound    pvc-22907e3f-2298-40a7-bb5f-3f0b14b4241b   30Gi       RWX            vastdata-filesystem   10s fedora-daily02   Bound    pvc-52300027-56d6-415d-9c2b-7cf8cd4cde19   30Gi       RWX            vastdata-filesystem   7s golden-fedora    Bound    pvc-d1cc5448-a8b8-4ea0-b3ea-59ac14dee5cb   30Gi       RWX            vastdata-filesystem   103m #### Du -sh ####### root@v55lg2:/opt/LLNL/process# du -sh /mnt/v55/kubevirt/* 5.0G /mnt/v55/kubevirt/pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699 15G /mnt/v55/kubevirt/pvc-22907e3f-2298-40a7-bb5f-3f0b14b4241b    <-- PVC is ready when reaches 29G 11G /mnt/v55/kubevirt/pvc-52300027-56d6-415d-9c2b-7cf8cd4cde19    <-- PVC is ready when reaches 29G 29G /mnt/v55/kubevirt/pvc-d1cc5448-a8b8-4ea0-b3ea-59ac14dee5cb ``` Finally, we'll create new VMs called daily-fedora01 && daily-fedora02, access them, and write more stuff to them. ```bash root@v55lg2:/opt/LLNL/process# kubectl get vmis NAME             AGE   PHASE     IP          NODENAME   READY daily-fedora01   18s   Running   10.36.0.8   v55lg4     True daily-fedora02   14s   Running   10.44.0.9   v55lg3     True vmfedora         50m   Running   10.44.0.8   v55lg3     True root@v55lg2:/opt/LLNL/process# kubectl get pods -o wide NAME                                 READY   STATUS    RESTARTS   AGE   IP          NODE     NOMINATED NODE   READINESS GATES virt-launcher-daily-fedora01-jsrwr   2/2     Running   0          26s   10.36.0.8   v55lg4             1/1 virt-launcher-daily-fedora02-wk9d5   2/2     Running   0          22s   10.44.0.9   v55lg3             1/1 virt-launcher-vmfedora-pbmwz         2/2     Running   0          50m   10.44.0.8   v55lg3             1/1 root@v55lg2:/opt/LLNL/process# ssh 10.44.0.9 root@daily-fedora02 ~]# date >> /home/22.01_14.02.14.txt [root@daily-fedora02 ~]# cat /home/22.01_14.02.14.txt Mon Jan 22 02:02:14 PM UTC 2024 Mon Jan 22 03:33:37 PM UTC 2024 [root@daily-fedora02 ~]# df -h Filesystem      Size  Used Avail Use% Mounted on devtmpfs        4.0M     0  4.0M   0% /dev tmpfs           459M     0  459M   0% /dev/shm tmpfs           184M  3.0M  181M   2% /run /dev/vda5        28G   27G  703M  98% / tmpfs           459M     0  459M   0% /tmp /dev/vda5        28G   27G  703M  98% /home /dev/vda2       966M   53M  848M   6% /boot /dev/vda3       100M  9.9M   90M  10% /boot/efi tmpfs            92M     0   92M   0% /run/user/0 [root@daily-fedora02 ~]# ps aufx|grep elbencho root         907  0.0  0.2   6216  2188 pts/0    S+   15:34   0:00              \_ grep --color=auto elbencho root         831  197  0.7 156028  7088 ?        Sl   15:29  10:49          \_ /usr/bin/elbencho --write --mkdirs --threads 2 --dirs 3 --files 4 --size 1m --block 1m /home/elben_data --infloop ``` Nice utilization and Data flow distribution : ### ![The image displays a data flow visualization with protocols, users, and hosts interconnected in real-time. The flow is sorted by bandwidth (BW) in megabytes per second (MB/s), showing various network segments such as VIPs and Cnodes handling different transactions simultaneously.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/ab7ec4dc-cc8e-4491-94c9-3a4ea4553804.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) ## Stress Test We'll measure how the system handles cloning x50 PVCs and attaching the I/O-generating VMs to each one. Then, initiate a snapshot from one PVC and clone x30 new PVCs from it. Finally, attach x30 new VMs to the newly created PVCs. Creating x50 PVC from the golden image at once, each PVC 30GB total of 1.5TB. | Data Flow | DNODES writes/read bandwidth | Cluster IOPS and Latency | Cluster Migrate Data Reduction Ratio | | --- | --- | --- | --- | | ![The data flow visualization highlights multiple network nodes, including user "nfsnobody," host addresses (e172.25.55.121 and 17.225..552,), virtual IP (VIP) addresses, cnode-3-x, and the default view directory. The connections illustrate various communication paths between these entities in real-time, sorted by bandwidth usage in MB/s.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/378fe433-35bf-4da8-8167-a50db3535de9.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) | ![The image displays bandwidth analysis for DNodes dnode-3-106 and dnode-dn2, showing the read (green) and write (redd) bandwidths over time within a 10-minute period. The interface includes options to choose predefined analytics graphs such as CNode IOPS, CPU load average, or latency for further detailed performance insights.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/c28a102d-ffe4-427c-9a22-ff606290ed61.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) | ![The image displays real-time performance metrics for a cluster named "vast55-kfs," focusing on CPU load and IOPS, with latency also monitored. The graph visualizes fluctuations over an interval from 16:00 to 17:02, highlighting periods of high activity and low idle times.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/74661ac8-e75f-4ec2-9eb3-9bf549f4b6bf.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) | ![The graph illustrates the data reduction ratio over time within the cluster vast55-kfs, with pronounced peaks indicating significant data compression events occurring between 16:20 and 17:00 UTC+02:0.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/30a0c348-f7b1-4238-bec2-ddf8b053d756.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) | Creating x50 PVC, roughly at 1.2GBps writes, all data is similar, takes roughly 28 minutes. ```bash Every 2.0s: du -sh /mnt/v55/kubevirt/*|sort -nk1                                        v55lg2: Wed Jan 17 14:37:09 2024 5.1G    /mnt/v55/kubevirt/pvc-44e28051-bb71-465f-860b-968b94e51f4b.  <----- Last PVC ready with 29G provisioned 13G     /mnt/v55/kubevirt/pvc-f0d01b22-9407-43d4-bfc6-d2c2f14fea61 16G     /mnt/v55/kubevirt/pvc-39ef66cc-7be0-40b2-b24a-88e64a682626 20G     /mnt/v55/kubevirt/pvc-4a13201a-cc18-48ec-8a7a-92bc166f2989 22G     /mnt/v55/kubevirt/pvc-4ab3012e-5f66-4bd6-937e-f150af3d69df 24G     /mnt/v55/kubevirt/pvc-b8595c12-d018-40e9-acda-7272e414f774 26G     /mnt/v55/kubevirt/pvc-86733fe8-ed5c-440f-bd51-6087bfdec940 27G     /mnt/v55/kubevirt/pvc-af7620f7-19f1-4961-a36d-059b18a18719 28G     /mnt/v55/kubevirt/pvc-07911809-15c0-45a2-b4d6-23a8af13b150 28G     /mnt/v55/kubevirt/pvc-603793aa-4177-4369-8a3c-627bafb713aa 29G     /mnt/v55/kubevirt/pvc-0ca42e3f-5f74-491b-85b9-ff43e3da82d1 29G     /mnt/v55/kubevirt/pvc-0cdd5162-c073-4e15-bb0e-f03dd02d69cf ... ``` Attaching VM to each PVC. VMs are running, each VM is attached to its own PVC. ```bash root@v55lg2:~# kubectl get pods -o wide NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES virt-launcher-vmfedora01-h9hjw   2/2     Running   0          94m   10.36.0.32   v55lg4             1/1 virt-launcher-vmfedora02-s7f7h   2/2     Running   0          94m   10.44.0.8    v55lg3             1/1 virt-launcher-vmfedora03-htsw5   2/2     Running   0          94m   10.44.0.9    v55lg3             1/1 virt-launcher-vmfedora04-7kpx8   2/2     Running   0          94m   10.36.0.11   v55lg4             1/1 virt-launcher-vmfedora05-q8pnz   2/2     Running   0          94m   10.36.0.19   v55lg4             1/1 virt-launcher-vmfedora06-z4cpp   2/2     Running   0          94m   10.44.0.10   v55lg3             1/1 virt-launcher-vmfedora07-bgrfg   2/2     Running   0          94m   10.44.0.11   v55lg3             1/1 virt-launcher-vmfedora08-wl8vr   2/2     Running   0          94m   10.36.0.16   v55lg4             1/1 virt-launcher-vmfedora09-whdts   2/2     Running   0          94m   10.36.0.20   v55lg4             1/1 virt-launcher-vmfedora10-cfghk   2/2     Running   0          94m   10.44.0.23   v55lg3             1/1 virt-launcher-vmfedora11-c9g2r   2/2     Running   0          94m   10.44.0.12   v55lg3             1/1 virt-launcher-vmfedora12-pvtxx   2/2     Running   0          94m   10.44.0.13   v55lg3             1/1 virt-launcher-vmfedora13-ttckk   2/2     Running   0          94m   10.36.0.30   v55lg4             1/1 virt-launcher-vmfedora14-fktgb   2/2     Running   0          93m   10.36.0.21   v55lg4             1/1 virt-launcher-vmfedora15-rsmz2   2/2     Running   0          93m   10.44.0.14   v55lg3             1/1 virt-launcher-vmfedora16-hqf4n   2/2     Running   0          93m   10.44.0.15   v55lg3             1/1 virt-launcher-vmfedora17-4b527   2/2     Running   0          93m   10.36.0.22   v55lg4             1/1 virt-launcher-vmfedora18-5nrwd   2/2     Running   0          93m   10.36.0.25   v55lg4             1/1 virt-launcher-vmfedora19-v7h8d   2/2     Running   0          93m   10.44.0.17   v55lg3             1/1 virt-launcher-vmfedora20-phtlz   2/2     Running   0          93m   10.44.0.24   v55lg3             1/1 virt-launcher-vmfedora21-2lrbj   2/2     Running   0          93m   10.44.0.16   v55lg3             1/1 virt-launcher-vmfedora22-l5drd   2/2     Running   0          93m   10.36.0.17   v55lg4             1/1 virt-launcher-vmfedora23-45drd   2/2     Running   0          93m   10.36.0.26   v55lg4             1/1 virt-launcher-vmfedora24-pksk8   2/2     Running   0          93m   10.36.0.14   v55lg4             1/1 virt-launcher-vmfedora25-k4c7x   2/2     Running   0          93m   10.36.0.28   v55lg4             1/1 virt-launcher-vmfedora26-rvqcl   2/2     Running   0          93m   10.44.0.22   v55lg3             1/1 virt-launcher-vmfedora27-qfv6b   2/2     Running   0          93m   10.36.0.18   v55lg4             1/1 virt-launcher-vmfedora28-mdhdj   2/2     Running   0          93m   10.44.0.18   v55lg3             1/1 virt-launcher-vmfedora29-6mp2v   2/2     Running   0          93m   10.36.0.23   v55lg4             1/1 virt-launcher-vmfedora30-wcsgh   2/2     Running   0          93m   10.44.0.25   v55lg3             1/1 virt-launcher-vmfedora31-bjkmg   2/2     Running   0          93m   10.44.0.19   v55lg3             1/1 virt-launcher-vmfedora32-79dhq   2/2     Running   0          93m   10.36.0.13   v55lg4             1/1 virt-launcher-vmfedora33-gfblt   2/2     Running   0          92m   10.36.0.24   v55lg4             1/1 virt-launcher-vmfedora34-kgxc2   2/2     Running   0          92m   10.36.0.27   v55lg4             1/1 virt-launcher-vmfedora35-zrf5n   2/2     Running   0          92m   10.44.0.21   v55lg3             1/1 virt-launcher-vmfedora36-4zwb4   2/2     Running   0          92m   10.44.0.20   v55lg3             1/1 virt-launcher-vmfedora37-5g9x8   2/2     Running   0          92m   10.36.0.8    v55lg4             1/1 virt-launcher-vmfedora38-b4sgg   2/2     Running   0          92m   10.44.0.27   v55lg3             1/1 virt-launcher-vmfedora39-rlhxr   2/2     Running   0          92m   10.44.0.26   v55lg3             1/1 virt-launcher-vmfedora40-d5lk5   2/2     Running   0          92m   10.36.0.29   v55lg4             1/1 virt-launcher-vmfedora41-cxr4q   2/2     Running   0          92m   10.36.0.31   v55lg4             1/1 virt-launcher-vmfedora42-n7xlf   2/2     Running   0          92m   10.44.0.28   v55lg3             1/1 virt-launcher-vmfedora43-x84j7   2/2     Running   0          92m   10.44.0.30   v55lg3             1/1 virt-launcher-vmfedora44-j7g5k   2/2     Running   0          92m   10.36.0.12   v55lg4             1/1 virt-launcher-vmfedora45-skbxl   2/2     Running   0          92m   10.44.0.31   v55lg3             1/1 virt-launcher-vmfedora46-pg6g2   2/2     Running   0          92m   10.44.0.29   v55lg3             1/1 virt-launcher-vmfedora47-7plzq   2/2     Running   0          92m   10.36.0.10   v55lg4             1/1 virt-launcher-vmfedora48-rfshb   2/2     Running   0          92m   10.36.0.15   v55lg4             1/1 virt-launcher-vmfedora49-7hgrg   2/2     Running   0          92m   10.36.0.9    v55lg4             1/1 virt-launcher-vmfedora50-8vb5p   2/2     Running   0          92m   10.44.0.32   v55lg3             1/1 ``` ### Clone from snapshot Taking a snapshot from the first PVC created: "pvc-fedora01", and creating x20 new PVC + VMs from it. While snapshot creation is completed immediately, creating x20 PVCs has a similar rate of 1.3 GB/s, as it took 13 minutes to create 600GB of x20 30GB PVCs. ```bash root@v55lg2:/opt/LLNL/snapshots# kubectl apply -f snap_fedora.yaml volumesnapshot.snapshot.storage.k8s.io/snap-fedora01 created root@v55lg2:/opt/LLNL/snapshots# kubectl get volumesnapshot NAME            READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT                                    CREATIONTIME   AGE snap-fedora01   true         pvc-fedora01                           0             vastdata-snapshot   snapcontent-2182a4 root@v55lg2:/opt/LLNL/vms# kubectl apply -f vm_fedora_fromsnap.yaml virtualmachine.kubevirt.io/vmfedora-fromsnap created root@v55lg2:/opt/LLNL/vms# kubectl get vms|grep snap vmfedora-fromsnap   17s    Running   True root@v55lg2:/opt/LLNL/vms# kubectl get vmis|grep snap vmfedora-fromsnap   2m1s   Running   10.44.0.33   v55lg3     True root@v55lg2:/opt/LLNL/vms# ssh 10.44.0.33 Last login: Wed Jan 17 13:29:46 2024 [[root@vmfedora-fromsnap ~]# touch /home/snapwrite [root@vmfedora-fromsnap ~]# ls -l /home total 409600 drwxr-xr-x. 1 root   root         144 Jan 17 13:27 elbencho drwx------. 1 fedora fedora        72 Jan 17 12:44 fedora -rw-r--r--. 1 root   root   104857600 Jan 17 13:19 files100mb1 -rw-r--r--. 1 root   root   104857600 Jan 17 13:19 files100mb2 -rw-r--r--. 1 root   root   104857600 Jan 17 13:19 files100mb3 -rw-r--r--. 1 root   root   104857600 Jan 17 13:19 files100mb4 drwxr-xr-x. 1 root   root           0 Jan 17 12:45 oz-vm1 -rw-r--r--. 1 root   root           0 Jan 17 17:49 snapwrite ``` ### Mass deletion of all PVCs and snapshots The best practice to remove PVC: - Shutdown associated VM - Delete the VM → this will also terminate its running pod - Delete the PVC Massive deletion of x70 PVC took less than 1 minute, with no queue detected on the cluster while checking the following metrics: - NVRamHashMapMetrics,name=new_token_mapper,block_used_space_percent__avg - TreeUnlinkerMetrics,unlinks_count__sum - TreeUnlinkerMetrics,unlinks_count__sum_squares - TreeUnlinkerMetrics,unlinks_count__max - TreeUnlinkerMetrics,unlinks_count__rate - TreeUnlinkerMetrics,unlinks_count__std - TreeUnlinkerMetrics,unlinks_count__avg ### Thoughts and tl;dr 1. VM/PVC Similarity is very impressive - ratio of **30.1:1** **![The data storage in the "/kubevirt" directory within the root system is highly efficient, achieving a data reduction ratio of 30:1 with only 45.264 GB usable and unique capacity out of the total logical 1.362 TB available. The "data" subdirectory shows significantly less usage at 87.3%, indicating potential optimization opportunities for this directory to match its parent's efficiency.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/8e947a0e-d0ac-4d96-968f-6b32d40186e2.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D)** 2. Writable snapshots cannot be initiated without disabling the VAST Catalog. 3. Deletion with CSI - **enable trash folder.** ## #2: Direct Clone a VM Cloning a VM is basically the fastest way to clone, but snapshots and PVCs are automatically deleted from the storage. ### Flow: 1. Once a VM is created and running from a golden image, we can clone it directly. 2. Clone the VM to a new VM. (Daily3 → Daily4) 3. The snapshot and PVC are automatically created during the process. ## ![The diagram illustrates the flow of data and virtual machine (VM) management within VAST, showcasing how PVCs and VMs interact with snapshots and data volumes to maintain consistency across different systems daily.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/071bd386-8da8-41bc-96a0-a0440d203467.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) ### Functionality This, in fact, can speed up and automate the cloning of VMs, since the underlying snapshot and the associated new PVC (restore) are created for you. After the new VM is created, the snapshot is automatically deleted. This use case should not be used against ransomware attacks, for example, by saving immutable snapshots, since they are removed. ### Method ```bash root@v55lg2:/opt/LLNL/clone# kubectl create -f clone.yaml virtualmachineclone.clone.kubevirt.io/testclone created root@v55lg2:/opt/LLNL/clone# cat clone.yaml kind: VirtualMachineClone apiVersion: "clone.kubevirt.io/v1alpha1" metadata:  name: testclone spec:  # source & target definitions  source:    apiGroup: kubevirt.io    kind: VirtualMachine    name: vmfedora01  target:    apiGroup: kubevirt.io    kind: VirtualMachine    name: vm-clone-target  # other identity stripping specs:  newMacAddresses:    interfaceName: "00-11-22"  newSMBiosSerial: "new-serial" root@v55lg2:/opt/LLNL/clone# kubectl get volumesnapshot NAME                                                           READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT                                    CREATIONTIME   AGE vmsnapshot-fd042778-7f39-4a86-8a8b-47784c2b7dd5-volume-disk0   true         fedora                              0             vastdata-snapshot   snapcontent-5b854b90-d157-4b84-bf02-74936adbb407   21s            22s root@v55lg2:/opt/LLNL/clone# kubectl get virtualmachine NAME                 AGE     STATUS    READY fedora-clone-pvc01   5m25s   Running   True vm-clone-target      91s     Running   True vmfedora01           73m     Running   True root@v55lg2:/opt/LLNL/clone# kubectl get volumesnapshot No resources found in default namespace. root@v55lg2:/opt/LLNL/clone# kubectl get pvc NAME                                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE clone-of-defora-pvc                                  Bound    pvc-fbeba0fd-c1bc-42ad-8878-b125ecdb96df   20Gi       RWX            vastdata-filesystem   11m fedora                                               Bound    pvc-6f82df62-f93a-4d12-8331-962da29282d8   20Gi       RWX            vastdata-filesystem   93m restore-6af6ca21-6711-4024-9cb2-f4fe61f70663-disk0   Bound    pvc-c0abba80-f8d2-40fc-84d0-55a94fff0b23   20Gi       RWX            vastdata-filesystem   2m9s ``` ## Appendix A - Environment setup 1. Create a K8s cluster with at least x1 worker node. This setup was tested with version 1.28.2 and the containerized container runtime. 2. Install the vastcsi 2.3 driver according to the [docs](/docs-5-5/docs/deploying-vast-csi-driver). 3. Ensure the VAST view for virtual machines is configured with no_root_squash on the k8s control plane host. 4. Validate hardware virtualization support: 1. VMware (for lab/testing) - enable "Expose hardware-assisted virtualization to the guest OS" in CPU options. 2. Bare Metal - BIOS ![The image displays an advanced configuration menu in BIOS, offering detailed settings such as processor information and system performance adjustments. Users can navigate through these options using keyboard shortcuts like F10 to save changes or F9 to reset defaults.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/2a6929a8-c65e-41c5-b0a9-87f2264d436f.jpg?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) ![The image displays the BIOS settings screen from an Intel motherboard, showing detailed processor configuration information including cache sizes and versions such as Intel Xeon E5 processors running at 2.40GHz. The screen also highlights virtualization options like Intel Virtualization Technology (VT), which is enabled in this setup.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/8bbc24ee-7cc8-4039-9940-7f3f8e5cc463.jpg?sv=2026-02-06&spr=https&st=2026-11-08T09%3A16%3A09Z&se=2026-11-08T10%3A16%3A09Z&sr=c&sp=r&sig=13NaA2XFJxKW6YBOYdhzbikE54EsKlvI%2F9dr57I5Da8%3D) 3. Linux OS - Install KVM and libvirt: ```bash apt-get install qemu-kvm libvirt-bin ``` Validate hardware support (don't forget BIOS conf in 2.b) to ensure that your host are capable of running virtualization workloads: ```plaintext $ virt-host-validate qemu  QEMU: Checking for hardware virtualization                                 : PASS  QEMU: Checking if device /dev/kvm exists                                   : PASS  QEMU: Checking if device /dev/kvm is accessible                            : PASS  QEMU: Checking if device /dev/vhost-net exists                             : PASS  QEMU: Checking if device /dev/net/tun exists                               : PASS ... ``` ## Appendix B - Deploy KubeVirt Deploy KubeVirt according to this guide: ```plaintext Deploy KubeVirt Deploy the KubeVirt operator using the latest KubeVirt version. An Operator is a method of packaging, deploying, and managing a Kubernetes application. A Kubernetes application is one that is deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. You can think of Operators as the runtime that manages this type of application on Kubernetes. If you want to learn more about Operators you can check the Kubernetes documentation Here, we query GitHub's API to get the latest available release: (click on the text to automatically execute the commands on the console): export KUBEVIRT_VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | jq -r .tag_name) echo $KUBEVIRT_VERSION Run the following command to deploy the KubeVirt Operator: kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-operator.yaml Now deploy KubeVirt by creating a Custom Resource that will trigger the 'operator' reaction and perform the deployment: kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-cr.yaml Next, we need to configure KubeVirt to use software emulation for virtualization. This is necessary for the course environment but results in poor performance so avoid this step in production environments. kubectl -n kubevirt patch kubevirt kubevirt --type=merge --patch '{"spec":{"configuration":{"developerConfiguration":{"useEmulation":true}}}}' Install Virtctl While we are waiting for the KubeVirt operator to start up all its Pods, we can take some time to download the client we will need to use in the next step. virtctl is a client utility that helps interact with VM's (start/stop/console, etc): wget -O virtctl https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/virtctl-${KUBEVIRT_VERSION}-linux-amd64 chmod +x virtctl Wait for KubeVirt deployment to finalize Let's check the deployment: kubectl get pods -n kubevirt Once it's ready, it will show something similar to: controlplane $ kubectl get pods -n kubevirt NAME                               READY     STATUS    RESTARTS   AGE virt-api-7fc57db6dd-g4s4w          1/1       Running   0          3m virt-api-7fc57db6dd-zd95q          1/1       Running   0          3m virt-controller-6849d45bcc-88zd4   1/1       Running   0          3m virt-controller-6849d45bcc-cmfzk   1/1       Running   0          3m virt-handler-fvsqw                 1/1       Running   0          3m virt-operator-5649f67475-gmphg     1/1       Running   0          4m virt-operator-5649f67475-sw78k     1/1       Running   0          4m As there are multiple deployments involved, the best way to determine whether the operator is fully installed is to check the operator's Custom Resource itself: kubectl -n kubevirt get kubevirt Once fully deployed, this will look like: NAME      AGE   PHASE kubevirt  3m    Deployed Now everything is ready to continue and launch a VM. Deploy a Virtual Machine The command below applies a YAML definition of a virtual machine into the current Kubernetes environment, defining the VM name, the resources required (disk, CPU, memory), etc. You can take a look at the vm.yaml file if you have interest in knowing more about a virtual machine definition: kubectl apply -f https://kubevirt.io/labs/manifests/vm.yaml We are creating a Virtual Machine in the same way as we would create any other Kubernetes resource thanks to the KubeVirt operator in our environment. Now we have a Virtual Machine as a Kubernetes resource. After the vm resource has been created, you can manage the VMs with standard 'kubectl' commands: kubectl get vms kubectl get vms -o yaml testvm | grep -E 'running:.*|$' Notice from the output that the VM is not running yet. To start a VM, use virtctl with the start verb: ./virtctl start testvm Again, check the VM status: kubectl get vms A VirtualMachine resource contains a VM's definition and status. An instance of a running VM has an additional associated resource, a VirtualMachineInstance. Once the VM is running you can inspect its status: kubectl get vmis Once it is ready, the command above will print something like: NAME      AGE       PHASE     IP           NODENAME testvm    1m        Running   10.32.0.11   controlplane Access a VM (serial console & vnc) Now that the VM is running you can access its serial console: WARNING: in some OS and browser environments you may not be able to escape the serial console in this course. NOTE: ^] means: press the "CTRL" and "]" keys to escape the console. ./virtctl console testvm If you opened the serial console within the Killercoda course environment and you can't escape from it by pressing ^], you can click on the + at the top of the terminal window to start a new shell. You should be able to continue with the following steps in the shutdown and cleanup section. In environments where VNC client access is available, the graphical console of a VM can be accessed with the virtctl vnc command. Shutdown and cleanup As with starting, stopping a VM also may be accomplished with the virtctl command: ./virtctl stop testvm Finally, the VM can be deleted as any other Kubernetes resource using kubectl: kubectl delete vms testvm ``` 5. Deploy CDI ```plaintext Install CDI Introduction to Containerized Data Importer CDI is a utility designed to import Virtual Machine images for use with Kubevirt. The process for importing disk images into KubeVirt involves a helper Custom Resource (CR) called a DataVolume (DV). DataVolumes handle the provisioning of PersistentVolumeClaims (PVCs) for KubeVirt. A custom controller watches for new DVs, and when one is discovered, starts an import process to create a new PVC into which it will import the desired content. In this exercise we start by deploying the CDI operator. Then, we import a CirrOS disk image and use it to start a VM. Wait for KubeVirt to deploy The setup for this scenario includes installation of KubeVirt and the virtctl utility. Before we can start, we need to wait for the KubeVirt initialization script to run. (a command prompt will appear once everything is ready). Check Default Storage Class Before we can install CDI, we have some prerequisites, namely a supported storage class and provisioner. For this example, we make use of the local-path StorageClass provided by k3s which provisions PVCs using node local storage. This is an option for proof of concept exercises like this one, but should not be used in production because it does not support accessing a volume across nodes. kubectl get storageclass Install the Containerized Data Importer Next we determine the latest version of CDI and apply both the Operator and the CR that starts the deployment: export VERSION=$(curl -Ls https://github.com/kubevirt/containerized-data-importer/releases/latest | grep -m 1 -o "v[0-9]\.[0-9]*\.[0-9]*") echo $VERSION Deploy the operator (and scale its replicas down to one due to the resource limitations of the environment): kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml kubectl -n cdi scale deployment/cdi-operator --replicas=1 Create CRD to trigger operator deployment of CDI: kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml Check status of CDI deployment. It may take some time before the cdi "PHASE" reads "Deployed" kubectl get cdi -n cdi To have kubectl do the checking for you and let you know when the operator finishes its deployment, use the wait command: kubectl wait -n cdi --for=jsonpath='{.status.phase}'=Deployed cdi/cdi At times, the environment this scenario runs in may be under heavy load; if the above command errors out, wait a moment and re-run. Review the "cdi" pods that were added. kubectl -n cdi get pods ``` ### References: - - [KubeVirt Architecture and Docs](https://kubevirt.io/user-guide/architecture/) - [KubeVirt CDI](https://kubevirt.io/user-guide/operations/containerized_data_importer/) - [VASTCSI 2.3 - Deploying VAST Block CSI Driver](/docs-5-5/docs/csi-driver)