--- title: "Monitor and Troubleshoot VAST CSI Driver" slug: "monitor-and-troubleshoot-vast-csi-driver-2-7" updated: 2026-11-03T22:38:39Z published: 2026-11-03T22:38:39Z canonical: "docs.vastdata.com/monitor-and-troubleshoot-vast-csi-driver-2-7" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Monitor and Troubleshoot VAST CSI Driver ## Export VAST CSI Driver Metrics to Prometheus VAST CSI Driver can be configured to expose CSI node and controller metrics in Prometheus format. The [node metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#csi-node-metrics) include total counts and average durations for CSI RPCs and mounts/umounts, and also NFS transport (xprt) statistics. The [controller metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#csi-controller-metrics) include total counts and average durations for CSI RPCs. - [Enable Export of CSI metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#enable-export-of-csi-metrics) - [Exposed CSI Metrics Endpoints and Ports](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#exposed-csi-metrics-endpoints-and-ports) - [Exported CSI Metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#exported-csi-metrics) - [Access Exported CSI Metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#access-exported-csi-metrics) - [Sample Metrics Values for Common Scenarios](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#sample-metrics-values-for-common-scenarios) ### Enable Export of CSI Metrics By default, the driver does not expose any metrics. To enable export of metrics: 1. Add the following to the driver's Helm chart configuration file: ``` node: metrics: enabled: true port: 9090 controller: metrics: enabled: true port: 9091 ``` 2. [Install](/develop/docs/deploy-vast-csi-driver-2-7#install-vast-csi-driver-helm-chart) or [upgrade](/develop/docs/upgrade-vast-csi-driver-2-7#upgrade-helmbased-deployment-of-vast-csi-driver) the driver's Helm chart. ### Exposed CSI Metrics Endpoints and Ports When metrics export is enabled: - A headless service is created that serves metrics requests at two endpoints: - `GET /metrics` for getting the metrics in Prometheus format (counters, histograms, gauges), - `GET /health` for health checks. - The node's `DaemonSet` pods expose the node metrics port 9090. - The controller's `Deployment/StatefulSet` pods expose the controller metrics port 9091. > **Note:** You can override default ports by specifying a different value in the `port` entry under node or controller metrics in the driver's Helm chart configuration file. ### Exported CSI Metrics > **Tip:** For a complete reference on CSI metrics, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_REFERENCE.md). - [CSI Node Metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#csi-node-metrics) - [CSI Controller Metrics](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#csi-controller-metrics) - [Metrics Details](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#metrics-details) #### CSI Node Metrics - **Mounts/umounts** | | | | --- | --- | | `csi_node_mount_operations_total` | Total number of mounts (of a PVC to a pod) | | `csi_node_mount_duration_seconds` | Duration of mounts (in seconds) | | `csi_node_umount_operations_total` | Total number of umounts | | `csi_node_umount_duration_seconds` | Duration of umounts (in seconds) | - **NFS transport (xprt) status per cluster** | | | | --- | --- | | `csi_node_nfs_xprt_total` | Total number of active NFS transports | | `csi_node_nfs_xprt_connected` | Number of connected NFS transports | | `csi_node_nfs_xprt_pending_requests_total` | Total number of pending requests across all mounts | | `csi_node_nfs_xprt_backlog_total` | Total number of backlog requests across all mounts | | `csi_node_nfs_xprt_unhealthy` | Number of unhealthy NFS transports | - **NFS transport (xprt) status per virtual IP** These metrics can only be exported while the virtual IP is connected. | | | | --- | --- | | `csi_node_nfs_xprt_connected_state` | `1.0` indicates a healthy connection. `0.0` means disconnected. | | `csi_node_nfs_xprt_congested_state` | `1.0` means flow control is active. | | `csi_node_nfs_xprt_locked_state` | `1.0` indicates that the connection is locked. | | `csi_node_nfs_xprt_pending_requests` | Number of RPC calls waiting for a response | | `csi_node_nfs_xprt_backlog_depth` | Backlog queue depth for this virtual IP | | `csi_node_nfs_xprt_mounts` | Number of active NFS mounts for this virtual IP | #### CSI Controller Metrics | | | | --- | --- | | `csi_plugin_operations_total` | Total number of all CSI gRPC method calls (CreateVolume, DeleteVolume, ControllerPublishVolume, and so on) | | `csi_plugin_operations_seconds` | Average duration of a CSI gRPC method call | #### Metrics Details In addition to the measured value per metric type, a metric may include labels that provide additional information about the measured operation. For example, the following metric: ``` csi_node_mount_operations_total{operation_type="nfs",status="success",node_name="worker-node-1",pvc_namespace="prod"} 15 ``` specifies that the measured value was taken for the mounts that: - were made through the NFS access protocol, - completed successfully, - occurred on a worker node named `worker-node-1`, - targeted the `prod` namespace. ### Access Exported CSI Metrics > **Tip:** For more detailed guidance, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_GUIDE.md). Run the following commands to verify that the metrics endpoints work as expected: - For node metrics: ``` kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-node kubectl port-forward -n vast-csi pod/ 9090:9090 curl -s http://localhost:9090/metrics curl -s http://localhost:9090/health ``` - For controller metrics: ``` kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-controller kubectl port-forward -n vast-csi pod/ 9091:9091 curl -s http://localhost:9091/metrics curl -s http://localhost:9091/health ``` ### Sample Metrics Values for Common Scenarios The following illustrates typical metrics values in common scenarios. > **Tip:** For more examples, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_EXAMPLES.md). - Upon creating a pod: - Aggregate metrics are set to 0. - Counters are not reported. - `csi_node_nfs_xprt_connected_state`, `csi_node_nfs_xprt_congested_state` and `csi_node_nfs_xprt_pending_requests` are not reported. - After mounting one PVC to virtual IP 192.168.1.10: ``` csi_node_mount_operations_total{node_name="worker-1",operation_type="nfs",pvc_namespace="default",status="success"} 1.0 csi_node_mount_duration_seconds_sum{...} 0.823 csi_node_mount_duration_seconds_count{...} 1.0 csi_node_nfs_xprt_total 1.0 csi_node_nfs_xprt_connected 1.0 csi_node_nfs_xprt_connected_state{destination="192.168.1.10"} 1.0 ``` - If the VAST cluster's virtual IP becomes unreachable: - `csi_node_mount_duration_seconds` is at approx. 30 seconds (or your configured timeout) - `csi_node_nfs_xprt_total` is `0.0`. - In case of network congestion/high latency: - `csi_node_nfs_xprt_congested_state` is `1.0`. - `csi_node_nfs_xprt_pending_requests` exceeds 100. - `csi_node_nfs_xprt_unhealthy` is set to a non-zero value. - On pod deletion: - `csi_node_mount_operations_total` stays at its last value. - `csi_node_umount_operations_total` is reported with a non-zero value. - After approx. 30 seconds, `csi_node_nfs_xprt_total` drops to `0.0`. ## Performance Tuning for VAST CSI Driver To fine-tune VAST CSI Driver performance on large-scale Kubernetes clusters: - [Adjust](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#adjust-volume-attachment) volume attachment - [Disable](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#disable-sending-of-usage-stats) sending of usage statistics - [Adjust](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#adjust-qps-and-burst-limits) QPS and burst limits - [Use](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#use-api-tokens-for-vms-authentication) API tokens for VMS authentication - [Increase](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#increase-number-of-workers) the number of workers - [Use](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#use-ephemeral-volumes) ephemeral volumes ### Adjust Volume Attachment Use the `attachRequired` parameter in the driver's Helm chart configuration file to fine-tune how the driver attaches volumes to workload nodes prior to mounting the volume into a pod. - `attachRequired: true` (default): The driver performs a separate ControllerPublishVolume RPC to attach the volume to the node. - `attachRequired: false`: The driver attaches the volume to the node as part of the NodePublishVolume operation, which can significantly speed up volume attachment. NOTE: This setup cannot be used with nodes where HTTP/HTTPS ports are not open. ### Disable Sending of Usage Stats Use the `disableUsageStats` parameter in the driver's Helm chart configuration file to prevent the driver from sending usage metrics to the VMS. - `disableUsageStats: false` (default): The driver sends usage metrics to the VMS. - `disableUsageStats: true`: The driver does not send usage metrics to the VMS. ### Adjust QPS and Burst Limits Adjust the `kube-api-qps` and `kube-api-burst` parameters of the CSI provisioner in the driver's Helm chart configuration file: - `kube-api-qps` sets the maximum allowed number of requests per second that the CSI provisioner can send to the Kubernetes API server. The default value is 5. - `kube-api-burst` sets the maximum allowed number of requests that can be sent at any given time after the `kube-api-qps` limit is exceeded. The default value is 10. For example: ``` controller: extraArgs: csiProvisioner: - kube-api-qps=500 - kube-api-burst=1000 ``` ### Use API Tokens for VMS Authentication Using [Kubernetes secrets](/develop/docs/deploy-vast-csi-driver-2-7#create-kubernetes-secret-for-vast-csi-driver) with API tokens for authentication on the VAST cluster may improve performance compared to using secrets with VMS credentials. ### Increase Number of Workers Adjust the `numWorkers` parameter in the driver's Helm chart configuration file to increase the number of CSI workers. ### Use Ephemeral Volumes With ephemeral volumes, the driver's interaction with the VMS is not bound to the CSI controller but is distributed among the CSI nodes. Using ephemeral volumes allows for similar performance improvements as running with `attachRequired: false`. ## Steps to Troubleshoot VAST CSI Driver - [Run](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#run-basic-troubleshooting-commands) basic troubleshooting commands - [Check](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#check-kubernetes-logs) Kubernetes logs - [Check](/develop/docs/monitor-and-troubleshoot-vast-csi-driver-2-7#check-vms-logs) VMS logs ### Run Basic Troubleshooting Commands Run the following commands to troubleshoot VAST CSI Driver: - Display all pods: ``` kubectl get pods --all-namespaces -o wide ``` - Check logs from the controller. Typically, there are four containers: `csi-provisioner`, `csi-attacher`, `csi-resizer`, and `csi-vast-plugin`. Check all of them. Run the following command to display the most important information: ``` kubectl logs csi-vast-controller-0 --namespace vast-csi -c csi-vast-plugin | less ``` - Check logs from the node. There are probably two containers: `csi-node-driver-registrar` and `csi-vast-plugin` . Check both. Run the following command to display the most important information: ``` kubectl logs csi-vast-node- --namespace vast-csi -c csi-vast-plugin ``` - Check to see if there are any system-level events, such as errors. Run the following command to list events sorted by timestamp: ``` kubectl get events --sort-by=.metadata.creationTimestamp ``` - Display persistent volume claims: ``` kubectl get pvc ``` The output is similar to the following: ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE shared-claim Bound pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi RWO managed-nfs-storage 12d ``` - Display persistent volumes: ``` kubectl get pv ``` The output is similar to the following: ``` NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi RWO Delete Bound default/shared-claim managed-nfs-storage 12d ``` - Display the defined storage classes: ``` kubectl get sc ``` The output is similar to the following: ``` NAME PROVISIONER AGE managed-nfs-storage nfs-client 13d ``` > **Note:** To get more detailed information, use the `describe` option on each command. ### Check Kubernetes Logs If none of the above helps, check the underlying Kubernetes logs. The location of Kubernetes logs depends on your operating system. On CentOS, the typical location is `/var/log/messages`. These logs often contain more detailed information about the container startup process and provide an indication of errors. ### Check VMS Logs VAST CSI Driver contacts VAST Management Service (VMS) to create directories and quotas on behalf of the Kubernetes cluster. So checking VMS logs can help troubleshoot your environment. VMS logs can be found in `/vast/vman/vms/log` on the VMS node. Useful information is often found in vms.log and vapi.log. For example, the vms.log can provide the following information (the mount for VAST CSI Driver is /keystest/k8s): ``` grep k8s vms.log [2022-01-05 17:18:47,313] INFO [services.py:382/430] AUDIT - VMS - Manager: admin (10.61.203.63) POST /api/quotas/ with body {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576} [2022-01-05 17:18:47,562] INFO [services.py:837/430] Creating Quota with {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576} [2022-01-05 17:18:47,573] INFO [services.py:843/430] Creating directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c [2022-01-05 17:18:47,578] INFO [services.py:852/430] Created directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c ``` ## Resolve PVC Timeouts ### PVC Timeout due to Missing NFS Client on Kubernetes Worker Node If a PVC does not mount with a timeout, a likely outcome is the Kubernetes worker node is missing the NFS client. Run the `kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin` command to check the node logs. The output can be similar to the following: ``` Warning FailedMount 50s kubelet, k8s-a-node03 Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[mariadb-credentials default-token-zpkp5 data config]: timed out waiting for the condition Warning FailedMount 37s kubelet, k8s-a-node03 MountVolume.SetUp failed for volume "pvc-4ebe1936-3613-4c37-957c-4d21e623aa98" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded ``` To resolve the problem, install the NFS client and test mounts locally on the client, then retry via Kubernetes. To install the NFS client: ``` sudo yum -y install nfs-utils ``` ### PVC Timeout due to No Communication with VMS Kubernetes nodes can be on the VAST data network and not on the VAST management network. In this case, the return path from VMS to the Kubernetes node will fail. Ensure that you have valid routes on the correct network. On the Kubernetes node, run the following command and verify that the correct interface is used: ``` ip route get ``` Run this command on the VMS node: ``` ip route get ```