--- title: "Monitor and Troubleshoot VAST Block CSI Driver" slug: "monitor-and-troubleshoot-vast-block-csi-driver-27" updated: 2026-10-28T02:14:43Z published: 2026-10-28T02:14:43Z canonical: "docs.vastdata.com/monitor-and-troubleshoot-vast-block-csi-driver-27" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Monitor and Troubleshoot VAST Block CSI Driver ## Export VAST Block CSI Driver Metrics to Prometheus VAST Block CSI Driver can be configured to expose CSI node and controller metrics in Prometheus format. The [node metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#csi-node-metrics) include total counts and average durations for CSI RPCs and NVMe connections. The [controller metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#csi-controller-metrics) include total counts and average durations for CSI RPCs. - [Enable Export of CSI metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#enable-export-of-csi-metrics) - [Exposed CSI Metrics Endpoints and Ports](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#exposed-csi-metrics-endpoints-and-ports) - [Exported CSI Metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#exported-csi-metrics) - [Access Exported CSI Metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#access-exported-csi-metrics) - [Sample Metrics Values for Common Scenarios](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#sample-metrics-values-for-common-scenarios) ## Enable Export of CSI Metrics By default, the driver does not expose any metrics. To enable export of metrics: 1. Add the following to the driver's Helm chart configuration file: ``` node: metrics: enabled: true port: 9092 controller: metrics: enabled: true port: 9093 ``` 2. Install or upgrade the driver's Helm chart. ## Exposed CSI Metrics Endpoints and Ports When metrics export is enabled: - A headless service is created that serves metrics requests at two endpoints: - `GET /metrics` for getting the metrics in Prometheus format (counters, histograms, gauges), - `GET /health` for health checks. - The node's `DaemonSet` pods expose the node metrics port 9092. - The controller's `Deployment/StatefulSet` pods expose the controller metrics port 9093. > **Note:** You can override default ports by specifying a different value in the `port` entry under node or controller metrics in the driver's Helm chart configuration file. ## Exported CSI Metrics > **Tip:** For a complete reference on CSI metrics, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_REFERENCE.md). - [CSI Node Metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#csi-node-metrics) - [CSI Controller Metrics](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#csi-controller-metrics) - [Metrics Details](/develop/docs/monitor-and-troubleshoot-vast-block-csi-driver-27#metrics-details) ### CSI Node Metrics - **Mounts/umounts** | | | | --- | --- | | `csi_node_mount_operations_total` | Total number of mounts (of a PVC to a pod) | | `csi_node_mount_duration_seconds` | Duration of mounts (in seconds) | | `csi_node_umount_operations_total` | Total number of umounts | | `csi_node_umount_duration_seconds` | Duration of umounts (in seconds) | - **NVMe connects per cluster** | | | | --- | --- | | `csi_node_nvme_connect_operations_total` | Total number of active NVMe connections | | `csi_node_nvme_connect_duration_seconds` | Duration of NVMe connections (in seconds) | - **NVMe controller metrics** | | | | --- | --- | | `csi_node_nvme_controller_info` | Lists VAST NVMe controller details as such the NVMe controller name, subsystem, NQN, host NQN, transport type (TCP or RDMA), IP address, controller state. | ### CSI Controller Metrics | | | | --- | --- | | `csi_plugin_operations_total` | Total number of all CSI gRPC method calls (CreateVolume, DeleteVolume, ControllerPublishVolume, and so on) | | `csi_plugin_operations_seconds` | Average duration of a CSI gRPC method call | ### Metrics Details In addition to the measured value per metric type, a metric may include labels that provide additional information about the measured operation. For example, the following metric: ``` csi_node_mount_operations_total{operation_type="block-mount",status="success",node_name="worker-node-1",pvc_namespace="prod"} 15 ``` specifies that the measured value was taken for the mounts that: - were made through the block access protocol, - completed successfully, - occurred on a worker node named `worker-node-1`, - targeted the `prod` namespace. ## Accessing Exported CSI Metrics > **Tip:** For more detailed guidance, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_GUIDE.md). Run the following commands to verify that the metrics endpoints work as expected: - For node metrics: ``` kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-node kubectl port-forward -n vast-csi pod/ 9092:9092 curl -s http://localhost:9092/metrics curl -s http://localhost:9092/health ``` - For controller metrics: ``` kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-controller kubectl port-forward -n vast-csi pod/ 9093:9093 curl -s http://localhost:9093/metrics curl -s http://localhost:9093/health ``` ## Sample Metrics Values for Common Scenarios The following illustrates typical metrics values in common scenarios. > **Note:** For more examples, see [here](https://github.com/vast-data/vast-csi/blob/v2.7/docs/METRICS_EXAMPLES.md). - Upon NVMe connect to the VAST cluster: - `csi_node_nvme_connect_operations_total` is set to `1.0`. - `csi_node_nvme_connect_duration_seconds` shows the connect duration. - After mounting a block device: - `csi_node_nvme_connect_operations_total` is set to `1.0`. - `csi_node_mount_duration_seconds` with `operation_type="block_mount"` shows the mount duration.