---
title: "Configuring an NVMe/TCP Client on Linux for VAST Cluster Block Storage"
slug: "configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg"
description: "Enable NVMe/TCP Linux clients to access VAST block storage with essential steps, including transport rules, retries, and subsystem connections."
tags: ["5.4"]
updated: 2026-10-28T01:54:23Z
published: 2026-10-28T01:54:23Z
canonical: "docs.vastdata.com/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuring an NVMe/TCP Client on Linux for VAST Cluster Block Storage

Complete these steps to enable ab NVMe/TCP Linux client to access VAST block storage:

1. [Install](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#installing-the-client) the NVMe CLI tool.
2. [Create](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#creating-transport-rules) transport rules to enable the client to connect to the VAST Cluster block controller.
3. [Verify](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#verifying-the-maximum-number-of-retries-for-nvme) that the host's maximum number of retries for NVMe allows for maintaining high availability.
4. [Obtain](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#obtaining-the-host-nqn) the host NQN that will identify your host for the VAST cluster.
5. [Connect](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#connecting-to-mapped-volumes) to the NVMe subsystem on the VAST cluster using your host NQN.

This step requires that block volumes have been [created](/docs-5-4/docs/en/creating-volumes-tg) on the VAST cluster and [mapped](/docs-5-4/docs/en/assigning-block-hosts-to-volumes-tg) to your hosts. it includes the substeps:

  1. Load kernel modules to enable NVMe over Fabrics.
  2. Discover available VAST NVMe subsystems over TCP.
  3. Connect to the VAST NVMe subsystem you need.
6. [Verify](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#listing-subsystems-paths-and-available-volumes) the configuration by listing connected NVMe subsystems and block volumes.
7. If necessary, [troubleshoot](/docs-5-4/docs/en/configuring-an-nvmetcp-client-on-linux-for-vast-cluster-block-storage-tg#troubleshooting) your configuration.

## Installing the Client

To configure client block hosts for interacting with the cluster as a remote NVMe device, install the NVMe CLI tool on the host:

```plaintext
sudo yum install nvme-cli
```

## Creating Transport Rules

Create transport rules to ensure that the client automatically discovers and connects or reconnects to the cluster's subsystems and volumes after reboot or new volume mappings:

1. Create the following file:

```plaintext
sudo vi /lib/udev/rules.d/71-nvmf-vastdata.rules
```
2. Add this content to the file:

```plaintext
# Enable round-robin for Vast Data Block Controller
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{subsystype}=="nvm", ATTR{model}=="VASTData", RUN+="/bin/sh -c 'echo round-robin > /sys/class/nvme-subsystem/%k/iopolicy'"
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{subsystype}=="nvm", ATTR{model}=="VastData", RUN+="/bin/sh -c 'echo round-robin > /sys/class/nvme-subsystem/%k/iopolicy'"
```
3. Run:

```plaintext
sudo udevadm control --reload-rules
sudo udevadm trigger
```

## Verifying the Maximum Number of Retries for NVMe

To provide for high availability, the maximum number of retries configured on the host for NVMe commands must be set to a non-zero value. On most systems, the default parameter value is 5.

To check the maximum number of retries for NVMe, run one of the following:

- ```plaintext
cat /sys/module/nvme_core/parameters/max_retries
```
- ```plaintext
grep . /sys/module/nvme_core/parameters/*
```

To persistently set a maximum number of retries for NVMe to 5:

- For hosts where `/sys/module/nvme_core` already exists:

  1. Create or edit this file:

```plaintext
sudo nano /etc/modprobe.d/nvme_core.conf
```
  2. Add this line to the file:

```plaintext
options nvme_core max_retries=5
```
  3. Run this command to have the new settings applied on boot:

    - On Ubuntu or Debian:

```plaintext
sudo update-initramfs -u
```
    - On RHEL, CentOS or Fedora:

```plaintext
sudo dracut -f
```
  4. Reboot the host:

```plaintext
sudo reboot
```
  5. Verify that the maximum number of retries for NVMe is now set to 5:

```plaintext
cat /sys/module/nvme_core/parameters/max_retries
```
- For hosts where `/sys/module/nvme_core` does not exist:

  1. Edit GRUB:

```plaintext
sudo nano /etc/default/grub
```
  2. Add the `nvme_core.max_retries=5` string to the options in `GRUB_CMDLINE_LINUX_DEFAULT`, for example:

```plaintext
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvme_core.max_retries=5"
```
  3. Apply the updates:

    - On Ubuntu or Debian:

```plaintext
sudo update-grub
```
    - On RHEL, CentOS or Fedora:

```plaintext
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
  4. Reboot the host:

```plaintext
sudo reboot
```
  5. Verify that the maximum number of retries for NVMe is now set to 5:

```plaintext
cat /sys/module/nvme_core/parameters/max_retries
```

## Obtaining the Host NQN

The host NQN is generated automatically when you install `nvme-cli`. The host NQN must be [specified](/docs-5-4/docs/en/creating-block-hosts-tg) in host properties maintained on the VAST cluster to allow for mapping VAST block volumes to the host.

To obtain your host NQN:

```plaintext
cat /etc/nvme/hostnqn
```

## Connecting to Mapped Volumes

> [!NOTE]
> **Note**
> 
> This step requires that block volumes have been [created](/docs-5-4/docs/en/creating-volumes-tg) on the VAST cluster and [mapped](/docs-5-4/docs/en/assigning-block-hosts-to-volumes-tg) to your host.

After volumes have been mapped to the host through VMS, you can do the following to connect to the mapped volumes:

1. Load the necessary kernel modules to enable NVMe over Fabrics (NVMe-oF).

  - To load the modules once:

```plaintext
sudo modprobe nvme
sudo modprobe nvme-fabrics
```
  - To have the modules load automatically on reboot:

    1. Create a file `/etc/modules-load.d/nvme.conf` and list the modules to be loaded in it:

```plaintext
nvme
nvme-fabrics
```
    2. Run this command so that the new settings apply on boot:

      - On Ubuntu or Debian:

```plaintext
sudo update-initramfs -u
```
      - On RHEL, CentOS or Fedora:

```plaintext
sudo dracut -f
```
2. Discover available VAST NVMe subsystems over TCP:

```plaintext
sudo nvme discover -t tcp -a VIRTUAL_IP -s 8009
```

For `VIRTUAL_IP`, provide a virtual IP from a virtual IP pool with *Protocol* role that is accessible to the relevant block-enabled view. The view policy might restrict/dedicate virtual IP pools.

> [!NOTE]
> **Note**
> 
> For information about creating subsystems on the cluster, see [Provisioning Block Storage with VMS](/docs-5-4/docs/provisioning-block-storage-with-vms-tg) and [Creating a Block Storage Subsystem (View)](/docs-5-4/docs/en/creating-a-block-storage-subsystem-view-tg).
3. Add the discovery parameters you used to `/etc/nvme/discovery.conf` so that the configuration sustains a reboot:

```plaintext
echo "-t tcp -a VIRTUAL_IP -s 8009" >> /etc/nvme/discovery.conf
```

Replace `VIRTUAL_IP` with the actual virtual IP.
4. Connect to a VAST NVMe subsystem:

  1. Obtain the subsystem NQN from VMS:

    1. In the VAST Web UI, open the Views tab in the Element Store page.
    2. Find the subsystem view.
    3. Right-click the view and select View to see its configuration.

The NQN is displayed in the Subsystem NQN field.
    4. Click the ![files](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/img-55f37bfe5c3574b72a9cae884a5b36f9(2).png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A58%3A31Z&se=2026-11-08T10%3A58%3A31Z&sr=c&sp=r&sig=hp3LowBO5rA40UY6ToWx27JV3puLI%2Fx3Li5E1uUeA2U%3D) button to copy the NQN to your clipboard.
  2. Establish connection to the subsystem:

```plaintext
sudo nvme connect -t tcp -n SUBSYSTEM_NQN -a VIRTUAL_IP
```

in which:

    - `SUBSYSTEM_NQN` is the subsystem NQN obtained in the previous step.
    - `VIRTUAL_IP` is a virtual IP configured on the cluster and accessible to the subsystem view.
  3. Run the `connect-all` command to connect all paths:

```plaintext
sudo nvme connect-all -t tcp -a VIRTUAL_IP -s 8009
```
  4. If volumes are added to the subsystem or removed from the subsystem, you can run `connect-all` again to update the volume mapping.

```plaintext
sudo nvme connect-all
```
  5. Run the following command to ensure that your NVMe connection sustains a reboot:

```plaintext
systemctl enable nvmf-autoconnect.service
```

## Listing Subsystems, Paths and Available Volumes

- To display a list of connected NVMe subsystems and paths, use the `nvme list-subsys` command.

For example:

```plaintext
sudo nvme list-subsys
nvme-subsys0 - NQN=nqn.2024-08.com.vastdata:ef992044-0c8e-557a-a629-4d3c9abd9f9d:default:subsystem-3
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:27590942-6282-d720-eae9-fdd2d81355d4
               iopolicy=round-robin
\
 +- nvme9 tcp traddr=172.27.133.9,trsvcid=4420 live
 +- nvme8 tcp traddr=172.27.133.8,trsvcid=4420 live
 +- nvme7 tcp traddr=172.27.133.7,trsvcid=4420 live
 +- nvme6 tcp traddr=172.27.133.6,trsvcid=4420 live
 +- nvme5 tcp traddr=172.27.133.5,trsvcid=4420 live
 +- nvme4 tcp traddr=172.27.133.4,trsvcid=4420 live
 +- nvme3 tcp traddr=172.27.133.3,trsvcid=4420 live
 +- nvme2 tcp traddr=172.27.133.2,trsvcid=4420 live
 +- nvme16 tcp traddr=172.27.133.16,trsvcid=4420 live
 +- nvme15 tcp traddr=172.27.133.15,trsvcid=4420 live
 +- nvme14 tcp traddr=172.27.133.14,trsvcid=4420 live
 +- nvme13 tcp traddr=172.27.133.13,trsvcid=4420 live
 +- nvme12 tcp traddr=172.27.133.12,trsvcid=4420 live
 +- nvme11 tcp traddr=172.27.133.11,trsvcid=4420 live
 +- nvme10 tcp traddr=172.27.133.10,trsvcid=4420 live
 +- nvme0 tcp traddr=172.27.133.1,trsvcid=4420 live
```
- To display a list of connected NVMe volumes, use the `nvme list` command:

```plaintext
sudo nvme list
```

## Disconnecting Existing Connections

The following commands disconnect the cluster's subsystems from the host:

- To disconnect all connected subsystems:

```plaintext
sudo nvme disconnect-all
```
- To disconnect a specific subsystem:

```plaintext
sudo nvme disconnect -n <NQN>
```

## Troubleshooting

### Issue: NVMe Subsystem Not Found

- Cause: Incorrect IP address or network issue.
- Solution: Verify that the virtual IP is correct and that the host has network connectivity to the VAST cluster.

### Issue: NVMe Device Not Appearing

- Cause: NVMe connection not established or missing kernel modules.
- Solution: Ensure kernel modules are loaded using:

```plaintext
sudo modprobe nvme
sudo modprobe nvme-fabrics
```

### Logs and Diagnostics

Use `dmesg` to check kernel logs for errors related to NVMe:

```plaintext
dmesg | grep nvme
```

### Issue: No Mapped Volumes

- Symptoms:

  - `sudo nvme discover` output:

```plaintext
Failed to write to /dev/nvme-fabrics: Connection refused
Failed to add controller, error connection refused
```
  - `dmesg` error: `nvme nvme0: failed to connect socket: -111`
- Root cause: There are no mapped volumes.
- Fix: Validate the volume mapping to this client's NQN.

### Issues Connecting to the Target

| Symptom | Root Cause |
| --- | --- |
| `sudo nvme discover` output: ```plaintext Failed to add controller, error cannot assign requested address ``` | The source IP in the command line is incorrect. |
| `sudo nvme connect` output: ```plaintext Failed to write to /dev/nvme-fabrics: Input/output error could not add new controller: failed to write to nvme-fabrics device ``` | The IP used for connection is incorrect or does not belong to the relevant tenant. |
| `sudo nvme discover` output: ```plaintext failed to get discovery log: Success ``` | The IP used for connection is incorrect or does not belong to the relevant tenant. |