---
title: "Connecting Linux Clients to VAST NVMe/TCP Block Devices"
slug: "connecting-linux-clients-to-vast-nvmetcp-block-devices"
description: "Learn to connect VAST Data Block Volumes to Linux via NVMe/TCP with our step-by-step guide for Ubuntu and RHEL, optimizing performance and multi-pathing."
updated: 2026-10-28T18:40:50Z
published: 2026-10-28T18:40:50Z
canonical: "docs.vastdata.com/connecting-linux-clients-to-vast-nvmetcp-block-devices"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Connecting Linux Clients to VAST NVMe/TCP Block Devices

## Install NVMe CLI

The NVMe CLI tool is used for all NVMe device management, including NVMe/TCP devices. Install it by running the following command.

On Ubuntu :

```bash
sudo apt-get install nvme-cli
```

On RHEL/Rocky :

```bash
sudo yum install nvme-cli -y
```

Confirm nvme-cli is installed by running the `nvme list` command, which will show all NVMe devices currently visible to the host. If your host has an internal NVMe device, then it will be shown here; otherwise, it should return an empty list :

```plaintext
user@ubuntu1:~$ sudo nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
user@ubuntu1:~$
```

## Load NVMe/TCP Kernel Module

Many Linux distributions will automatically load the required kernel modules to use NVMe/TCP, to make sure they are loaded, run the following command :

```bash
sudo modprobe nvme-tcp
```

## Map VAST Volumes to Host

Mapping of VAST Volumes to a host is done using the Host’s NQN, which is automatically generated when the nvme-cli package is installed. This NQN is specific to a host and must be configured in the VAST cluster.

The host’s NQN is contained within the file`/etc/nvme/hostnqn` :

```bash
user@ubuntu1:~$ cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:9e6b1d42-4bad-9fca-939c-4de9d5c73d0e
```

Using this NQN, create a Host within VAST and map one or more Volumes to that host. Details on how to do this can be found in the [VAST Administrator's Guide](/docs-5-5/docs/provisioning-block-storage-with-vms).

## Discovering/Connecting NVMe/TCP Volumes

Once one or more volumes have been mapped to the host in the VAST GUI, they must be connected to the Linux system.

Firstly, we can use the `nvme discover` command to query the VAST cluster and determine the IP addresses available on the cluster to connect to. This command requires two options - one to tell it to use NVMe/TCP for the connection (`--transport=tcp`) and a second to provide a single IP address for a VAST VIP Pool (`--traddr=x.x.x.x`). An additional IP address to use will be automatically discovered by the Linux system.

When run, the `nvme discover` should return details of the NVMe “subsystems” being presented by the VAST cluster, and up to 16 IP addresses (“traddr”) for the cluster. Note that there is no need to record this information - as long as it is displayed, then we know that things are configured correctly.

Run the following command, replacing x.x.x.x with an IP address from the VAST VIP Pool :

```bash
sudo nvme discover --transport=tcp --traddr=x.x.x.x
```

You should see output similar to this, showing the VAST Clusters subsystem NQN (subnqn) and the discovered IP addresses (traddr).

```bash
user@ubuntu1:~$ sudo nvme discover --transport=tcp --traddr=172.31.1.1

Discovery Log Number of Records 6, Generation counter 216
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  1
trsvcid: 4420
subnqn:  nqn.2024-08.com.vastdata:127db70c-0197-5f4f-8af8-44bead61cda2:default:mysubsys
traddr:  172.31.1.6
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  1
trsvcid: 4420
[...etc...]
```

Having confirmed we have proper connectivity to the array, we can add the options passed above to the `/etc/nvme/discover.conf` file. This will remove the need to pass them for subsequent commands, and more importantly, it will also allow them to be used when auto-connecting NVMe devices after an OS reboot.

These 2 options should be added to this file on a new line, with nothing else on the line.

```bash
user@ubuntu1:~$ echo '--transport=tcp --traddr=172.31.1.1' | sudo tee -a /etc/nvme/discovery.conf
--transport=tcp --traddr=172.31.1.1
user@ubuntu1:~$
```

Finally, we can tell the system to connect all of the discoverable NVMe devices to the system, using the `connect-all` command. There is no need to provide the transport and traddr options to this command, as they will be automatically read from the `discovery.conf` file above.

```bash
sudo nvme connect-all
```

Confirm that the device(s) are now visible by running `nvme list`. You should see one entry for each VAST Volume mapped to the host.

```bash
user@ubuntu1:~$ nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme1n1          /dev/ng1n1            VastData             VastData                                 0x3          0.00   B / 250.00  GB    512   B +  0 B   24.05
/dev/nvme1n2          /dev/ng1n2            VastData             VastData                                 0x3          0.00   B / 500.00  GB    512   B +  0 B   24.05
user@ubuntu1:~$
```

The Linux NVMe drivers include built-in support for multi-pathing. The `nvme list-subsys` command allows you to view the paths being used to the array, and confirm they are healthy :

```bash
user@ubuntu1:~$ sudo nvme list-subsys
nvme-subsys1 - NQN=nqn.2024-08.com.vastdata:127db70c-0197-5f4f-8af8-44bead61cda2:default:mysubsys
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:9e6b1d42-4bad-9fca-939c-4de9d5c73d0e
               iopolicy=numa
\
 +- nvme1 tcp traddr=172.31.1.6,trsvcid=4420,src_addr=172.31.2.25 live
 +- nvme2 tcp traddr=172.31.1.2,trsvcid=4420,src_addr=172.31.2.25 live
 +- nvme3 tcp traddr=172.31.1.4,trsvcid=4420,src_addr=172.31.2.25 live
 +- nvme4 tcp traddr=172.31.1.1,trsvcid=4420,src_addr=172.31.2.25 live
 +- nvme5 tcp traddr=172.31.1.3,trsvcid=4420,src_addr=172.31.2.25 live
 +- nvme6 tcp traddr=172.31.1.5,trsvcid=4420,src_addr=172.31.2.25 live
```

In this case, we can see that there are a total of 6 paths, corresponding to the 6 IP addresses in the VIP Pool on the cluster (172.31.1.1 to .6). The “live” indication at the end of each line confirms that the path is active and in use.

The volume(s) can now be accessed using the path listed under the “Node” heading in the NVMe list.

```bash
user@ubuntu1:~$ sudo fdisk /dev/nvme1n1

Welcome to fdisk (util-linux 2.37.4).

Command (m for help): p
Disk /dev/nvme1n1: 232.83 GiB, 250000000000 bytes, 488281250 sectors
Disk model: VastData
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 1048576 bytes
```

## Optimizing Multi-Pathing for Performance

As mentioned above, the Linux NVMe drivers include native Multi-pathing support that defaults to the “NUMA” I/O policy, which is suboptimal for VAST clusters and should be changed to “round-robin” for all VAST devices.

The current iopolicy for a subsystem can normally be displayed in the output of the nvme list-subsys command :

```bash
user@ubuntu1:~$ sudo nvme list-subsys
nvme-subsys1 - NQN=nqn.2024-08.com.vastdata:127db70c-0197-5f4f-8af8-44bead61cda2:default:mysubsys
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:9e6b1d42-4bad-9fca-939c-4de9d5c73d0e
               iopolicy=numa
[...]
```

If the iopolicy is not shown in this output, it can also be found using the below command, replacing “nvme-subsys1” with the relevant identifier for the subsystem from the output above :

```bash
user@ubuntu1:~$ cat /sys/class/nvme-subsystem/nvme-subsys1/iopolicy
numa
```

To configure the default iopolicy used for VAST subsystems, create/edit the file

```bash
/etc/udev/rules.d/71-nvmf-vastdata.rules
```

And place the following lines in it :

```bash
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{subsystype}=="nvm", ATTR{model}=="VASTData", ATTR{iopolicy}="round-robin"
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{subsystype}=="nvm", ATTR{model}=="VastData", ATTR{iopolicy}="round-robin"
```

(For VAST clusters installed on 5.3.2 or later, only the first line is required; there is no harm in including both.

After saving, run the following 2 commands to cause the system to load and apply the new settings :

```bash
sudo udevadm control --reload-rules
sudo udevadm trigger
```

Confirm the correct iopolicy is being used using the same method as previously described.

```bash
user@ubuntu1:~$ cat /sys/class/nvme-subsystem/nvme-subsys1/iopolicy
round-robin
```

## Auto-Connecting Devices on Reboot

NVMe/TCP devices will not be automatically discovered/connected on reboot unless the `nvmf-autoconnect` service is enabled. Depending on the distribution, this may be enabled by default, but confirm that it is by running the following :

```bash
sudo systemctl enable nvmf-autoconnect
```

## Additional Considerations

For VAST, `ctrl_loss_tmo` should be set to `off `(same as `-1`).

This means that the host will never give up on a non-responsive controller (as we cannot bound the duration of a network outage). The host should not disconnect from the array unless explicit administrative action is taken.

This setting exists out-of-the-box in newer `nvme-cli` versions:

See: [linux-nvme/nvme-cli](https://github.com/linux-nvme/nvme-cli/blob/master/nvmf-autoconnect/udev-rules/71-nvmf-vastdata.rules.in)

VAST CSI does effectively the same to achieve the same outcome with older versions of `nvme-cli`.

As for `fast_io_fail_tmo` : It does not specify the timeout to failover, but rather the time to fail the IO. Meaning the filesystem and/or userspace will see EIO errors. By default, this setting is set to off (e.g., `-1`), which means that I/O will be queued for as long as available paths exist (which is similar to the dm-multipath `queue_if_no_path` setting if you are familiar with traditional DM-multipathing).

You should take into account that if you set `fast_io_fail_tmo=30` you are effectively instructing the driver to propagate IO errors to the upper filesystem and/or userspace after 30 seconds. VAST CSI does not touch this setting.

`keep_alive_tmo` - VAST CSI doesn't touch the default (10s), which means that the nvme-host will send keep-alive to the controller every 5s (with a timeout of 5s) in case there is no online traffic.

`reconnect_delay` - VAST CSI doesn't touch the default (10s), which means that the nvme-host will attempt a periodic reconnect every 10s when disconnected from the controller (and will do that for as long as needed because `ctrl_loss_tmo=off` )

## Additional Resources:

- [Block Storage on VAST](/kb/docs/block-storage-on-vast)