Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

7. VAST Client NFS Multipathing

Prev Next

VAST provides a high-performance NFS interface enhanced with multipathing, enabling clients to communicate with multiple VAST server nodes (CNodes) in parallel. This is achieved through the VAST NFS client, which builds on the standard Linux kernel NFS driver and provides multipathing support for both TCP and RDMA transport modes.

Multipathing enables:

  • Load-balanced traffic across multiple VIPs.

  • Parallel data paths for improved throughput.

  • Support for multiple client-side NICs to utilize all available bandwidth.

Note: The VAST NFS client is optional and best used in high-performance scenarios with large, sequential I/O—like AI training or HPC—where it enables multi-path, RDMA, or GPU Direct for higher throughput. For general-purpose workloads or many small files, it offers only a small benefit.

Mounting with Multipathing

The VAST Enhanced NFS client introduces several mount options to leverage multipathing fully:

NFSv3:

sudo mount -o vers=3,nconnect=34,pconnect=4,mdconnect=2,spread_reads,spread_writes,noextend,lookupcache=pos,remoteports=dns localports=192.168.0.100~192.168.0.200 vast.example.com:/export /mnt/vast

NFSv4:

sudo mount -o vers=4,nconnect=32,pconnect=4,remoteports=dns localports=192.168.0.100~192.168.0.200 vast.example.com:/export /mnt/vast

Option

Description

nconnect=<N>

Sets the total number of TCP connections per mount. These connections are distributed across VIPs.

Note:

  1. Use 8 connections for every 100Gbps with TCP (e.g., nconnect=32 for 400Gbps).

  2. Use 2 connections for every 100Gbps with RDMA (and a minimum of 8 to reach good load balancing).

pconnect=<NUM>

Controls the number of connections made by each client. The nconnect parameter still controls the total number of connections. For example, passing nconnect=32 and pconnect=4 will create 8 clients with 4 connections each, all to the same destination address, and regardless of how big is remoteports. (Note: pconnect applies to vastnfs 4.5 or later)

When using mdconnect=2, there are two connections out of nconnect settings that are dedicated to metadata, hence the values of 34 (32 + 2)

remoteports

Enables connections to multiple VIPs (server-side IPs).

Note: It is recommended to use remoteport=dns, so the client can issue DNS queries with the mount target DNS name and retrieve an array of addresses.

localports

Specifies multiple client-side IPs for outgoing traffic (NFSv3 only).

spread_reads

Distributes read operations across all connections to increase throughput. Only applies to NFSv3

spread_writes

Distributes write operations across all connections. Only applies to NFSv3

lookupcache=pos

Reduces the delay in discovering newly created files on other clients by enabling positive lookup caching, which is useful when one client waits for another to create a file.

Note:

  1. Recommended for distributed applications where one client creating a file should be immediately visible to another.

  2. Not recommended for home directories or source code repositories.

mdconnect=2

Enables dedicated metadata channels, preventing metadata operations from being blocked behind large read/write I/O.

Note: Requires vastnfs 4.0.34 or later. mdconnect Only applies to NFSv3

noextend

Turns off the default extend-to-page optimization in writes. Useful for specific NFSv3 workflows with multiple clients. No tradeoffs; safe default.

nocto (optional):

Disables Close-to-Open (CTO) consistency checks to optimize read/write latency for NFSv3.

Note: Recommended primarily for single-host workloads (e.g., dedicated home directories or temporary local scratch space). Disabling CTO cache validation means the client does not guarantee that data written by one host is immediately visible to another, which can result in stale reads or inconsistent file states across nodes.

NFSv3: nconnect & mdconnect Partition

When nconnect=34 & mdconnect=2, the 34 total connections are split as follows:

  • 2 dedicated connections (from the 34-connection) are used exclusively for metadata operations.

  • The remaining 32 connections are dedicated to regular Data I/O.

NFSv3 connections

NFSv3 Data I/O Multipathing:

  • Data connections are distributed based on pconnect. For example, with pconnect=4 and nconnect - mdconnect = 32, traffic is spread across 8 VIPs (32 Data Connections split across 8 target VIPs, providing up to 4 parallel connections per VIP).

NFSv3 VIP connections

NFSv3 Data Striping (spread_reads / spread_writes):

  • Stripes I/O for individual files across all 8 VIPs/CNodes concurrently, maximizing single-file bandwidth and bypassing the 1-file = 1-VIP limitation.

NFSv3 VIPs to CNodes

NFSv4: Understanding nconnect & pconnect Allocation

  • No spread_reads / spread_writes support in NFSv4.

  • No mdconnect support for NFSv4.

NFSv4 Data I/O Multipathing:

  • With nconnect=8 & pconnect=2, the 8 total connections are split across 4 unique Target VIPs (VIP 1, VIP 2, VIP 3, VIP 4) with 2 parallel connections per VIP.

NFSv4 connections

NFSv4 File-to-VIP Mapping (1-File = 1-VIP Limitation):

  • Each file is pinned to a single VIP, but can leverage both parallel connections to its assigned VIP (pconnect=2) if multiple processes or open instances request I/O to that same file.

NFSv4 file to VIP mapping

VAST NFS readahead considerations

VAST NFS Client 4.5.7 and later introduces an enhanced readahead implementation with a per-thread sequential access detector. The client can identify multiple threads performing sequential reads from different offsets within the same file and independently optimize readahead for each thread.

This closely matches how SGLang loads SafeTensors model weights in parallel using multiple worker threads. As a result, VAST NFS significantly improves page cache prefetch efficiency during model initialization, reducing model load times by approximately 5–10× for SGLang workloads.

For more information on readahead configuration: VAST Quick NFS Read Ahead Tuning

Additional Resources

Related articles