---
title: "Cumulus RoCEv2 Configuration Guide"
slug: "cumulus-rocev2-configuration-guide"
description: "Explore Lossless RoCEv2 RDMA over Converged Ethernet for high-throughput, low-latency data transfers, enhancing network efficiency and performance."
updated: 2026-10-27T19:37:32Z
published: 2026-10-28T18:36:57Z
canonical: "docs.vastdata.com/cumulus-rocev2-configuration-guide"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cumulus RoCEv2 Configuration Guide

## Introduction to Lossless RoCEv2

RDMA over Converged Ethernet (RoCE) allows high-throughput, low-latency data transfers between compute nodes at the memory-to-memory level, bypassing the CPU and OS network stack. While the original RoCE (v1) was restricted to a single Layer 2 broadcast domain, **RoCEv2** enables routing across Layer 3 environments.

RoCEv2 achieves routability by encapsulating the InfiniBand (IB) transport protocol within standard Ethernet, IP, and UDP headers. A specific UDP destination port (**4791**) is used to identify RoCEv2 traffic, allowing standard IP routers to handle the packets.

**Comparison of RoCE and RoCEv2 Frame Formats**

| Feature | RoCE (v1) | RoCEv2 |
| --- | --- | --- |
| **Layer** | Layer 2 (Ethernet) | Layer 3 (IP/UDP) |
| **Header Additions** | Ethernet Header + IB Payloads | Ethernet + IP + UDP (Port 4791) |
| **Routability** | Non-routable (Local Subnet) | Routable (via IP L3 Routers) |
| **Congestion Control** | Layer 2 Pause | ECN and PFC |

---

## The Three Pillars of RoCEv2 Flow Control

RoCEv2 employs a "Data Center Quantized Congestion Notification" (**DCQCN**) triad to maintain a lossless environment:

- **DSCP (Traffic Classification):** Marks IP headers with a QoS label to ensure packets are mapped to the correct traffic class across all switches.
- **ECN (Congestion Signaling):** A proactive mechanism that signals endpoints to "slow down" transmission before buffers overflow.
- **PFC (Buffer Protection):** A reactive "safety net" that pauses specific traffic classes on a hop-by-hop basis to prevent buffer exhaustion.

**Congestion Management: AFD vs. WRED**

- **Weighted Random Early Detection (WRED):** Marks traffic with ECN bits based on general buffer occupancy. It treats all traffic in a queue equally.
- **Approximate Fair Drop (AFD):** A more granular mechanism that distinguishes between "**Elephant**" (high-bandwidth) and "**Mice**" (low-bandwidth) flows. AFD is used to trigger ECN marking specifically on elephant flows, allowing short-lived "mice" communications to complete without being penalized by large data transfers.

---

## Detailed Mechanism: Priority Flow Control (PFC)

PFC enables lossless Ethernet by managing congestion on a per-priority basis. Unlike standard Ethernet PAUSE frames, which halt all traffic on a link, PFC pauses only the specific Class of Service (CoS) assigned to RDMA (typically Priority 3), allowing management or heartbeat traffic to continue.

**Threshold Logic: xOFF and xON**

- **xOFF:** When a switch buffer reaches this high-water mark, it sends a PFC pause frame upstream.
- **Buffer Headroom:** This is the dedicated portion of the buffer that absorbs packets already "in flight" between when the pause is signaled and when the upstream sender halts. It is essentially a buffer for packet flight time over the wire.
- **xON:** Once the buffer drains below this lower threshold, the switch signals the upstream device to resume transmission.

**PFC Storms & Watchdog:** A "PFC Storm" occurs when a misbehaving host continuously transmits PFC frames, potentially deadlocking the fabric. The **PFC Watchdog Interval** monitors queues; if a queue is not drained within a specified timeout, the watchdog detects the deadlock and drops the undrained packets to restore network flow.

---

## Detailed Mechanism: Explicit Congestion Notification (ECN)

ECN allows switches to signal congestion to end hosts without dropping packets by using the two least significant bits of the IP Header Type of Service (TOS) field.

**ECN Bit Values**

| Bit Value | Meaning |
| --- | --- |
| **00** | Non-ECN Capable |
| **10** | ECN Capable Transport (0) |
| **01** | ECN Capable Transport (1) |
| **11** | Congestion Encountered (CE) |

**The ECN Interaction Loop**

1. **Marking (Switch):** When buffer occupancy exceeds the WRED/AFD minimum threshold, the switch marks packets with **0x11 (CE)**.
2. **Notification (Receiver NIC):** The Receiver detects the 0x11 mark and generates a **Congestion Notification Packet (CNP)**.
3. **Throttling (Sender NIC):** The Sender receives the CNP and reduces its transmission rate for that flow.

**Contrast:** ECN is a **proactive, end-to-end** mechanism, whereas PFC is a **reactive, hop-by-hop** link-level mechanism.

---

## Step-by-Step RoCEv2 Packet Journey (DCQCN)

Lifecycle of an RDMA write operation across the fabric:

1. **Classification (Sender NIC):** The application initiates a write; the NIC sets the IP packet's **DSCP value** (e.g., 26).
2. **Priority Mapping (Switch):** The switch receives the packet and maps the DSCP to a **Lossless Queue** (e.g., Priority 3).
3. **Buffer Monitoring (Switch Queue):** The switch monitors buffer occupancy against WRED/AFD thresholds.
4. **ECN Marking (Switch):** If the AFD threshold for an elephant flow is reached, the switch sets the packet ECN bits to **0x11**.
5. **PFC Trigger (Switch/Sender):** If congestion hits the **xOFF** threshold, the switch sends a PFC frame upstream to pause Priority 3 traffic.
6. **Congestion Detection (Receiver NIC):** The receiver detects the ECN 0x11 mark in the arrival packet.
7. **CNP Generation (Receiver NIC):** The receiver generates a **CNP** and sends it back to the sender.
8. **Rate Reduction (Sender NIC):** The sender receives the CNP and throttles its transmission rate, allowing fabric buffers to drain.

---

### NVIDIA Cumulus Linux NVUE Configuration

- [Cumulus Linux 5.11 RoCE Documentation](https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-511/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/)
- [Cumulus Linux 5.16 RoCE Documentation](https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-516/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/)

Configuration on NVIDIA Spectrum ASICs is managed via the NVUE CLI.

### Enable Lossless RoCE (Default Mode)

**Command:**

```bash
nv set qos roce
nv config apply -y
```

NVUE defaults to `roce mode lossless`. The commands `nv set qos roce` and `nv set qos roce mode lossless` are equivalent.

**Verify:**

```bash
nv show qos roce
```

**Expected Result (key fields):**

```plaintext
mode          : lossless
enabled       : on
traffic-class : 3
pfc           : enabled
ecn           : enabled
```

### Lossless Single Ingress Pool

**Command:**

```bash
nv set qos roce mode lossless-single-ipool
nv config apply -y
```

Combines lossy/lossless pools for better absorption during burst traffic.

### v5.16: Lossy Multi-TC with Packet Trimming

**Command:**

```bash
nv set qos roce mode lossy-multi-tc
nv set qos remark dscp-on-trim port-level
nv config apply -y
```

**Specific Remarking Values (Standardized for AI POD):**

- **DSCP 21:** Applied to host-downlinks (`lossy-multi-tc-host-group`).
- **DSCP 11:** Applied to network uplinks (`lossy-multi-tc-network-group`).

**Verify DSCP Remarking:**

```bash
nv show qos remark
```

**Expected Result:**

```plaintext
lossy-multi-tc-host-group    : dscp 21
lossy-multi-tc-network-group : dscp 11
dscp-on-trim                 : port-level
```

**Technical Note: Spectrum-4 ASIC Reporting** Engineers may notice a reporting discrepancy on Spectrum-4 switches. `nv show qos roce` may report an ECN threshold of **146.48 KB**, while interface-level commands report **156 KB** (159,744 bytes). The latter is the actual hardware-programmed value; this is expected behavior and does not indicate a configuration error.

### Switch Verification Checklist

| Check | Command | Expected Result |
| --- | --- | --- |
| Global RoCE config | `nv show qos roce` | `mode: lossless`, `enabled: on`, `traffic-class: 3` |
| DSCP remarking | `nv show qos remark` | DSCP 21 (hosts) / DSCP 11 (network) |
| Buffer allocation | `nv show interface qos-roce-status-pool-map` | Lossless pool mapped to Priority 3 |
| PFC/ECN counters | `nv show interface swp1 qos roce counters` | Counters increment only under congestion |

**Example - Interface Counters:**

```bash
nv show interface swp1 qos roce counters
```

**Expected Result (healthy fabric):**

```plaintext
pfc_rx_prio3    : 0
pfc_tx_prio3    : 0
ecn_marked      : 0
cnp_rx          : 0
cnp_tx          : 0
```

---

## Operational Summary Table

| Feature | Component Responsibility | Key Parameter / Value |
| --- | --- | --- |
| **MTU** | Switch & NIC | **9216** (Jumbo Frames) |
| **Traffic Class (RoCE)** | Switch | Priority 3 (Lossless) |
| **PFC** | Switch & NIC | Enabled on CoS 3 |
| **ECN Marking** | Switch | AFD Thresholds (marked 0x11) |
| **CNP Generation** | Receiver NIC | UDP Port 4791 |
| **v5.16 Remarking** | Switch | DSCP 21 (Hosts) / DSCP 11 (Network) |
| **PFC Watchdog** | Switch | Enabled for Deadlock Prevention |
| **RoCE Mode** | Switch (Cumulus) | `nv set qos roce mode lossless` |

---

## NFS over RDMA (NFSoRDMA) on VAST

NFS over RDMA (also called **RPC-over-RDMA**) layers the NFS protocol on top of the RDMA transport instead of TCP. On VAST, this is the primary storage access method for AI/HPC workloads that require low latency and high throughput over a lossless RoCEv2 fabric.

**How it works:**

1. Client sends an initial MOUNT request over TCP to discover the NFS service.
2. Client then establishes an RDMA connection (RPC-over-RDMA) for data path I/O.
3. Data transfers bypass the CPU/kernel network stack via zero-copy RDMA operations.

> **Prerequisite:** The lossless RoCEv2 fabric (PFC + ECN on Priority 3) described in the sections above **must** be configured end-to-end across client NICs, switches, and VAST CNodes before NFSoRDMA will perform reliably.

### NFSoRDMA vs. NFS over TCP

| Feature | NFS over RDMA | NFS over TCP |
| --- | --- | --- |
| Transport | RDMA (RoCEv2 / InfiniBand) | TCP/IP |
| CPU overhead | Low (zero-copy) | Moderate to high |
| Latency | Very low | Higher |
| Throughput | Higher (parallel connections) | Lower |
| Network requirement | Lossless fabric (PFC/ECN) | Standard Ethernet |
| Best use case | AI/HPC, parallel I/O | General-purpose storage |

### Prerequisites

| Requirement | Verification Command | Expected Result |
| --- | --- | --- |
| RDMA-capable NIC | `lspci \\| grep Mellanox` | ConnectX-6/7 detected |
| `rpcrdma` kernel module | `modprobe rpcrdma && lsmod \\| grep rpcrdma` | Module loaded |
| RDMA link active | `rdma link` | State: ACTIVE, Rate: 200/400 |
| Lossless network | `nv show qos roce` (switch) | `mode: lossless` |
| VAST NFS client | `rpm -qa \\| grep vastnfs` or `dpkg -l \\| grep vastnfs` | VAST NFS package installed |
| NIC trust mode | `mlnx_qos -i <iface> \\| grep trust` | `Priority trust state: dscp` |

### Load rpcrdma Module

**Command:**

```bash
modprobe rpcrdma
lsmod | grep rpcrdma
```

**Expected Result:**

```plaintext
rpcrdma               131072  0
rdma_cm                36864  3 rpcrdma,ib_iser,rdma_ucm
```

### VAST NFS RDMA Mount (Example)

**Command:**

```bash
sudo mkdir -p /mnt/vast
sudo mount -t nfs -o \
  rw,noatime,nodiratime,vers=3,\
  rsize=1048576,wsize=1048576,namlen=255,\
  acregmin=60,acdirmin=60,soft,noacl,\
  forcerdirplus,proto=rdma,port=20049,nconnect=4,timeo=600,\
  remoteports=10.100.100.1-10.100.100.100,\
  spread_reads,spread_writes \
  10.100.100.1:/nfs3 /mnt/vast
```

**Key VAST mount option for RDMA:**

| Parameter | Value | Purpose |
| --- | --- | --- |
| `proto=rdma` | Required | Selects RDMA transport |

### Verify NFSoRDMA Mount

**Command:**

```bash
findmnt /mnt/vast
mount | grep vast
```

**Expected Result:**

```plaintext
TARGET     SOURCE                FSTYPE OPTIONS
/mnt/vast  10.100.100.1:/nfs3    nfs4   rw,noatime,...,proto=rdma,nconnect=4,...

10.100.100.1:/nfs3 on /mnt/vast type nfs (rw,...,proto=rdma,nconnect=4,...)
```

> **Pass criteria:** `proto=rdma` appears in mount options.

### Basic NFSoRDMA I/O Test

**Command:**

```bash
ls -la /mnt/vast/
touch /mnt/vast/qa-test-$$ && rm /mnt/vast/qa-test-$$
dd if=/dev/zero of=/mnt/vast/qa-write-test bs=1M count=100 oflag=direct
rm /mnt/vast/qa-write-test
```

**Expected Result:**

```plaintext
(total) ...
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.12 s, 873 MB/s
```

### Verify RDMA Transport is Active

**Command:**

```bash
rdma link
rdma statistic show | egrep "np_cnp_sent|rp_cnp_handled|roce_adp_retrans"
```

**Expected Result (healthy):**

```plaintext
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev enp225s0f0
    rate 200 Gb/sec

np_cnp_sent:        0
rp_cnp_handled:     0
roce_adp_retrans:   0
```

### Client NIC QoS (Required for RDMA)

**Command:**

```bash
sudo mlnx_qos -i enp225s0f0
```

**Expected Result:**

```plaintext
Priority trust state: dscp
PFC configuration:
    priority    0   1   2   3   4   5   6   7
    enabled     0   0   0   1   0   0   0   0
```

> PFC must be enabled on **Priority 3** (Traffic Class 3). `Priority trust state: pcp` indicates a configuration mismatch.

### NFSoRDMA Troubleshooting

| Symptom | Check | Command |
| --- | --- | --- |
| Mount fails "Protocol error" | Verify TCP mount first, then RDMA | `mount -o vers=3,tcp <vip>:/export /mnt/test` |
| Server not responding | Check rpcrdma module | `lsmod \\| grep rpcrdma` |
| Intermittent hangs | PFC/ECN counters | `ethtool -S <iface> \\| egrep "pfc\\|pause\\|drop"` |
| High latency / retrans | RDMA congestion stats | `rdma statistic show` |
| RPC debugging | Enable NFS/RPC trace | `rpcdebug -m nfs -s all` |

**Enable RPC debug (triage):**

```bash
rpcdebug -m nfs -s all
rpcdebug -m rpc -s all
echo -n 'module rpcrdma +pflmt' > /sys/kernel/debug/dynamic_debug/control
```

**Collect logs:**

```bash
dmesg -T | grep -iE "nfs|rpc|rdma" > nfsordma_debug.out
```

**Disable debug after collection:**

```bash
rpcdebug -m nfs -c all
rpcdebug -m rpc -c all
echo -n 'module rpcrdma -pflmt' > /sys/kernel/debug/dynamic_debug/control
```

---

## RoCEv2 Diagnostic and Troubleshooting Cheat Sheet on Server

Use this checklist to verify the RoCEv2 environment on ConnectX-based Linux systems.

**Mellanox RoCEv2 Health Checklist**

| Check Type | Command | Healthy Result |
| --- | --- | --- |
| **NIC Detection** | `lspci \\| grep Mellanox` | ConnectX-6/7/BlueField detected. |
| **RDMA Devices** | `ibv_devices` | Active `mlx5_x` devices listed. |
| **Ethernet Link** | `ibstat` | Link layer reports "Ethernet". |
| **RoCE Version** | `show_gids` | **GID Type 2** (RoCEv2) present. |
| **Mapping** | `ibdev2netdev` | Verify RDMA device ↔ Linux interface mapping. |
| **Firmware** | `ethtool -i <interface>` | Driver/firmware versions current. |
| **MTU** | `ip link show` | Jumbo Frames: **9000** or **9216**. |

### Example: NIC Detection

**Command:**

```bash
lspci | grep Mellanox
```

**Expected Result:**

```plaintext
e1:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
e1:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
```

### Example: RDMA Devices

**Command:**

```bash
ibv_devices
```

**Expected Result:**

```plaintext
    device          	   node GUID
    ------          	----------------
    mlx5_0          	e46dab03005db966
    mlx5_1          	e46dab03005db967
```

### Example: RoCE Version (GID Type)

**Command:**

```bash
show_gids
```

**Expected Result:**

```plaintext
DEV	PORT	INDEX	GID					IPv4  		VER	DEV
---	----	-----	---					------------  	---	---
mlx5_0	1	0	fe80:0000:0000:0000:e66d:abff:fe5d:b966			v1	enp225s0f0
mlx5_0	1	1	fe80:0000:0000:0000:e66d:abff:fe5d:b966			v2	enp225s0f0
mlx5_0	1	10	0000:0000:0000:0000:0000:ffff:ac15:5214	172.21.82.20  	v2	enp225s0f0
```

> **GID Type 2 (v2)** confirms RoCEv2 is active.

### Example: Device-to-Interface Mapping

**Command:**

```bash
ibdev2netdev
```

**Expected Result:**

```plaintext
mlx5_0 port 1 ==> enp225s0f0 (Up)
mlx5_1 port 1 ==> enp225s0f1 (Up)
```

---

## Performance Verification and Statistics

Validate throughput and monitor hardware counters to distinguish between a healthy fabric and one experiencing "PFC Storms" or buffer exhaustion.

### Critical Congestion Counters

**Command:**

```bash
rdma statistic show
```

**Expected Result (healthy fabric):**

```plaintext
np_cnp_sent:        0
rp_cnp_handled:     0
roce_adp_retrans:   0
roce_slow_restart:  0
```

| Counter | Meaning | Healthy State |
| --- | --- | --- |
| **np_cnp_sent** | CNPs generated by this node as receiver | Near zero |
| **rp_cnp_handled** | CNPs sender reacted to by throttling | Near zero |
| **roce_adp_retrans** | RoCE adaptive retransmissions (packet loss) | Near zero |
| **roce_slow_restart** | Congestion recovery events (slow-start) | Near zero |

> Continuous increases in `np_cnp_sent` indicate active congestion. Increasing `roce_adp_retrans` suggests the lossless path is failing.

### Bandwidth Test

**Command (server):**

```bash
ib_write_bw -d mlx5_0 -a
```

**Command (client):**

```bash
ib_write_bw -d mlx5_0 -a <server_ip>
```

**Expected Result (200G link):**

```plaintext
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 Link type       : Ethernet
 GID index       : 6
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8388608    5000             22825.74            19233.22                  0.002404
---------------------------------------------------------------------------------------
```

### PFC Counters on Host NIC

**Command:**

```bash
ethtool -S enp225s0f0 | egrep "pfc|pause|prio"
```

**Expected Result (healthy fabric):**

```plaintext
rx_prio3_pause: 0
rx_prio3_pause_duration: 0
tx_prio3_pause: 0
tx_prio3_pause_duration: 0
```

### Quick One-Liner Health Check

**Command:**

```bash
echo "=== RDMA Devices ==="; ibv_devices; \
echo; echo "=== Mapping ==="; ibdev2netdev; \
echo; echo "=== Link Layer ==="; ibstat | grep "Link layer"; \
echo; echo "=== RoCE GIDs ==="; show_gids; \
echo; echo "=== Congestion Counters ==="; rdma statistic show | egrep "roce_adp_retrans|np_cnp_sent|rp_cnp_handled"; \
echo; echo "=== PFC Counters ==="; ethtool -S eth0 | egrep "pfc|pause|prio"
```

**Expected Result (summary):**

```plaintext
=== RDMA Devices ===
device    node GUID
mlx5_0    248a070300a0b1c2

=== Mapping ===
mlx5_0 port 1 ==> eth0 (Up)

=== Link Layer ===
        Link layer: Ethernet

=== RoCE GIDs ===
mlx5_0  1  2  ...  10.0.0.100  v2

=== Congestion Counters ===
np_cnp_sent: 0
rp_cnp_handled: 0
roce_adp_retrans: 0

=== PFC Counters ===
rx_prio3_pause: 0
tx_prio3_pause: 0
```