---
title: "Troubleshooting NFS Performance Using ebpf Tools"
slug: "troubleshooting-nfs-performance-using-ebpf-tools"
description: "Troubleshooting NFS Performance Using ebpf Tools"
updated: 2026-10-27T19:37:32Z
published: 2026-10-28T19:52:20Z
canonical: "docs.vastdata.com/troubleshooting-nfs-performance-using-ebpf-tools"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting NFS Performance Using ebpf Tools

## Overview

This guide provides instructions for diagnosing NFS client performance issues using eBPF tools from the BCC (BPF Compiler Collection) toolkit. These tools provide real-time visibility into NFS operations without requiring application changes or system restarts.

## Prerequisites

- Root access on the client system.
- BCC tools installed (`apt-get install bpfcc-tools` or `yum install bcc-tools`).
- Linux kernel 4.1+ (newer kernels have better eBPF support).

## Quick Diagnosis Workflow

### Step 1: Check Overall NFS Operation Latency

**Tool: nfsslower**

```plaintext
# Show NFS operations slower than 10ms (default)
sudo nfsslower

# Show NFS operations slower than 1ms
sudo nfsslower 1

# Show NFS operations slower than 1ms with CSV output
```

Example:

```bash
$ sudo /usr/share/bcc/tools/nfsslower -j 1
ENDTIME_us,TASK,PID,TYPE,BYTES,OFFSET_b,LATENCY_us,FILE
805100595998,b'python3',77159,W,2665025,0,1079,b'f000000189903.dat'
805103189703,b'python3',77162,W,2178908,0,4069,b'f000000160747.dat'
805104571360,b'python3',77155,W,2837177,0,1213,b'f000000293438.dat'
805106258905,b'python3',77172,W,3114077,0,1431,b'f000000193137.dat'
805107286939,b'python3',77158,W,2617788,0,1125,b'f000000264222.dat'
805107693680,b'python3',77142,W,2768168,0,1193,b'f000000259502.dat'
805108374832,b'python3',77145,W,2840249,0,1194,b'f000000246017.dat'
805108716805,b'python3',77152,W,3040844,0,7169,b'f000000184216.dat'
805108748857,b'python3',77160,W,2950676,0,1220,b'f000000258253.dat'
805109104171,b'python3',77171,W,2796053,0,9722,b'f000000250545.dat'
^C $
$
$ sudo /usr/share/bcc/tools/nfsslower 1
Tracing NFS operations that are slower than 1 ms... Ctrl-C to quit
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
23:30:13 python3        77157  W 2921397 0           1.08 f000000267036.dat
23:30:14 python3        77143  W 3040384 0           1.18 f000000275327.dat
23:30:16 python3        77159  W 2862232 0           1.26 f000000189908.dat
23:30:18 python3        77149  W 2980817 0           1.24 f000000285363.dat
23:30:19 python3        77145  W 2682881 0           1.10 f000000246021.dat
23:30:21 python3        77152  W 2938159 0           1.14 f000000184219.dat
23:30:23 python3        77147  W 2763934 0           1.16 f000000224322.dat
23:30:25 python3        77168  W 2601147 0           1.05 f000000279048.dat
23:30:25 python3        77142  W 3073096 0           1.25 f000000259506.dat
^C $
```

**What to look for:**

- **Normal**: Most operations complete in <1ms for cached data, <10ms for network operations.
- **Warning**: Regular operations taking 50-100ms.
- **Bad**: Operations consistently over 100ms or occasional multi-second delays.

### Step 2: Analyze File System Activity Patterns

**Tool: vfsstat**

```plaintext
# Show VFS statistics with 1-second intervals
sudo vfsstat 1

# Show 5 samples then exit
sudo vfsstat 1 5
```

**Example output:**

```plaintext
$ sudo /usr/share/bcc/tools/vfsstat 1
TIME         READ/s  WRITE/s  FSYNC/s   OPEN/s CREATE/s UNLINK/s  MKDIR/s  RMDIR/s
23:34:12:      3651       77        1        7        0        0        0        0
23:34:13:        30       56        0        1        0        0        0        0
23:34:14:        55       62        0       18        0        0        0        0
23:34:15:        24       47        0        1        0        0        0        0
23:34:16:        46       72        0        1        0        0        0        0
23:34:17:        28       53        0        1        0        0        0        0
23:34:18:        45       72        0        1        0        0        0        0
23:34:19:        25       52        0        4        0        0        0        0
23:34:20:        40       63        0        3        0        0        0        0
....

$ sudo /usr/share/bcc/tools/vfsstat 1 5
TIME         READ/s  WRITE/s  FSYNC/s   OPEN/s CREATE/s UNLINK/s  MKDIR/s  RMDIR/s
23:35:01:      3655       57        1       21        0        0        0        0
23:35:02:        45       70        0        1        0        0        0        0
23:35:03:        26       52        0        3        0        0        0        0
23:35:04:        56       64        0       17        0        0        0        0
23:35:05:        27       54        0        2        0        0        0        0
```

### Step 3: Identify Slow File Operations

**Tool: fileslower**

```bash
# Trace file operations slower than 10ms
sudo fileslower 10

# Include file type information
sudo fileslower -t 10

# Filter for specific process
sudo fileslower -p <PID> 10
```

**What to look for:**

- **Normal**: Occasional slow operations during cache misses.
- **Warning**: Frequent slow operations on the same files.
- **Bad**: All operations are slow, indicating systemic issues.

Example:

```bash
$ sudo /usr/share/bcc/tools/fileslower 1
Tracing sync read/writes slower than 1 ms
TIME(s)  COMM           TID    D BYTES   LAT(ms) FILENAME
0.492    python3        77141  W 2906967    1.14 f000000219886.dat
0.694    python3        77171  W 2959081    1.30 f000000250571.dat
1.230    python3        77147  W 2543445    1.25 f000000224344.dat
2.534    python3        77154  W 2507605    1.41 f000000283097.dat
4.481    python3        77156  W 2380353    1.21 f000000298343.dat
5.626    python3        77162  W 2872629    7.31 f000000160773.dat
5.803    python3        77145  W 2697660    1.73 f000000246044.dat
```

### Step 4: Check Network Health (Critical for NFS)

**Tool: tcpretrans**

```bash
# Monitor TCP retransmissions
sudo tcpretrans

# Include TCP state
sudo tcpretrans -s

# Filter for NFS server IP
sudo tcpretrans -i <NFS_SERVER_IP>
```

**What to look for:**

- **Normal**: Occasional retransmissions (<0.1% of packets).
- **Warning**: Regular retransmissions (0.1-1%).
- **Bad**: Frequent retransmissions (>1%), indicating network problems.

Example:

```bash
$ sudo /usr/share/bcc/tools/tcpretrans
Tracing retransmits ... Hit Ctrl-C to end
TIME     PID     IP LADDR:LPORT          T> RADDR:RPORT          STATE
23:39:17 77143   4  172.200.13.56:937    R> 172.200.204.4:2049   ESTABLISHED
23:39:17 77161   4  172.200.13.56:880    R> 172.200.204.4:2049   ESTABLISHED
23:39:20 77172   4  172.200.13.56:937    R> 172.200.204.4:2049   ESTABLISHED
23:40:11 77154   4  172.200.13.56:937    R> 172.200.204.4:2049   ESTABLISHED
23:40:38 77149   4  172.200.13.56:957    R> 172.200.204.4:2049   ESTABLISHED
...
$ sudo /usr/share/bcc/tools/tcpretrans -s
Tracing retransmits ... Hit Ctrl-C to end
TIME     PID     IP LADDR:LPORT          T> RADDR:RPORT          STATE        SEQ
23:42:16 77156   4  172.200.13.56:880    R> 172.200.204.4:2049   ESTABLISHED  4014085456
23:44:09 77161   4  172.200.13.56:937    R> 172.200.204.4:2049   ESTABLISHED  3479636909
...
```

**Tool: tcptop**

```bash
# Show TCP throughput by connection
sudo tcptop

# Update every 1 seconds, stop after 5 samples
sudo tcptop 1 5 

# Filter for specific port (2049 for NFS, and -C to not clear the screen)
sudo tcptop -p 2049 -C
```

### Step 5: Analyze Metadata Operations

**Tool: statsnoop**

```bash
# Trace all stat() calls
sudo statsnoop

# Include timestamps
sudo statsnoop -t

# Filter for failed stat calls
sudo statsnoop -x
```

**What to look for:**

- **Normal**: Stat calls complete quickly (<1ms).
- **Warning**: High volume of stat calls or calls taking >10ms.
- **Bad**: Applications doing repeated stats on the same files (indicates poor caching).

## Quick Reference Card

| **Tool** | **Purpose** | **Key Flag** | **Normal Range** |
| --- | --- | --- | --- |
| nfsslower | NFS operation latency | threshold ms (10ms default) | <5ms |
| vfsstat | VFS layer statistics | interval | Displays the current operation mix |
| filesslower | Slow file operations | threshold ms (10ms default) | <5ms |
| tcpretrans | Network retransmissions | -s (state) -i (ip of vast ip) | <0.1% |
| tcptop | TCP Throughput | -p 2049 | Consistent flow |
| statssnoop | stat() calls | -t (timestamp) | <1ms latency |

Remember: Always establish baseline performance during normal operations to better identify anomalies during issues.