NFS for applications

Applies to VAST Cluster 5.5.

This guide is for developers whose applications read and write files on a VAST NFS export: which NFS version to ask for, which mount options matter, and how locking, renames, extended attributes and caching behave. Mounting and the export itself are set up with your platform team; the examples then work on the mounted directory with ordinary Python file calls, under docs-nfs-for-applications/, and the last block removes it.

The mount commands are shown for reference and aren't run here. The Python blocks are complete on their own, and below each is the output you should see.

Choose an NFS version

All three versions reach the same files. Pick by what your application needs:

Notes:

• A view lists NFSv4 as one protocol; clients mount it as 4.1 or 4.2. NFSv4.2 also has to be enabled for the tenant.
• An export with NFSv4 needs NFSv4 on every view above it in the path, or mounting fails even though I/O through an existing mount works.
• With NFSv4.1 or 4.2, a user or group the cluster can't resolve shows up as nobody on the client (the server answers BADOWNER). That usually means ID mapping isn't set up on the client; see NFSv4.1 ID Mapping.

Mount options

These are client-side options, set in your mount command or /etc/fstab. Use the address or DNS name your platform team gives you, and the view's path or its NFS alias (NFSv3).

# NFSv3
sudo mount -t nfs -o vers=3 vast.example.com:/projects /mnt/projects

# NFSv4.2 (also needed for extended attributes and recommended for delegations)
sudo mount -t nfs -o vers=4.2 vast.example.com:/projects /mnt/projects

# Several TCP connections to the same virtual IP (NFSv3 or NFSv4.1; up to 16)
sudo mount -t nfs -o vers=3,nconnect=8 vast.example.com:/projects /mnt/projects

For speed, see performance. TLS, mTLS and Kerberos for NFS are set up with your platform team: Configuring TLS Encryption with NFS, mTLS Authentication for NFS, Kerberos Authentication for NFSv4.1.

Before you run the examples

Mount an export you can write to and point NFS_MOUNT at it:

export NFS_MOUNT=/mnt/projects

Publish files atomically

A reader should never see half a file. Write under a temporary name in the same directory, flush it to the server, then rename. A rename within a directory is atomic, and over S3 too the object appears only under its final name (multiprotocol access):

import os

mount = os.environ["NFS_MOUNT"]
directory = os.path.join(mount, "docs-nfs-for-applications")
os.makedirs(directory, exist_ok=True)

final = os.path.join(directory, "results.csv")
tmp = os.path.join(directory, ".results.csv.tmp")
with open(tmp, "wb") as f:
    f.write(b"id,score\n1,0.91\n2,0.87\n")
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, final)

print(sorted(os.listdir(directory)), os.path.getsize(final), "bytes")

['results.csv'] 23 bytes

os.replace overwrites an existing file in one step. Readers that already have the old file open keep reading the old data.

Lock a byte range

VAST supports byte-range locks: NLM locks for NFSv3 and the built-in locking of NFSv4.1 and later. Locks are advisory: they stop other processes that also ask for a lock, not processes that just write. This example takes an exclusive lock and shows that a second process can't take it at the same time:

import fcntl
import os
import subprocess
import sys

mount = os.environ["NFS_MOUNT"]
path = os.path.join(mount, "docs-nfs-for-applications", "counter.lock")
os.makedirs(os.path.dirname(path), exist_ok=True)

# The second process tries the same lock without waiting.
second = """
import fcntl, sys
with open(sys.argv[1], "r+b") as f:
    try:
        fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB, 1, 0)
        print("second process: got the lock")
    except OSError as e:
        print(f"second process: lock refused ({e.__class__.__name__})")
"""

with open(path, "w+b") as f:
    f.write(b"0")
    f.flush()
    fcntl.lockf(f, fcntl.LOCK_EX, 1, 0)
    print("first process: holds the lock on byte 0")
    out = subprocess.run([sys.executable, "-c", second, path], capture_output=True, text=True, check=True)
    print(out.stdout.strip())
    fcntl.lockf(f, fcntl.LOCK_UN, 1, 0)
print("released")

first process: holds the lock on byte 0
second process: lock refused (BlockingIOError)
released

Over NFSv3 this needs the client's lock service (NLM); a mount with nolock only locks on the local machine. The number of locks one file can hold at once is limited, so lock files or ranges, not millions of tiny ranges in one file. If a client crashes, its locks are released when its NFS lock state is cleaned up; your platform team can also release stale locks.

Tag files with extended attributes

On NFSv4.2, files and directories can carry your own key-value metadata as extended attributes, for example user.dataset. On other mounts the call fails, and this example says why:

import errno
import os

mount = os.environ["NFS_MOUNT"]
path = os.path.join(mount, "docs-nfs-for-applications", "results.csv")

try:
    os.setxattr(path, "user.dataset", b"trial-7")
    print("xattrs:", {name: os.getxattr(path, name) for name in os.listxattr(path)})
    os.removexattr(path, "user.dataset")
except OSError as e:
    print("extended attributes not available on this mount:", errno.errorcode.get(e.errno, e.errno))

Extended attributes need an NFSv4.2 mount and NFSv4.2 enabled for the tenant. Linux clients usually allow only the user. namespace. By default names can be up to 128 characters and values up to 256; your platform team can raise that per view policy. Symlinks can't carry attributes: an attribute set through a symlink goes on its target. If the VAST Catalog is enabled, your platform team can index chosen attributes as Catalog columns (VAST Catalog concepts).

Caching and consistency

• Close-to-open. Linux NFS clients cache file data and attributes. A change made by another client is guaranteed to be visible after you close and reopen the file; check-then-read on an open file can see stale data. Write with fsync (or close) before you tell another process the data is ready.
• Delegations. If NFSv4 file delegations are enabled for the view, a client holding a delegation caches more aggressively. Changes made over S3, SMB or NFSv3 may not be visible to it at once. If your application mixes protocols on the same files, ask whether delegations are on.
• Names. By default a single file or directory name can be up to 255 characters, with the characters every supported protocol allows. A view policy can relax this to each protocol's own limits. Files that use the relaxed limits may then be unreachable over the other protocols.

Clean up

import os
import shutil

mount = os.environ["NFS_MOUNT"]
shutil.rmtree(os.path.join(mount, "docs-nfs-for-applications"), ignore_errors=True)
print("left:", os.path.exists(os.path.join(mount, "docs-nfs-for-applications")))

left: False

For deleting very large directory trees, your platform team can enable the cluster's trash folder: you move a directory into it and the cluster deletes it in the background (Trash Folder Access).

See also

• NFS reference.
• Multiprotocol access: the same files over S3.
• KB: NFS Overview, NFS Features for Enhanced Performance and Byte Range File Locking with NFS.