Multiprotocol access

Applies to VAST Cluster 5.5.

On a view with both S3 and NFS, every object is a file and every file is an object. This guide shows how keys map to paths, writes data through one protocol and reads it through the other, and explains what decides ownership and permissions, which keys have no file, and what else to watch. It uses the prefix (and directory) docs-multiprotocol/, and the last block removes it.

Each code block is complete on its own. It uses boto3 for S3 and ordinary Python file operations on an NFS mount of the same bucket. Below each block is the output you should see.

Before you start

You need:

• A view with S3 Bucket and NFS (NFSv3, NFSv4 or both). Admin task: send to your platform team: ask for both protocols on the view, a host rule that lets your client mount it read-write, and tell them which user your NFS client runs as. Ask which security flavor the view policy uses; the rest of this page depends on it.
• The bucket mounted over NFS on the machine where you run the examples, at the view's path, so that the bucket's root is the mount's root. For example:

sudo mkdir -p /mnt/datastore
sudo mount -t nfs -o vers=3 s3.example.com:/datastore-view /mnt/datastore
export NFS_MOUNT=/mnt/datastore

• The bucket, key pair and environment variables from the quickstart, for the same bucket.

Mount options are explained in NFS for applications.

Keys are paths

One view, two protocols: a key is a path.

The object key is the file's path relative to the view. Write an object over S3 and read it as a file:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

s3.put_object(Bucket=bucket, Key="docs-multiprotocol/from-s3/data.csv", Body=b"id,value\n1,10\n2,20\n")

path = os.path.join(mount, "docs-multiprotocol", "from-s3", "data.csv")
print("is a file:", os.path.isfile(path), "| parent is a directory:", os.path.isdir(os.path.dirname(path)))
try:
    with open(path, "rb") as f:
        print("read over NFS:", f.read())
except PermissionError as e:
    print("read over NFS refused:", e.errno, e.strerror)

is a file: True | parent is a directory: True
read over NFS: b'id,value\n1,10\n2,20\n'

The directories in the key (docs-multiprotocol/from-s3/) exist as real directories over NFS. If the read is refused, see ownership and permissions below.

Files are objects

Write a file over NFS and read it as an object. Its ETag isn't the MD5 of its data: for files created or changed over NFS or SMB, the cluster computes an ETag from the file's metadata, which starts with a fixed prefix so you can tell it apart:

import hashlib
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

directory = os.path.join(mount, "docs-multiprotocol", "from-nfs")
os.makedirs(directory, exist_ok=True)
data = b"written over NFS\n"
with open(os.path.join(directory, "note.txt"), "wb") as f:
    f.write(data)

obj = s3.get_object(Bucket=bucket, Key="docs-multiprotocol/from-nfs/note.txt")
etag = obj["ETag"].strip('"')
print("read over S3:", obj["Body"].read())
print("ETag is the MD5 of the data:", etag == hashlib.md5(data).hexdigest())
print("ETag has the non-S3 prefix:", etag.startswith("48ed760a74"))

listed = s3.list_objects_v2(Bucket=bucket, Prefix="docs-multiprotocol/", Delimiter="/")
print("prefixes:", [p["Prefix"] for p in listed.get("CommonPrefixes", [])])

read over S3: b'written over NFS\n'
ETag is the MD5 of the data: False
ETag has the non-S3 prefix: True
prefixes: ['docs-multiprotocol/from-nfs/', 'docs-multiprotocol/from-s3/']

So don't use the ETag of a file written over NFS or SMB as a content checksum. Copying such an object over S3 gives the copy a normal ETag computed from the data.

Renames and deletes are shared

A rename over NFS is a rename of the object. There's no copy:

import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

base = os.path.join(mount, "docs-multiprotocol", "from-nfs")
os.rename(os.path.join(base, "note.txt"), os.path.join(base, "note-final.txt"))

print("new key:", s3.head_object(Bucket=bucket, Key="docs-multiprotocol/from-nfs/note-final.txt")["ContentLength"], "bytes")
try:
    s3.head_object(Bucket=bucket, Key="docs-multiprotocol/from-nfs/note.txt")
except ClientError as e:
    print("old key:", e.response["Error"]["Code"])

s3.delete_object(Bucket=bucket, Key="docs-multiprotocol/from-nfs/note-final.txt")
print("file after an S3 delete:", os.path.exists(os.path.join(base, "note-final.txt")))

new key: 17 bytes
old key: 404
file after an S3 delete: False

head_object errors have no response body, so the error code is the HTTP status, 404.

That makes a common pattern safe: a producer writes a file over NFS under a temporary name and renames it when it's complete, and S3 readers only ever see complete objects under the final name.

Keys that have no file

Some S3 keys can't be represented as paths, so NFS and SMB clients can't see those objects at all:

• keys that start with /, end with /, or contain //;
• keys with ./, ../, /./ or /../ path parts. These are refused at upload unless the view policy allows them ("S3 special chars support").

A key ending in / is a valid object (S3 tools use them as "folder" markers), but it isn't a directory over NFS:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

s3.put_object(Bucket=bucket, Key="docs-multiprotocol/marker/", Body=b"")
print("marker object over S3:", s3.head_object(Bucket=bucket, Key="docs-multiprotocol/marker/")["ContentLength"], "bytes")
path = os.path.join(mount, "docs-multiprotocol", "marker")
print("over NFS: exists", os.path.exists(path), "| directory", os.path.isdir(path))

marker object over S3: 0 bytes
over NFS: exists False | directory False

Also, one name can't be a file and a directory at once: with an object a/b you can't also have a/b/c. If an application might write both, keep its keys file-like: no leading or trailing slashes, no empty path parts.

Ownership and permissions

The security flavor decides which protocol's permission model applies to everyone. On a view with S3 and NFS:

What this means in practice:

• S3 Native: the identity that wrote an object owns it, and by default only the owner can read it, over S3 or NFS. If another user must read over NFS what your application writes over S3, the platform team grants that user s3:GetObject (and s3:ListBucket) in an identity or bucket policy. There's no traverse check on directories. View policy options decide whether NFS listings follow the policies and whether policies can name individual paths.
• NFS: S3 behaves like a POSIX user. An S3 user needs traverse permission on every directory in the key's path, and write permission on the parent directory to upload. Identity and bucket policies don't apply. The mode bits that objects written over S3 get come from the view policy's "permissions for new files and directories", or from the parent directory's POSIX ACL.
• Mixed Last Wins: any protocol can set permissions, and the last change wins. Policies don't apply.
• Groups: on SMB and NFSv3 views, a new file's group is the creating user's group by default. The platform team can switch it to BSD-style inheritance from the parent directory.

Run this to see who owns what on your view:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

st = os.stat(os.path.join(mount, "docs-multiprotocol", "from-s3", "data.csv"))
print("this NFS client runs as uid", os.getuid())
print("object written over S3: uid", st.st_uid, "gid", st.st_gid, "mode", oct(st.st_mode & 0o777))
acl = s3.get_object_acl(Bucket=bucket, Key="docs-multiprotocol/from-s3/data.csv")
print("S3 ACL grants:", [g["Permission"] for g in acl["Grants"]])

this NFS client runs as uid 8801
object written over S3: uid 8801 gid 65534 mode 0o700
S3 ACL grants: ['FULL_CONTROL']

Here the NFS client runs as the UID of the key's VAST user, so it owns the file that user wrote over S3. The group is 65534 because that user has no leading group, and the mode bits are 0700. A process running as another UID, with no policy granting it access, was refused (EACCES).

Ask your platform team which VAST user your S3 key belongs to and which UID that user has; with that, the numbers above tell you how the two identities line up.

Versioned and locked buckets

• Versioning: objects in a versioned bucket can be read over NFS and SMB but not changed through them. If an application needs to modify files in place over NFS, don't use a versioned bucket.
• Object locking: a bucket with object locking can't also be exposed over NFS or SMB.
• SSE-C: objects encrypted with a customer-provided key can only be read over S3.

Concurrent access

Access from several protocols to the same files is supported, but not as a substitute for coordination:

• There's no lock shared across protocols for S3. S3 has no locks. A file can be replaced over S3 while an NFS client has it open. Coordinate writers yourself, for example with write-then-rename or with conditional writes.
• NFSv4 delegations delay what other protocols see. On a view with NFSv4 file delegations enabled, close-to-open consistency holds only between NFSv4 clients. A change made over S3, SMB or NFSv3 may not be visible at once to an NFSv4 client holding a delegation on the file. Ask whether delegations are on for your view.
• SMB alternate data streams aren't visible over other protocols.

Clean up

import os
import shutil

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
mount = os.environ["NFS_MOUNT"]

keys = [{"Key": o["Key"]} for o in s3.list_objects_v2(Bucket=bucket, Prefix="docs-multiprotocol/").get("Contents", [])]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
shutil.rmtree(os.path.join(mount, "docs-multiprotocol"), ignore_errors=True)
print("cleaned up; directory left:", os.path.exists(os.path.join(mount, "docs-multiprotocol")))

cleaned up; directory left: False

See also

• Concepts: what decides whether a request is allowed.
• NFS for applications and the SMB reference.
• KB: Controlling File and Directory Permissions Across Protocols, S3 Directories and Multiprotocol Best Practices and Limitations.