Performance

Applies to VAST Cluster 5.5.

A VAST cluster serves S3 and NFS from many CNodes at once. A single client connection, though, reaches one virtual IP and so one CNode. Most of what you control as a developer comes down to that: use enough parallel connections, spread them over the cluster's virtual IPs, and size transfers so each request does useful work. This page explains the mechanisms and shows how to apply them in boto3 and on NFS mounts. It doesn't give target numbers; measure with your own data.

The Python blocks are complete on their own. They use the prefix docs-performance/ and the last block removes it.

Where the parallelism is

Clients spread over virtual IPs, and so over CNodes.

Virtual IPs. Clients connect to virtual IPs (VIPs) from a pool, and each VIP is served by one CNode. More VIPs in use means more CNodes working for you.
DNS. If you connect by name, the cluster's DNS server returns one VIP from the pool for each lookup (or your own DNS server returns the pool's addresses and the client picks one). Many clients, or one client that resolves the name for each new connection, end up spread over the pool. One long-lived connection doesn't move.
Connections. HTTP clients reuse connections. A boto3 client with a pool of connections opens them as it needs them, each to the address the name resolved to when it opened.

So a single-threaded client on one connection uses one CNode however big the cluster is. Throughput grows with concurrent requests, and with the number of VIPs they're spread over.

Parallel and multipart transfers

For objects:

Use concurrency. Upload and download many objects at once with a thread pool, and make large objects multipart so their parts transfer in parallel. In boto3, TransferConfig(max_concurrency=...) sets the threads for one transfer, and Config(max_pool_connections=...) the connections a client keeps. Make the pool at least as large as the number of threads that share the client.
Size parts for the work per request. Each part is one request, and the cluster checks each part's checksum when the client sends one. Very small parts spend more time on per-request overhead; very large parts leave fewer requests to run in parallel and take longer to retry. Keep parts at least 5 MiB, as S3 clients expect, and see large objects.
Read in ranges. Parallel ranged GETs speed up large downloads (boto3's download_file does this), and columnar readers such as PyArrow read only the byte ranges they need.
Avoid listing as a queue. Listing a large prefix is sequential, 1,000 keys per call. To find new objects, use bucket notifications or query the VAST Catalog instead of polling listings.

This example uploads 32 small objects from a thread pool, with one client per endpoint. Set AWS_S3_ENDPOINT_URLS to several of the pool's VIPs (comma-separated) to spread the work; with only AWS_S3_ENDPOINT_URL it uses that one:

import os
from concurrent.futures import ThreadPoolExecutor

import boto3
from botocore.config import Config

endpoints = [e for e in os.environ.get("AWS_S3_ENDPOINT_URLS", "").split(",") if e] or [os.environ["AWS_S3_ENDPOINT_URL"]]
config = Config(signature_version="s3v4", s3={"addressing_style": "path"}, max_pool_connections=16)
clients = [boto3.client("s3", endpoint_url=e, region_name="us-east-1", config=config) for e in endpoints]
bucket = os.environ["DATASTORE_BUCKET"]

def upload(i):
    client = clients[i % len(clients)]  # round-robin over the endpoints
    client.put_object(Bucket=bucket, Key=f"docs-performance/part-{i:03d}.bin", Body=b"x" * 256 * 1024)
    return i

with ThreadPoolExecutor(max_workers=16) as pool:
    done = list(pool.map(upload, range(32)))

print(len(done), "objects uploaded over", len(endpoints), "endpoint(s)")

32 objects uploaded over 2 endpoint(s)

To compare settings, time the same block with different max_workers, object sizes and endpoint lists, against data like yours.

HTTP or HTTPS

Both are enabled by default. HTTPS adds encryption work on the client and the cluster; whether that matters depends on your workload, so measure both if you have the choice. Some sites disable plain HTTP. With HTTPS, reuse clients rather than creating one per request, so connections (and their TLS handshakes) are reused.

NFS

The same principle applies to NFS mounts:

A default mount is one TCP connection to one VIP, and so one CNode.
nconnect=N opens up to 16 connections to the same VIP. That raises the bandwidth of one mount, but all of them still go to one CNode, so it doesn't add IOPS.
Multipath (with the VAST-NFS client driver) spreads a mount's connections over a range of VIPs with remoteports=, and so over many CNodes. Give it the pool's whole range for an even spread.
NFS over RDMA (proto=rdma) bypasses the TCP stack, for lower latency and CPU use, on InfiniBand or RoCE networks. GPUDirect Storage needs an RDMA mount.
Many clients each mounting by name spread over the pool in the same way as S3 clients.

Mount syntax is in NFS for applications.

What's not yours to tune

Quality-of-service limits, the number of VIPs in a pool and which CNodes serve them are cluster settings. If you hit a limit you can't explain from the client side, ask your platform team whether a QoS policy applies to your user or view (QoS Overview). QoS counts every metadata request as an I/O, so workloads with many small files or listings reach IOPS limits first; see quotas and quality of service.

Clean up

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [{"Key": o["Key"]} for o in s3.list_objects_v2(Bucket=bucket, Prefix="docs-performance/").get("Contents", [])]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")

deleted 32 objects