Overview

Applies to VAST Cluster 5.5.

VAST DataStore is the file and object storage of a VAST cluster. Applications reach it over S3, NFS and SMB, and the same data can be reached through more than one of them: an object your application writes over S3 can be read as a file over NFS, and a file written over NFS is an object in the bucket. These docs are for developers who build against VAST S3 and NFS: which S3 calls work and how they differ from AWS, how to configure the usual clients, how to move large data quickly, and what to watch when two protocols share the same data.

What it is

• One namespace, several protocols. A cluster stores files, objects and tables in one namespace, the Element Store. A view exposes a path in it to one or more protocols: an NFS export, an SMB share, an S3 bucket, or several of these at once. A bucket is a view with the S3 Bucket protocol. See concepts.
• S3-compatible, not a copy of AWS. The cluster implements a subset of the S3 REST API, with AWS Signature Version 2 and 4, over HTTP or HTTPS. Most everyday calls work as they do on AWS. Some differ, some are missing, and there are a few VAST additions, such as renaming an object in one request. See S3 compatibility.
• NFSv3, NFSv4.1 and NFSv4.2, with byte-range locks, NFSv4 ACLs, NFSv4.2 extended attributes and NFSv4 file delegations, plus client-side options for parallel connections. See NFS for applications.
• SMB 2.1 and 3.0 for Windows clients. These docs cover SMB only as it affects data shared with S3 and NFS: see the SMB reference.
• Your identity is a VAST user. S3 requests are signed with an access key pair that belongs to one VAST user in one tenant. The same user can own files over NFS and SMB. How S3 permissions and file permissions combine is decided per view, by the view policy's security flavor. See multiprotocol access.
• Set up by your platform team. Views, view policies, protocols, quotas, users and S3 keys are admin tasks. Access lists what to ask for.

One namespace; a view exposes a path to one or more protocols.

Use cases and considerations

Example use cases:

• Object storage for applications with boto3, the AWS CLI, s3fs or PyArrow, using the same code you would use against AWS S3 with a different endpoint. See the quickstart and Python clients.
• A data lake for Spark, Trino and pandas over S3, with Parquet files that VAST DataBase can also import. See Spark and Hadoop and, for importing, Import Parquet files.
• Large files and fast transfers: multipart uploads, ranged and parallel reads, and spreading clients over the cluster's virtual IPs. See large objects and multipart uploads and performance.
• The same data over S3 and NFS: an instrument or a POSIX application writes files over NFS, and a cloud-native service reads them as objects, or the other way round. See multiprotocol access.
• Keeping old copies and protecting objects: object versioning, object locking and legal holds. See versioning and object lock.
• Reacting to new objects with bucket notifications to a VAST Event Broker topic. See Publish S3 bucket events to a topic.
• Finding objects by tag, size or age without listing buckets, with VAST Catalog. See Find objects by S3 tags and metadata.

Important to note:

• Bucket names are unique across the whole cluster, not only in your tenant, and a bucket can't be renamed or moved once it exists.
• Bucket configuration that AWS exposes through the S3 API is partly admin-side here. Lifecycle rules (expiration) are set by your platform team, not with PutBucketLifecycleConfiguration. Versioning, once enabled, can't be suspended. See S3 compatibility.
• Some features keep a bucket to S3 only. A bucket with object locking can't also be exposed over NFS or SMB, and versioned objects are read-only over NFS and SMB.
• Multiprotocol behaviour depends on the view's security flavor. It decides which protocol's permissions apply. Identity and bucket policies apply only with the S3 Native flavor. Ask which flavor your view uses before you design around it.

How to read these docs

• Start: this overview, then the quickstart: put, list, read and delete objects with boto3.
• Concepts: buckets, views, keys, identities and security flavors.
• Guides: S3 objects (metadata, tags, ranged reads, presigned URLs, conditional requests, rename), large objects and multipart uploads, versioning and object lock, multiprotocol access, NFS for applications and performance.
• Best practices: planned topics.
• Integrations: Python (boto3, s3fs, PyArrow), Spark and Hadoop and the AWS CLI, s3cmd and rclone.
• Reference: S3 compatibility, limits, NFS and SMB.
• Operations: access (keys, identity and bucket policies, temporary credentials) and CI/CD and automation.
• Troubleshooting: symptom, cause and fix.

Where to start

1. Admin task: send to your platform team. Ask for a bucket (or an NFS export) and an S3 access key pair. Access lists what to ask for.
2. Run the quickstart.
3. If the same data will be used over S3 and NFS, read multiprotocol access before you write any data.

See also

• VAST DataBase overview: tables in the same cluster, reached with the same S3 keys and endpoints.
• VAST Catalog overview: a queryable index of your files and objects.
• KB: Overview of VAST Cluster S3 Implementation, NFS Overview and Overview of Multi-Protocol Storage Access: the admin guides.