Quickstart

Applies to VAST Cluster 5.5.

In this quickstart you'll connect to VAST S3 with boto3, write an object with metadata, list it, read it back and delete it. Everything happens under the key prefix docs-quickstart/ in a bucket you already have, and the last step removes what the quickstart created. It takes about 10 minutes once your platform team has given you a bucket and a key pair.

Each code block is complete on its own: it imports what it needs and connects. Below each block is the output you should see.

Before you start

You need:

• A bucket you can write to, and its name. Admin task: send to your platform team: a view with the S3 Bucket protocol, with you (or your application's user) as the bucket owner or granted access through an identity policy. See access.
• An S3 access key pair for that user, and an endpoint: the DNS name or a virtual IP of a virtual IP pool that serves S3, with http:// or https:// in front. The port isn't configurable: use 80 for HTTP and 443 for HTTPS, which is what leaving it out does.
• Python 3.10 or later, and boto3:

pip install "boto3>=1.36"

Put your connection details in environment variables:

export AWS_S3_ENDPOINT_URL="http://s3.example.com"   # your endpoint
export AWS_ACCESS_KEY_ID="<access key>"
export AWS_SECRET_ACCESS_KEY="<secret key>"
export DATASTORE_BUCKET="<your bucket>"

If your endpoint uses HTTPS with a certificate your system doesn't trust, also set AWS_CA_BUNDLE to a file with the certificate. See Python clients.

What your platform team sets up, in order, before you connect.

Connect

Create a client and check that you can reach the bucket. The client uses path-style addressing (http://endpoint/bucket/key) and Signature Version 4, the configuration to use with VAST:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",  # any string works; SigV4 needs one
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

s3.head_bucket(Bucket=bucket)
print("bucket", bucket, "is reachable")

bucket dst-data is reachable

If this fails with 403, your key can't access the bucket; with 404, the bucket doesn't exist or isn't in your key's tenant. See troubleshooting.

Write an object

Write a small object with a content type and one piece of your own metadata. The ETag the cluster returns for a single-part upload is the MD5 checksum of the data:

import hashlib
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

body = b"hello from the VAST DataStore quickstart\n"
resp = s3.put_object(
    Bucket=bucket,
    Key="docs-quickstart/hello.txt",
    Body=body,
    ContentType="text/plain",
    Metadata={"source": "quickstart"},
)
print("ETag is the MD5 of the data:", resp["ETag"].strip('"') == hashlib.md5(body).hexdigest())

ETag is the MD5 of the data: True

List objects

List what's under the prefix. A listing returns up to 1,000 keys at a time; the paginator asks for the next page for you:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix="docs-quickstart/"):
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"])

docs-quickstart/hello.txt 41

Read it back

Read the object, and then only its metadata with head_object:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

obj = s3.get_object(Bucket=bucket, Key="docs-quickstart/hello.txt")
print(obj["Body"].read().decode(), end="")

head = s3.head_object(Bucket=bucket, Key="docs-quickstart/hello.txt")
print(head["ContentType"], head["ContentLength"], head["Metadata"])

hello from the VAST DataStore quickstart
text/plain 41 {'source': 'quickstart'}

Your metadata comes back without its x-amz-meta- prefix, as on AWS. More on metadata, tags and ranged reads is in S3 objects.

Clean up

Delete everything under the prefix:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [
    {"Key": obj["Key"]}
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix="docs-quickstart/")
    for obj in page.get("Contents", [])
]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")

deleted 1 objects

Next steps

• Concepts: how buckets, views and keys relate, and what decides your permissions.
• S3 objects: metadata, tags, ranged reads, presigned URLs, conditional requests and renames.
• Multiprotocol access: read the same objects as files over NFS.
• Python clients: s3fs and PyArrow.