Work with S3 objects

Applies to VAST Cluster 5.5.

This guide covers the object operations an application uses every day on VAST S3: writing objects with metadata and tags, listing by prefix and delimiter, reading part of an object, copying and renaming, conditional writes and reads, and sharing an object through a presigned URL. Everything happens under the key prefix docs-s3-objects/, and the last block removes it.

Each code block is complete on its own. Below each block is the output you should see.

Before you start

You need the bucket, key pair and environment variables from the quickstart, and a bucket without versioning: conditional writes aren't supported in versioned buckets. Your key needs to read, write, list, delete and tag objects in the bucket.

Write objects with metadata and tags

Write a few objects. The first carries a content type, your own metadata (sent as x-amz-meta- headers) and tags (sent URL-encoded in the x-amz-tagging header):

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
prefix = "docs-s3-objects/"

s3.put_object(
    Bucket=bucket,
    Key=prefix + "reports/2026/q3.csv",
    Body=b"region,total\nnorth,10\nsouth,20\n",
    ContentType="text/csv",
    Metadata={"owner-team": "finance"},
    Tagging="department=finance&processed=false",
)
for name in ["reports/2026/q1.csv", "reports/2026/q2.csv", "reports/2025/q4.csv", "readme.txt"]:
    s3.put_object(Bucket=bucket, Key=prefix + name, Body=b"x" * 100)

head = s3.head_object(Bucket=bucket, Key=prefix + "reports/2026/q3.csv")
print(head["ContentType"], head["Metadata"])
tags = s3.get_object_tagging(Bucket=bucket, Key=prefix + "reports/2026/q3.csv")["TagSet"]
print(sorted((t["Key"], t["Value"]) for t in tags))

text/csv {'owner-team': 'finance'}
[('department', 'finance'), ('processed', 'false')]

An object can have up to 40 tags. Keys and values are case-sensitive, a key can be up to 128 characters and a value up to 256. Metadata is fixed when the object is written; tags can be changed later.

Change tags

put_object_tagging replaces the whole tag set; it can't add one tag to the existing set. To change one tag, read the set, change it and write it back:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/reports/2026/q3.csv"

tags = {t["Key"]: t["Value"] for t in s3.get_object_tagging(Bucket=bucket, Key=key)["TagSet"]}
tags["processed"] = "true"
s3.put_object_tagging(
    Bucket=bucket, Key=key,
    Tagging={"TagSet": [{"Key": k, "Value": v} for k, v in tags.items()]},
)

obj = s3.get_object(Bucket=bucket, Key=key)
print("tag count on GET:", obj["TagCount"])
print(sorted(s3.get_object_tagging(Bucket=bucket, Key=key)["TagSet"], key=lambda t: t["Key"]))

tag count on GET: 2
[{'Key': 'department', 'Value': 'finance'}, {'Key': 'processed', 'Value': 'true'}]

If the VAST Catalog is enabled, tags are indexed with the rest of the object's metadata, so you can find objects by tag without listing the bucket: see Find objects by S3 tags and metadata. Tags are also copied by replication.

List by prefix and delimiter

An S3 bucket is flat, but a delimiter groups keys into "folders". With Delimiter="/", keys under the next / come back as common prefixes instead of objects:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

resp = s3.list_objects_v2(Bucket=bucket, Prefix="docs-s3-objects/", Delimiter="/")
print("objects:", [o["Key"] for o in resp.get("Contents", [])])
print("prefixes:", [p["Prefix"] for p in resp.get("CommonPrefixes", [])])

resp = s3.list_objects_v2(Bucket=bucket, Prefix="docs-s3-objects/reports/2026/")
print("2026:", [o["Key"].rsplit("/", 1)[-1] for o in resp.get("Contents", [])])

objects: ['docs-s3-objects/readme.txt']
prefixes: ['docs-s3-objects/reports/']
2026: ['q1.csv', 'q2.csv', 'q3.csv']

Each listing call returns at most 1,000 keys. For more, use the paginator as in the quickstart, or pass the NextContinuationToken back as ContinuationToken.

Read part of an object

A Range header reads only some bytes. That's how readers of Parquet and similar formats fetch a footer or one column chunk without downloading the file:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/reports/2026/q3.csv"

first = s3.get_object(Bucket=bucket, Key=key, Range="bytes=0-11")
print(first["ContentRange"], first["Body"].read())

last = s3.get_object(Bucket=bucket, Key=key, Range="bytes=-9")
print(last["ContentRange"], last["Body"].read())

bytes 0-11/31 b'region,total'
bytes 22-30/31 b'south,20\n'

Copy and rename

copy_object copies within the cluster, without the data passing through your client. By default the copy keeps the source's tags; TaggingDirective="REPLACE" sets new ones.

In S3 a rename is a copy followed by a delete. VAST S3 can also rename in one request: a CopyObject with the header x-amz-copy-disposition: replace moves the object, with its metadata and ACL, to the new key. With that header you can't replace the metadata or set ACL grants in the same request. boto3 has no parameter for the header, so this example adds it with an event hook on a client used only for renames:

import os

import boto3
from botocore.config import Config

def client():
    return boto3.client(
        "s3",
        endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
        region_name="us-east-1",
        config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
    )

s3 = client()
bucket = os.environ["DATASTORE_BUCKET"]
prefix = "docs-s3-objects/"

# An ordinary copy, with new tags.
s3.copy_object(
    Bucket=bucket, Key=prefix + "archive/q3.csv",
    CopySource={"Bucket": bucket, "Key": prefix + "reports/2026/q3.csv"},
    TaggingDirective="REPLACE", Tagging="department=finance&archived=true",
)

# A rename: the same call, with the VAST rename header.
renamer = client()
def add_rename_header(params, **kwargs):
    params["headers"]["x-amz-copy-disposition"] = "replace"
renamer.meta.events.register("before-call.s3.CopyObject", add_rename_header)
renamer.copy_object(
    Bucket=bucket, Key=prefix + "readme-renamed.txt",
    CopySource={"Bucket": bucket, "Key": prefix + "readme.txt"},
)

keys = [o["Key"][len(prefix):] for o in s3.list_objects_v2(Bucket=bucket, Prefix=prefix)["Contents"]]
print(sorted(keys))

['archive/q3.csv', 'readme-renamed.txt', 'reports/2025/q4.csv', 'reports/2026/q1.csv', 'reports/2026/q2.csv', 'reports/2026/q3.csv']

readme.txt is gone and readme-renamed.txt is there. Renaming a "directory" works the same way with a key that ends in /; see Renaming Objects for how keys with slashes are matched.

Conditional writes and reads

Conditional requests let several writers share a bucket safely:

IfNoneMatch="*" on put_object (or on completing a multipart upload) writes only if no object with that key exists yet. Use it for "create once" keys such as locks or idempotent outputs.
IfMatch=<ETag> on get_object or head_object reads only if the object hasn't changed since you saw that ETag.
IfMatch on delete_object deletes only if the object still exists ("*") or still has that ETag.

A failed condition returns 412 Precondition Failed:

import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/locks/job-42"
s3.delete_object(Bucket=bucket, Key=key)  # start clean if an earlier run stopped here

first = s3.put_object(Bucket=bucket, Key=key, Body=b"worker-a", IfNoneMatch="*")
print("first writer: created")
try:
    s3.put_object(Bucket=bucket, Key=key, Body=b"worker-b", IfNoneMatch="*")
except ClientError as e:
    print("second writer:", e.response["ResponseMetadata"]["HTTPStatusCode"], e.response["Error"]["Code"])

etag = first["ETag"]
print("read if unchanged:", s3.get_object(Bucket=bucket, Key=key, IfMatch=etag)["Body"].read())
s3.put_object(Bucket=bucket, Key=key, Body=b"worker-a, updated")
try:
    s3.get_object(Bucket=bucket, Key=key, IfMatch=etag)
except ClientError as e:
    print("read after a change:", e.response["ResponseMetadata"]["HTTPStatusCode"], e.response["Error"]["Code"])

first writer: created
second writer: 412 PreconditionFailed
read if unchanged: b'worker-a'
read after a change: 412 PreconditionFailed

IfNoneMatch="*" needs a bucket without versioning. A bucket policy can require conditional deletes (an If-Match on every delete); ask your platform team if yours does.

Share an object with a presigned URL

A presigned URL carries your signature in its query string, so someone without keys can perform that one request until the URL expires. VAST S3 supports presigned GET, PUT, DELETE and POST. Objects uploaded through a presigned URL don't expire with it. This example creates a URL to upload and one to download, and uses them with only the Python standard library, as a client without credentials would:

import os
import urllib.request

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/shared/upload.bin"

put_url = s3.generate_presigned_url("put_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
req = urllib.request.Request(put_url, data=b"uploaded without keys", method="PUT")
with urllib.request.urlopen(req) as resp:
    print("PUT", resp.status)

get_url = s3.generate_presigned_url("get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
print("signature in the URL:", "X-Amz-Signature=" in get_url)
with urllib.request.urlopen(get_url) as resp:
    print("GET", resp.status, resp.read())

PUT 200
signature in the URL: True
GET 200 b'uploaded without keys'

The URL works with the permissions of the key that signed it, and it stops working when it expires. If you signed it with temporary credentials, it stops when they expire, if that's sooner. For browser uploads, generate_presigned_post creates a presigned POST form; on VAST, the object name must be ASCII and the POST policy at most 4,800 bytes.

Clean up

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [
    {"Key": obj["Key"]}
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix="docs-s3-objects/")
    for obj in page.get("Contents", [])
]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")

deleted 8 objects