---
title: "Work with S3 objects"
slug: "dev-datastore-guides-s3-objects"
updated: 2026-10-28T02:15:03Z
published: 2026-10-28T02:15:03Z
canonical: "docs.vastdata.com/dev-datastore-guides-s3-objects"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Work with S3 objects

*Applies to VAST Cluster 5.5.*

This guide covers the object operations an application uses every day on VAST S3: writing objects with metadata and tags, listing by prefix and delimiter, reading part of an object, copying and renaming, conditional writes and reads, and sharing an object through a presigned URL. Everything happens under the key prefix `docs-s3-objects/`, and the last block removes it.

Each code block is complete on its own. Below each block is the output you should see.

## Before you start

You need the bucket, key pair and environment variables from the [quickstart](/develop/docs/dev-datastore-quickstart#before-you-start), and a bucket without versioning: conditional writes aren't supported in versioned buckets. Your key needs to read, write, list, delete and tag objects in the bucket.

## Write objects with metadata and tags

Write a few objects. The first carries a content type, your own metadata (sent as `x-amz-meta-` headers) and tags (sent URL-encoded in the `x-amz-tagging` header):

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
prefix = "docs-s3-objects/"

s3.put_object(
    Bucket=bucket,
    Key=prefix + "reports/2026/q3.csv",
    Body=b"region,total\nnorth,10\nsouth,20\n",
    ContentType="text/csv",
    Metadata={"owner-team": "finance"},
    Tagging="department=finance&processed=false",
)
for name in ["reports/2026/q1.csv", "reports/2026/q2.csv", "reports/2025/q4.csv", "readme.txt"]:
    s3.put_object(Bucket=bucket, Key=prefix + name, Body=b"x" * 100)

head = s3.head_object(Bucket=bucket, Key=prefix + "reports/2026/q3.csv")
print(head["ContentType"], head["Metadata"])
tags = s3.get_object_tagging(Bucket=bucket, Key=prefix + "reports/2026/q3.csv")["TagSet"]
print(sorted((t["Key"], t["Value"]) for t in tags))
```

```
text/csv {'owner-team': 'finance'}
[('department', 'finance'), ('processed', 'false')]
```

An object can have up to 40 tags. Keys and values are case-sensitive, a key can be up to 128 characters and a value up to 256. Metadata is fixed when the object is written; tags can be changed later.

## Change tags

`put_object_tagging` replaces the whole tag set; it can't add one tag to the existing set. To change one tag, read the set, change it and write it back:

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/reports/2026/q3.csv"

tags = {t["Key"]: t["Value"] for t in s3.get_object_tagging(Bucket=bucket, Key=key)["TagSet"]}
tags["processed"] = "true"
s3.put_object_tagging(
    Bucket=bucket, Key=key,
    Tagging={"TagSet": [{"Key": k, "Value": v} for k, v in tags.items()]},
)

obj = s3.get_object(Bucket=bucket, Key=key)
print("tag count on GET:", obj["TagCount"])
print(sorted(s3.get_object_tagging(Bucket=bucket, Key=key)["TagSet"], key=lambda t: t["Key"]))
```

```
tag count on GET: 2
[{'Key': 'department', 'Value': 'finance'}, {'Key': 'processed', 'Value': 'true'}]
```

If the VAST Catalog is enabled, tags are indexed with the rest of the object's metadata, so you can find objects by tag without listing the bucket: see [Find objects by S3 tags and metadata](/develop/docs/dev-catalog-guides-tags-and-metadata). Tags are also copied by replication.

## List by prefix and delimiter

An S3 bucket is flat, but a delimiter groups keys into "folders". With `Delimiter="/"`, keys under the next `/` come back as common prefixes instead of objects:

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

resp = s3.list_objects_v2(Bucket=bucket, Prefix="docs-s3-objects/", Delimiter="/")
print("objects:", [o["Key"] for o in resp.get("Contents", [])])
print("prefixes:", [p["Prefix"] for p in resp.get("CommonPrefixes", [])])

resp = s3.list_objects_v2(Bucket=bucket, Prefix="docs-s3-objects/reports/2026/")
print("2026:", [o["Key"].rsplit("/", 1)[-1] for o in resp.get("Contents", [])])
```

```
objects: ['docs-s3-objects/readme.txt']
prefixes: ['docs-s3-objects/reports/']
2026: ['q1.csv', 'q2.csv', 'q3.csv']
```

Each listing call returns at most 1,000 keys. For more, use the paginator as in the [quickstart](/develop/docs/dev-datastore-quickstart#list-objects), or pass the `NextContinuationToken` back as `ContinuationToken`.

## Read part of an object

A `Range` header reads only some bytes. That's how readers of Parquet and similar formats fetch a footer or one column chunk without downloading the file:

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/reports/2026/q3.csv"

first = s3.get_object(Bucket=bucket, Key=key, Range="bytes=0-11")
print(first["ContentRange"], first["Body"].read())

last = s3.get_object(Bucket=bucket, Key=key, Range="bytes=-9")
print(last["ContentRange"], last["Body"].read())
```

```
bytes 0-11/31 b'region,total'
bytes 22-30/31 b'south,20\n'
```

## Copy and rename

`copy_object` copies within the cluster, without the data passing through your client. By default the copy keeps the source's tags; `TaggingDirective="REPLACE"` sets new ones.

In S3 a rename is a copy followed by a delete. VAST S3 can also rename in one request: a `CopyObject` with the header `x-amz-copy-disposition: replace` moves the object, with its metadata and ACL, to the new key. With that header you can't replace the metadata or set ACL grants in the same request. boto3 has no parameter for the header, so this example adds it with an event hook on a client used only for renames:

```
import os

import boto3
from botocore.config import Config

def client():
    return boto3.client(
        "s3",
        endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
        region_name="us-east-1",
        config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
    )

s3 = client()
bucket = os.environ["DATASTORE_BUCKET"]
prefix = "docs-s3-objects/"

# An ordinary copy, with new tags.
s3.copy_object(
    Bucket=bucket, Key=prefix + "archive/q3.csv",
    CopySource={"Bucket": bucket, "Key": prefix + "reports/2026/q3.csv"},
    TaggingDirective="REPLACE", Tagging="department=finance&archived=true",
)

# A rename: the same call, with the VAST rename header.
renamer = client()
def add_rename_header(params, **kwargs):
    params["headers"]["x-amz-copy-disposition"] = "replace"
renamer.meta.events.register("before-call.s3.CopyObject", add_rename_header)
renamer.copy_object(
    Bucket=bucket, Key=prefix + "readme-renamed.txt",
    CopySource={"Bucket": bucket, "Key": prefix + "readme.txt"},
)

keys = [o["Key"][len(prefix):] for o in s3.list_objects_v2(Bucket=bucket, Prefix=prefix)["Contents"]]
print(sorted(keys))
```

```
['archive/q3.csv', 'readme-renamed.txt', 'reports/2025/q4.csv', 'reports/2026/q1.csv', 'reports/2026/q2.csv', 'reports/2026/q3.csv']
```

`readme.txt` is gone and `readme-renamed.txt` is there. Renaming a "directory" works the same way with a key that ends in `/`; see [Renaming Objects](/docs-5-5/docs/renaming-objects) for how keys with slashes are matched.

## Conditional writes and reads

Conditional requests let several writers share a bucket safely:

- `IfNoneMatch="*"` on `put_object` (or on completing a multipart upload) writes only if no object with that key exists yet. Use it for "create once" keys such as locks or idempotent outputs.
- `IfMatch=<ETag>` on `get_object` or `head_object` reads only if the object hasn't changed since you saw that ETag.
- `IfMatch` on `delete_object` deletes only if the object still exists (`"*"`) or still has that ETag.

A failed condition returns `412 Precondition Failed`:

```
import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/locks/job-42"
s3.delete_object(Bucket=bucket, Key=key)  # start clean if an earlier run stopped here

first = s3.put_object(Bucket=bucket, Key=key, Body=b"worker-a", IfNoneMatch="*")
print("first writer: created")
try:
    s3.put_object(Bucket=bucket, Key=key, Body=b"worker-b", IfNoneMatch="*")
except ClientError as e:
    print("second writer:", e.response["ResponseMetadata"]["HTTPStatusCode"], e.response["Error"]["Code"])

etag = first["ETag"]
print("read if unchanged:", s3.get_object(Bucket=bucket, Key=key, IfMatch=etag)["Body"].read())
s3.put_object(Bucket=bucket, Key=key, Body=b"worker-a, updated")
try:
    s3.get_object(Bucket=bucket, Key=key, IfMatch=etag)
except ClientError as e:
    print("read after a change:", e.response["ResponseMetadata"]["HTTPStatusCode"], e.response["Error"]["Code"])
```

```
first writer: created
second writer: 412 PreconditionFailed
read if unchanged: b'worker-a'
read after a change: 412 PreconditionFailed
```

`IfNoneMatch="*"` needs a bucket without versioning. A bucket policy can require conditional deletes (an `If-Match` on every delete); ask your platform team if yours does.

## Share an object with a presigned URL

A presigned URL carries your signature in its query string, so someone without keys can perform that one request until the URL expires. VAST S3 supports presigned GET, PUT, DELETE and POST. Objects uploaded through a presigned URL don't expire with it. This example creates a URL to upload and one to download, and uses them with only the Python standard library, as a client without credentials would:

```
import os
import urllib.request

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]
key = "docs-s3-objects/shared/upload.bin"

put_url = s3.generate_presigned_url("put_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
req = urllib.request.Request(put_url, data=b"uploaded without keys", method="PUT")
with urllib.request.urlopen(req) as resp:
    print("PUT", resp.status)

get_url = s3.generate_presigned_url("get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
print("signature in the URL:", "X-Amz-Signature=" in get_url)
with urllib.request.urlopen(get_url) as resp:
    print("GET", resp.status, resp.read())
```

```
PUT 200
signature in the URL: True
GET 200 b'uploaded without keys'
```

The URL works with the permissions of the key that signed it, and it stops working when it expires. If you signed it with [temporary credentials](/develop/docs/dev-datastore-operations-access#temporary-credentials), it stops when they expire, if that's sooner. For browser uploads, `generate_presigned_post` creates a presigned POST form; on VAST, the object name must be ASCII and the POST policy at most 4,800 bytes.

## Clean up

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [
    {"Key": obj["Key"]}
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix="docs-s3-objects/")
    for obj in page.get("Contents", [])
]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")
```

```
deleted 8 objects
```

## See also

- [Large objects and multipart uploads](/develop/docs/dev-datastore-guides-large-objects-and-multipart).
- [S3 compatibility](/develop/docs/dev-datastore-reference-s3-compatibility): every supported request and header.
- KB: [S3 Tagging](/docs-5-5/docs/s3-tagging), [Conditional S3 Requests](/docs-5-5/docs/conditional-s3-requests), [Sharing S3 Access via Presigned URLs](/docs-5-5/docs/sharing-s3-access-via-presigned-urls) and [Renaming Objects](/docs-5-5/docs/renaming-objects).