Versioning and object lock

Applies to VAST Cluster 5.5.

With versioning, a bucket keeps every earlier version of an object when it's overwritten or deleted. With object locking, versions can also be protected from deletion, for a retention period or under a legal hold. This guide shows how to work with versions (list, read, undelete, restore, delete for good) and with legal holds and retention, and what's different from AWS. It uses the prefix docs-versioning-and-object-lock/ in two buckets your platform team has set up, and the last block deletes every version it created.

Each code block is complete on its own. Below each block is the output you should see.

Before you start

Versioning and object locking can't be turned off once they're on, so these examples need buckets set aside for them. Admin task: send to your platform team: ask for

a bucket with versioning enabled, for the versioning examples (Object Versioning), and
a bucket with object locking enabled and no default retention period, for the object-lock examples. It must be on a view with no protocol other than S3 (S3 Object Locking).

Your key needs permission to list, read and delete object versions, and to set and read legal holds and retention. The bucket owner can do all of this unless a policy says otherwise.

Set these variables in addition to the ones from the quickstart:

export DATASTORE_VERSIONED_BUCKET="<bucket with versioning>"
export DATASTORE_LOCK_BUCKET="<bucket with object locking>"

What's different from AWS

Versioning can't be suspended. Once it's enabled on a bucket, it stays enabled. PutBucketVersioning with Suspended is refused with 501 NotImplemented.
No MFA Delete.
Versioned objects are S3-only for writing. If the bucket's view also has NFS or SMB, those protocols can read versioned objects but not change them.
No conditional writes in versioned buckets. IfNoneMatch="*" isn't supported there (501 NotImplemented). Conditional deletes do work.
Object locking is for S3-only buckets. It can't be enabled on a view that also has NFS or SMB, and it can't be disabled.
Keys must differ by more than a trailing slash in a versioned bucket: the cluster reserves the trailing slash for its version directories.

Check the bucket

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_VERSIONED_BUCKET"]

print("versioning:", s3.get_bucket_versioning(Bucket=bucket).get("Status"))

versioning: Enabled

Versions, delete markers, retention and legal holds.

Write and list versions

Every overwrite adds a version. list_object_versions returns them, newest first, with IsLatest on the current one:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_VERSIONED_BUCKET"]
key = "docs-versioning-and-object-lock/config.json"

for n in range(1, 4):
    s3.put_object(Bucket=bucket, Key=key, Body=f'{{"revision": {n}}}'.encode())

versions = s3.list_object_versions(Bucket=bucket, Prefix=key)["Versions"]
print(len(versions), "versions")
for v in versions:
    body = s3.get_object(Bucket=bucket, Key=key, VersionId=v["VersionId"])["Body"].read()
    print("latest" if v["IsLatest"] else "older ", body)

3 versions
latest b'{"revision": 3}'
older  b'{"revision": 2}'
older  b'{"revision": 1}'

A plain get_object returns the current version; with VersionId, any version. head_object takes VersionId the same way.

Delete and undelete

Deleting an object in a versioned bucket doesn't remove any data. It adds a delete marker, which becomes the current version, so the object looks deleted. Deleting the marker (by its version ID) brings the object back:

import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_VERSIONED_BUCKET"]
key = "docs-versioning-and-object-lock/config.json"

s3.delete_object(Bucket=bucket, Key=key)
try:
    s3.get_object(Bucket=bucket, Key=key)
except ClientError as e:
    print("after delete:", e.response["Error"]["Code"])

listing = s3.list_object_versions(Bucket=bucket, Prefix=key)
markers = listing.get("DeleteMarkers", [])
print(len(listing["Versions"]), "versions and", len(markers), "delete marker")

s3.delete_object(Bucket=bucket, Key=key, VersionId=markers[0]["VersionId"])
print("after removing the marker:", s3.get_object(Bucket=bucket, Key=key)["Body"].read())

after delete: NoSuchKey
3 versions and 1 delete marker
after removing the marker: b'{"revision": 3}'

Restore an earlier version

To make an earlier version current again, copy it onto the same key. The copy becomes a new version, and all the others are kept:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_VERSIONED_BUCKET"]
key = "docs-versioning-and-object-lock/config.json"

oldest = s3.list_object_versions(Bucket=bucket, Prefix=key)["Versions"][-1]
s3.copy_object(Bucket=bucket, Key=key, CopySource={"Bucket": bucket, "Key": key, "VersionId": oldest["VersionId"]})

print("current:", s3.get_object(Bucket=bucket, Key=key)["Body"].read())
print(len(s3.list_object_versions(Bucket=bucket, Prefix=key)["Versions"]), "versions")

current: b'{"revision": 1}'
4 versions

The other way to restore is to delete the current version by its version ID: the previous version becomes current.

Object locking

In a bucket with object locking, each object version can have:

A retention period with a mode. In governance mode, nobody can delete or overwrite the version, shorten its retention or change its mode, unless they have the s3:BypassGovernanceRetention permission and send x-amz-bypass-governance-retention: true (BypassGovernanceRetention=True in boto3). You can extend a retention period at any time.
A legal hold, which protects the version with no end date until someone with s3:PutObjectLegalHold removes it. It's independent of retention.
Nothing. If the bucket has a default retention period, each new version gets it automatically.

Protection applies to versions. Deleting a locked object without a version ID still works: it adds a delete marker, and the protected version stays.

Design for governance mode and legal holds. In compliance mode, no one can delete the version or shorten its retention; ask your platform team whether compliance mode is available on your cluster before you depend on it.

Use a legal hold

Put a legal hold on a version, see that it can't be deleted, then release it and delete the version:

import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_LOCK_BUCKET"]
key = "docs-versioning-and-object-lock/evidence.txt"

lock = s3.get_object_lock_configuration(Bucket=bucket)["ObjectLockConfiguration"]
print("object lock:", lock["ObjectLockEnabled"], "| default retention:", "Rule" in lock)

version = s3.put_object(Bucket=bucket, Key=key, Body=b"keep me")["VersionId"]
s3.put_object_legal_hold(Bucket=bucket, Key=key, VersionId=version, LegalHold={"Status": "ON"})
print("legal hold:", s3.get_object_legal_hold(Bucket=bucket, Key=key, VersionId=version)["LegalHold"]["Status"])

try:
    s3.delete_object(Bucket=bucket, Key=key, VersionId=version)
except ClientError as e:
    print("delete while on hold:", e.response["Error"]["Code"])

s3.put_object_legal_hold(Bucket=bucket, Key=key, VersionId=version, LegalHold={"Status": "OFF"})
s3.delete_object(Bucket=bucket, Key=key, VersionId=version)
print("deleted after release")

object lock: Enabled | default retention: False
legal hold: ON
delete while on hold: AccessDenied
deleted after release

Set a retention period

Retention is set per version with put_object_retention, or per upload with ObjectLockMode and ObjectLockRetainUntilDate on put_object. This fragment isn't run here, because a version under retention can't be deleted until the date passes unless you have the bypass permission:

from datetime import datetime, timedelta, timezone

until = datetime.now(timezone.utc) + timedelta(days=1)
s3.put_object_retention(
    Bucket=bucket, Key=key, VersionId=version,
    Retention={"Mode": "GOVERNANCE", "RetainUntilDate": until},
)
print(s3.get_object_retention(Bucket=bucket, Key=key, VersionId=version)["Retention"])

# Removing it early: only with s3:BypassGovernanceRetention.
s3.delete_object(Bucket=bucket, Key=key, VersionId=version, BypassGovernanceRetention=True)

get_object and head_object return the lock mode, retain-until date and legal-hold status (ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus) if your key may read them.

Clean up

Delete every version and delete marker under the prefix, in both buckets. Deleting by version ID is permanent:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
prefix = "docs-versioning-and-object-lock/"

buckets = {"versioned bucket": os.environ["DATASTORE_VERSIONED_BUCKET"], "lock bucket": os.environ["DATASTORE_LOCK_BUCKET"]}
for label, bucket in buckets.items():
    listing = s3.list_object_versions(Bucket=bucket, Prefix=prefix)
    entries = listing.get("Versions", []) + listing.get("DeleteMarkers", [])
    for v in entries:
        s3.delete_object(Bucket=bucket, Key=v["Key"], VersionId=v["VersionId"])
    print(f"{label}: {len(entries)} versions and markers deleted")

versioned bucket: 4 versions and markers deleted
lock bucket: 0 versions and markers deleted