Python clients

Applies to VAST Cluster 5.5.

Any Python library that speaks S3 works with VAST S3 if you give it the cluster's endpoint and your key pair. This page shows the settings for the three most used: boto3 (the AWS SDK), s3fs (the fsspec filesystem that pandas and Dask use for s3:// paths) and PyArrow's S3FileSystem (for Parquet and datasets). The examples use the prefix docs-python/, and the last block removes it.

Each code block is complete on its own. Below each block is the output you should see.

Before you start

You need the bucket, key pair and environment variables from the quickstart, and the libraries:

pip install "boto3>=1.36" s3fs pyarrow

boto3

These are the settings that matter for VAST:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
        max_pool_connections=16,
        retries={"max_attempts": 5, "mode": "standard"},
    ),
)
bucket = os.environ["DATASTORE_BUCKET"]

s3.put_object(Bucket=bucket, Key="docs-python/boto3.txt", Body=b"from boto3\n")
print(s3.meta.config.signature_version, s3.meta.config.s3)
print(s3.get_object(Bucket=bucket, Key="docs-python/boto3.txt")["Body"].read())

s3v4 {'addressing_style': 'path'}
b'from boto3\n'

The higher-level boto3.resource("s3", ...) takes the same arguments.

Recent boto3 and botocore releases can also take the endpoint from the environment variable AWS_ENDPOINT_URL_S3 or from endpoint_url in ~/.aws/config, which is handy for code you don't want to change. Passing endpoint_url explicitly works with every version.

HTTPS and certificates

The cluster comes with a self-signed certificate for S3, which your platform team may have replaced with one signed by your CA. If your system doesn't trust the certificate, point the client at a file that contains it (or its CA) rather than turning verification off:

s3 = boto3.client("s3", endpoint_url="https://s3.example.com", verify="/path/to/vast-s3-ca.pem",
                  region_name="us-east-1", config=Config(signature_version="s3v4", s3={"addressing_style": "path"}))

Setting AWS_CA_BUNDLE in the environment does the same for every boto3 client. If the certificate's host name doesn't match the name you connect to, use a name that's in the certificate. verify=False turns checking off; use it only on a test system.

s3fs (pandas, Dask and fsspec)

s3fs gives s3:// paths a file-like interface. Give it the endpoint and path-style addressing; it reads the keys from the same environment variables as boto3:

import os

import s3fs

fs = s3fs.S3FileSystem(
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    config_kwargs={"s3": {"addressing_style": "path"}, "signature_version": "s3v4"},
    client_kwargs={"region_name": "us-east-1"},
)
bucket = os.environ["DATASTORE_BUCKET"]

with fs.open(f"{bucket}/docs-python/s3fs/hello.txt", "w") as f:
    f.write("from s3fs\n")
print(fs.ls(f"{bucket}/docs-python/s3fs/"))
print(fs.cat(f"{bucket}/docs-python/s3fs/hello.txt"))

['dst-data/docs-python/s3fs/hello.txt']
b'from s3fs\n'

pandas uses s3fs for s3:// URLs, so pass the same options as storage_options:

import pandas as pd

df = pd.read_csv(
    f"s3://{bucket}/path/to/file.csv",
    storage_options={
        "endpoint_url": os.environ["AWS_S3_ENDPOINT_URL"],
        "config_kwargs": {"s3": {"addressing_style": "path"}},
    },
)

PyArrow

pyarrow.fs.S3FileSystem is Arrow's own S3 client, used by pyarrow.parquet and pyarrow.dataset. It takes the endpoint as a host name (and optional port) plus a scheme, and uses path-style addressing when you give it an endpoint. Write a Parquet file and read it back:

import os
from urllib.parse import urlparse

import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow import fs

endpoint = urlparse(os.environ["AWS_S3_ENDPOINT_URL"])
s3 = fs.S3FileSystem(
    access_key=os.environ["AWS_ACCESS_KEY_ID"],
    secret_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    endpoint_override=endpoint.netloc,
    scheme=endpoint.scheme,
    region="us-east-1",
)
bucket = os.environ["DATASTORE_BUCKET"]

table = pa.table({"id": [1, 2, 3], "city": ["Paris", "Lima", "Osaka"]})
for part in range(2):
    pq.write_table(table, f"{bucket}/docs-python/parquet/part-{part}.parquet", filesystem=s3)

dataset = pq.ParquetDataset(f"{bucket}/docs-python/parquet/", filesystem=s3)
result = dataset.read()
print(result.num_rows, "rows from", len(dataset.files), "files")
print(result.schema)

6 rows from 2 files
id: int64
city: string

Readers fetch Parquet footers and column chunks with ranged GETs, so reading a few columns of a large file reads only those bytes. To load Parquet files into a VAST DataBase table without reading them through your client, use the cluster-side import: Import Parquet files.

Clean up

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [{"Key": o["Key"]} for o in s3.list_objects_v2(Bucket=bucket, Prefix="docs-python/").get("Contents", [])]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")

deleted 4 objects

See also

• S3 objects and large objects with boto3.
• Spark and Hadoop and the AWS CLI and other tools.
• KB: Using Boto3 with VAST Cluster S3 and Boto3 Client.