---
title: "Python clients"
slug: "dev-datastore-integrations-python"
updated: 2026-10-30T18:18:46Z
published: 2026-10-30T18:18:46Z
canonical: "docs.vastdata.com/dev-datastore-integrations-python"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Python clients

*Applies to VAST Cluster 5.5.*

Any Python library that speaks S3 works with VAST S3 if you give it the cluster's endpoint and your key pair. This page shows the settings for the three most used: boto3 (the AWS SDK), s3fs (the fsspec filesystem that pandas and Dask use for `s3://` paths) and PyArrow's `S3FileSystem` (for Parquet and datasets). The examples use the prefix `docs-python/`, and the last block removes it.

Each code block is complete on its own. Below each block is the output you should see.

## Before you start

You need the bucket, key pair and environment variables from the [quickstart](/develop/docs/dev-datastore-quickstart#before-you-start), and the libraries:

```
pip install "boto3>=1.36" s3fs pyarrow
```

## boto3

These are the settings that matter for VAST:

| Setting | Value | Why |
| --- | --- | --- |
| `endpoint_url` | `http://<name or VIP>` or `https://...` | The cluster's S3 endpoint. Without it boto3 goes to AWS. No port: the cluster listens on 80 and 443. |
| `region_name` | Any string, for example `us-east-1` | Signature Version 4 needs a region; the cluster accepts any. |
| `signature_version` | `s3v4` | The recommended signature. Version 2 (`s3`) also works. |
| `s3={"addressing_style": "path"}` | Path-style URLs (`endpoint/bucket/key`) | The addressing style to use with VAST. Bucket-in-hostname (virtual-hosted) URLs aren't documented. |
| `max_pool_connections` | At least your number of threads | See [performance](/develop/docs/dev-datastore-guides-performance). |

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
        max_pool_connections=16,
        retries={"max_attempts": 5, "mode": "standard"},
    ),
)
bucket = os.environ["DATASTORE_BUCKET"]

s3.put_object(Bucket=bucket, Key="docs-python/boto3.txt", Body=b"from boto3\n")
print(s3.meta.config.signature_version, s3.meta.config.s3)
print(s3.get_object(Bucket=bucket, Key="docs-python/boto3.txt")["Body"].read())
```

```
s3v4 {'addressing_style': 'path'}
b'from boto3\n'
```

The higher-level `boto3.resource("s3", ...)` takes the same arguments.

Recent boto3 and botocore releases can also take the endpoint from the environment variable `AWS_ENDPOINT_URL_S3` or from `endpoint_url` in `~/.aws/config`, which is handy for code you don't want to change. Passing `endpoint_url` explicitly works with every version.

### HTTPS and certificates

The cluster comes with a self-signed certificate for S3, which your platform team may have replaced with one signed by your CA. If your system doesn't trust the certificate, point the client at a file that contains it (or its CA) rather than turning verification off:

```
s3 = boto3.client("s3", endpoint_url="https://s3.example.com", verify="/path/to/vast-s3-ca.pem",
                  region_name="us-east-1", config=Config(signature_version="s3v4", s3={"addressing_style": "path"}))
```

Setting `AWS_CA_BUNDLE` in the environment does the same for every boto3 client. If the certificate's host name doesn't match the name you connect to, use a name that's in the certificate. `verify=False` turns checking off; use it only on a test system.

## s3fs (pandas, Dask and fsspec)

s3fs gives `s3://` paths a file-like interface. Give it the endpoint and path-style addressing; it reads the keys from the same environment variables as boto3:

```
import os

import s3fs

fs = s3fs.S3FileSystem(
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    config_kwargs={"s3": {"addressing_style": "path"}, "signature_version": "s3v4"},
    client_kwargs={"region_name": "us-east-1"},
)
bucket = os.environ["DATASTORE_BUCKET"]

with fs.open(f"{bucket}/docs-python/s3fs/hello.txt", "w") as f:
    f.write("from s3fs\n")
print(fs.ls(f"{bucket}/docs-python/s3fs/"))
print(fs.cat(f"{bucket}/docs-python/s3fs/hello.txt"))
```

```
['dst-data/docs-python/s3fs/hello.txt']
b'from s3fs\n'
```

pandas uses s3fs for `s3://` URLs, so pass the same options as `storage_options`:

```
import pandas as pd

df = pd.read_csv(
    f"s3://{bucket}/path/to/file.csv",
    storage_options={
        "endpoint_url": os.environ["AWS_S3_ENDPOINT_URL"],
        "config_kwargs": {"s3": {"addressing_style": "path"}},
    },
)
```

## PyArrow

`pyarrow.fs.S3FileSystem` is Arrow's own S3 client, used by `pyarrow.parquet` and `pyarrow.dataset`. It takes the endpoint as a host name (and optional port) plus a scheme, and uses path-style addressing when you give it an endpoint. Write a Parquet file and read it back:

```
import os
from urllib.parse import urlparse

import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow import fs

endpoint = urlparse(os.environ["AWS_S3_ENDPOINT_URL"])
s3 = fs.S3FileSystem(
    access_key=os.environ["AWS_ACCESS_KEY_ID"],
    secret_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    endpoint_override=endpoint.netloc,
    scheme=endpoint.scheme,
    region="us-east-1",
)
bucket = os.environ["DATASTORE_BUCKET"]

table = pa.table({"id": [1, 2, 3], "city": ["Paris", "Lima", "Osaka"]})
for part in range(2):
    pq.write_table(table, f"{bucket}/docs-python/parquet/part-{part}.parquet", filesystem=s3)

dataset = pq.ParquetDataset(f"{bucket}/docs-python/parquet/", filesystem=s3)
result = dataset.read()
print(result.num_rows, "rows from", len(dataset.files), "files")
print(result.schema)
```

```
6 rows from 2 files
id: int64
city: string
```

Readers fetch Parquet footers and column chunks with ranged GETs, so reading a few columns of a large file reads only those bytes. To load Parquet files into a VAST DataBase table without reading them through your client, use the cluster-side import: [Import Parquet files](/develop/docs/dev-database-guides-ingest#import-parquet-files).

## Clean up

```
import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
    region_name="us-east-1",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
bucket = os.environ["DATASTORE_BUCKET"]

keys = [{"Key": o["Key"]} for o in s3.list_objects_v2(Bucket=bucket, Prefix="docs-python/").get("Contents", [])]
if keys:
    s3.delete_objects(Bucket=bucket, Delete={"Objects": keys})
print("deleted", len(keys), "objects")
```

```
deleted 4 objects
```

## See also

- [S3 objects](/develop/docs/dev-datastore-guides-s3-objects) and [large objects](/develop/docs/dev-datastore-guides-large-objects-and-multipart) with boto3.
- [Spark and Hadoop](/develop/docs/dev-datastore-integrations-spark-and-hadoop) and [the AWS CLI and other tools](/develop/docs/dev-datastore-integrations-aws-cli-and-tools).
- KB: [Using Boto3 with VAST Cluster S3](/docs-5-5/docs/using-boto3-with-vast-cluster-s3) and [Boto3 Client](/docs-5-5/docs/boto3-client).