Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

AWS CLI, s3cmd and rclone

Prev Next

Applies to VAST Cluster 5.5.

Command-line tools are the quickest way to move data in and out of VAST S3 and to check what's in a bucket. This page gives the configuration each tool needs for a VAST endpoint. The commands themselves are the tools' own; see their documentation for everything else.

The configurations and commands on this page were run against VAST Cluster 5.5 with AWS CLI 1.46.1 and 2.27.0, s3cmd 2.4.0 and rclone 1.68.2. The outputs below are from those runs, with the bucket name replaced by <bucket>.

AWS CLI

Create a profile for VAST in ~/.aws/config, with the endpoint and path-style addressing:

# ~/.aws/config (not captured)
[profile vast]
region = us-east-1
endpoint_url = http://s3.example.com
s3 =
    addressing_style = path
    max_concurrent_requests = 16
    multipart_threshold = 64MB
    multipart_chunksize = 16MB

and the keys in ~/.aws/credentials:

# ~/.aws/credentials (not captured)
[vast]
aws_access_key_id = <access key>
aws_secret_access_key = <secret key>

If you'd rather keep the keys in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, select the profile with AWS_PROFILE=vast instead of --profile vast. When you name a profile with --profile, the CLI ignores the environment variables and fails with Unable to locate credentials if the profile has no keys.

Then:

# not captured
aws --profile vast s3 ls s3://<bucket>/
aws --profile vast s3 cp ./data.parquet s3://<bucket>/docs-cli/data.parquet
aws --profile vast s3 sync ./results/ s3://<bucket>/docs-cli/results/
aws --profile vast s3 presign s3://<bucket>/docs-cli/data.parquet --expires-in 300
aws --profile vast s3 rm --recursive s3://<bucket>/docs-cli/

With AWS CLI 2.27, the copy, sync, presign and remove print:

upload: ./data.parquet to s3://<bucket>/docs-cli/data.parquet
upload: results/a.txt to s3://<bucket>/docs-cli/results/a.txt
upload: results/b.txt to s3://<bucket>/docs-cli/results/b.txt
http://s3.example.com/<bucket>/docs-cli/data.parquet?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<credential>&X-Amz-Date=20261012T145621Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature>
delete: s3://<bucket>/docs-cli/data.parquet
delete: s3://<bucket>/docs-cli/results/a.txt
delete: s3://<bucket>/docs-cli/results/b.txt

The presigned URL downloads the object without credentials until it expires. AWS CLI version 1 signs presigned URLs with Signature Version 2 query parameters (AWSAccessKeyId, Signature, Expires); those work too.

Older AWS CLI versions don't read endpoint_url from the config file; pass --endpoint-url http://s3.example.com on each command instead (that works on every version). For HTTPS with a certificate your system doesn't trust, set ca_bundle = /path/to/vast-s3-ca.pem in the profile (or --ca-bundle), and connect with a name the certificate lists: with an address that isn't in the certificate, the CLI fails with hostname '<address>' doesn't match ... even with the CA bundle. The aws s3api commands accept the same profile, for example aws --profile vast s3api get-object-tagging --bucket <bucket> --key <key>.

The s3 settings in the profile control the CLI's transfers: how many requests run at once, and the size above which, and in which parts, it uploads multipart. With the settings above, an 80 MB file goes up in five parts, and its ETag ends in -5. What the settings do on VAST is explained in performance.

s3cmd

A minimal configuration file, for example ~/vast.s3cfg:

# ~/vast.s3cfg (not captured)
[default]
access_key = <access key>
secret_key = <secret key>
host_base = s3.example.com
host_bucket = s3.example.com
use_https = False
  • Set host_base and host_bucket to the same name or VIP, without a port. The port isn't configurable, and adding the default port can break Signature Version 4.
  • For HTTPS, set use_https = True. If your system doesn't trust the cluster's certificate, set ca_certs_file to it; if the name doesn't match the certificate, set check_ssl_certificate = False only on a test system.
  • s3cmd uses Signature Version 4 by default; signature_v2 = True switches to version 2.
# not captured
s3cmd -c ~/vast.s3cfg ls s3://<bucket>/
s3cmd -c ~/vast.s3cfg put --multipart-chunk-size-mb=64 big.bin s3://<bucket>/docs-cli/
s3cmd -c ~/vast.s3cfg info s3://<bucket>/docs-cli/big.bin
s3cmd -c ~/vast.s3cfg del --recursive s3://<bucket>/docs-cli/

For an 80 MB big.bin, the upload and the start of info print:

upload: 'big.bin' -> 's3://<bucket>/docs-cli/big.bin' (80000000 bytes in 2.0 seconds, 38.21 MB/s) [1 of 1]
s3://<bucket>/docs-cli/big.bin (object):
   File size: 80000000

s3cmd uploads files larger than its chunk size (15 MB by default) as multipart uploads (an 80 MB file goes up in six parts). KB: S3cmd.

rclone

rclone treats VAST S3 as a generic S3 provider. Add a remote to ~/.config/rclone/rclone.conf:

# ~/.config/rclone/rclone.conf (not captured)
[vast]
type = s3
provider = Other
endpoint = http://s3.example.com
region = us-east-1
force_path_style = true
acl = private
env_auth = true

With env_auth = true, rclone takes the keys from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Then:

# not captured
rclone lsf vast:<bucket>/
rclone copy ./results vast:<bucket>/docs-cli/results --transfers 16 --s3-chunk-size 16M
rclone check ./results vast:<bucket>/docs-cli/results
rclone purge vast:<bucket>/docs-cli

copy and purge print nothing when they succeed. check prints:

2026/10/12 14:56:49 NOTICE: S3 bucket <bucket> path docs-cli/results: 0 differences found
2026/10/12 14:56:49 NOTICE: S3 bucket <bucket> path docs-cli/results: 2 matching files

--transfers sets how many files move at once and --s3-upload-concurrency how many parts of one file. rclone checks files by MD5. For files it uploaded itself, multipart ones included, that works, because rclone records the MD5 in the object's metadata. Objects that other tools uploaded in parts have a multipart ETag (large objects), so rclone compares only their sizes and reports 1 hashes could not be checked. Files written over NFS or SMB are worse: their ETag isn't the file's MD5 (multiprotocol access), and rclone check reports md5 differ although the contents match. Compare those with rclone check --size-only or rclone check --download.

See also

Related articles