--- title: "AWS CLI, s3cmd and rclone" slug: "dev-datastore-integrations-aws-cli-and-tools" updated: 2026-10-28T02:15:03Z published: 2026-10-28T02:15:03Z canonical: "docs.vastdata.com/dev-datastore-integrations-aws-cli-and-tools" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # AWS CLI, s3cmd and rclone *Applies to VAST Cluster 5.5.* Command-line tools are the quickest way to move data in and out of VAST S3 and to check what's in a bucket. This page gives the configuration each tool needs for a VAST endpoint. The commands themselves are the tools' own; see their documentation for everything else. The configurations and commands on this page were run against VAST Cluster 5.5 with AWS CLI 1.46.1 and 2.27.0, s3cmd 2.4.0 and rclone 1.68.2. The outputs below are from those runs, with the bucket name replaced by `<bucket>`. ## AWS CLI Create a profile for VAST in `~/.aws/config`, with the endpoint and path-style addressing: ``` # ~/.aws/config (not captured) [profile vast] region = us-east-1 endpoint_url = http://s3.example.com s3 = addressing_style = path max_concurrent_requests = 16 multipart_threshold = 64MB multipart_chunksize = 16MB ``` and the keys in `~/.aws/credentials`: ``` # ~/.aws/credentials (not captured) [vast] aws_access_key_id = aws_secret_access_key = ``` If you'd rather keep the keys in `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, select the profile with `AWS_PROFILE=vast` instead of `--profile vast`. When you name a profile with `--profile`, the CLI ignores the environment variables and fails with `Unable to locate credentials` if the profile has no keys. Then: ``` # not captured aws --profile vast s3 ls s3:/// aws --profile vast s3 cp ./data.parquet s3:///docs-cli/data.parquet aws --profile vast s3 sync ./results/ s3:///docs-cli/results/ aws --profile vast s3 presign s3:///docs-cli/data.parquet --expires-in 300 aws --profile vast s3 rm --recursive s3:///docs-cli/ ``` With AWS CLI 2.27, the copy, sync, presign and remove print: ``` upload: ./data.parquet to s3:///docs-cli/data.parquet upload: results/a.txt to s3:///docs-cli/results/a.txt upload: results/b.txt to s3:///docs-cli/results/b.txt http://s3.example.com//docs-cli/data.parquet?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=20261012T145621Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature= delete: s3:///docs-cli/data.parquet delete: s3:///docs-cli/results/a.txt delete: s3:///docs-cli/results/b.txt ``` The presigned URL downloads the object without credentials until it expires. AWS CLI version 1 signs presigned URLs with Signature Version 2 query parameters (`AWSAccessKeyId`, `Signature`, `Expires`); those work too. Older AWS CLI versions don't read `endpoint_url` from the config file; pass `--endpoint-url http://s3.example.com` on each command instead (that works on every version). For HTTPS with a certificate your system doesn't trust, set `ca_bundle = /path/to/vast-s3-ca.pem` in the profile (or `--ca-bundle`), and connect with a name the certificate lists: with an address that isn't in the certificate, the CLI fails with `hostname '<address>' doesn't match ...` even with the CA bundle. The `aws s3api` commands accept the same profile, for example `aws --profile vast s3api get-object-tagging --bucket <bucket> --key <key>`. The `s3` settings in the profile control the CLI's transfers: how many requests run at once, and the size above which, and in which parts, it uploads multipart. With the settings above, an 80 MB file goes up in five parts, and its ETag ends in `-5`. What the settings do on VAST is explained in [performance](/develop/docs/dev-datastore-guides-performance#parallel-and-multipart-transfers). ## s3cmd A minimal configuration file, for example `~/vast.s3cfg`: ``` # ~/vast.s3cfg (not captured) [default] access_key = secret_key = host_base = s3.example.com host_bucket = s3.example.com use_https = False ``` - Set `host_base` and `host_bucket` to the same name or VIP, without a port. The port isn't configurable, and adding the default port can break Signature Version 4. - For HTTPS, set `use_https = True`. If your system doesn't trust the cluster's certificate, set `ca_certs_file` to it; if the name doesn't match the certificate, set `check_ssl_certificate = False` only on a test system. - s3cmd uses Signature Version 4 by default; `signature_v2 = True` switches to version 2. ``` # not captured s3cmd -c ~/vast.s3cfg ls s3:/// s3cmd -c ~/vast.s3cfg put --multipart-chunk-size-mb=64 big.bin s3:///docs-cli/ s3cmd -c ~/vast.s3cfg info s3:///docs-cli/big.bin s3cmd -c ~/vast.s3cfg del --recursive s3:///docs-cli/ ``` For an 80 MB `big.bin`, the upload and the start of `info` print: ``` upload: 'big.bin' -> 's3:///docs-cli/big.bin' (80000000 bytes in 2.0 seconds, 38.21 MB/s) [1 of 1] s3:///docs-cli/big.bin (object): File size: 80000000 ``` s3cmd uploads files larger than its chunk size (15 MB by default) as multipart uploads (an 80 MB file goes up in six parts). KB: [S3cmd](/docs-5-5/docs/s3cmd). ## rclone rclone treats VAST S3 as a generic S3 provider. Add a remote to `~/.config/rclone/rclone.conf`: ``` # ~/.config/rclone/rclone.conf (not captured) [vast] type = s3 provider = Other endpoint = http://s3.example.com region = us-east-1 force_path_style = true acl = private env_auth = true ``` With `env_auth = true`, rclone takes the keys from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. Then: ``` # not captured rclone lsf vast:/ rclone copy ./results vast:/docs-cli/results --transfers 16 --s3-chunk-size 16M rclone check ./results vast:/docs-cli/results rclone purge vast:/docs-cli ``` `copy` and `purge` print nothing when they succeed. `check` prints: ``` 2026/10/12 14:56:49 NOTICE: S3 bucket path docs-cli/results: 0 differences found 2026/10/12 14:56:49 NOTICE: S3 bucket path docs-cli/results: 2 matching files ``` `--transfers` sets how many files move at once and `--s3-upload-concurrency` how many parts of one file. rclone checks files by MD5. For files it uploaded itself, multipart ones included, that works, because rclone records the MD5 in the object's metadata. Objects that other tools uploaded in parts have a multipart ETag ([large objects](/develop/docs/dev-datastore-guides-large-objects-and-multipart#the-multipart-calls)), so rclone compares only their sizes and reports `1 hashes could not be checked`. Files written over NFS or SMB are worse: their ETag isn't the file's MD5 ([multiprotocol access](/develop/docs/dev-datastore-guides-multiprotocol#files-are-objects)), and `rclone check` reports `md5 differ` although the contents match. Compare those with `rclone check --size-only` or `rclone check --download`. ## See also - [Python clients](/develop/docs/dev-datastore-integrations-python). - [Access](/develop/docs/dev-datastore-operations-access): keys and temporary credentials.