--- title: "S3cmd" slug: "s3cmd" tags: ["5.5"] updated: 2026-10-27T23:00:54Z published: 2026-10-27T23:00:54Z canonical: "docs.vastdata.com/s3cmd" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # S3cmd ## Using S3cmd with VAST Cluster S3 > [!NOTE] > Note > > Documentation for S3cmd can be found on [s3cmd's website](https://s3tools.org/usage). Please use the S3cmd documentation and inline command help as needed in conjunction with our specific guidelines for using S3cmd with VAST Cluster. [S3cmd](https://s3tools.org/s3cmd) is a command line client for uploading, retrieving and managing data with the S3 service. The sections below describe how to install and configure S3cmd for use with VAST Cluster and how to use it to perform supported tasks. ## Install and Configure S3cmd Browse to [https://s3tools.org/s3cmd](https://s3tools.org/s3cmd) for system requirements and download. Make sure the s3cmd is installed on your client machine. For example, on CentOS/Redhat Linux systems: ```plaintext sudo yum install s3cmd -y ``` Once s3cmd is installed, you need to configure it to connect to your VAST Cluster S3 account. Configurations, including user account credentials, are specified in a configuration file. The default configuration file is located under your home directory ($HOME/.s3cfg). This file contains all of the S3cmd settings. All s3cmd commands default to the default configuration file. You can create an additional configuration file for your VAST-specific settings, or add them to the default configuration file. When using an additional configuration file, specify the file in your S3cmd commands. ### Minimal Configuration File Content Your configuration file should minimally include the following content. Save it to your home directory as *.s3cfg*, where ** is a name of your choice. In our examples. we will use *vast.s3cfg*. ```plaintext [default] access_key = secret_key = host_base = host_bucket = use_https = false|true ``` In which: - `<ACCESS-KEY>` and `<SECRET-KEY>` are your [S3 key pair](/docs-5-5/docs/managing-s3-user-permissions#managing-s3-user-access-from-thevast-web-ui). - `<CLUSTER VIRTUAL IP>` can be any of the cluster's Virtual IPs. > [!NOTE] > Note > > To retrieve the cluster's virtual lPs: > > - In the VAST Web UI, from the left navigation menu choose Network Access -> Virtual IPs to open the Virtual IPs page. It shows you which virtual IPs are configured on each CNode. > - In the VAST CLI, run the `vip list` command. > Caution > > Do not specify a port number. The port is not configurable. Specifying the default port can cause an error with signature version 4. - To connect over HTTPS, set `use_https=true`. When this is set, the service will listen on port 443, the secure port. HTTPS connection will only work if an SSL certificate for S3 is installed on the VMS. See [Enabling an HTTPS Connection](/docs-5-5/docs/managing-s3-connection-settings#enabling-an-https-connection). - To connect over HTTP, set `use_https=false`. ### Additional Configuration Options For more configuration options, see [s3cmd documentation](https://s3tools.org/usage). Configuration options can be passed in individual commands or in the configuration file. #### For HTTPS Connections If you are configuring S3cmd to connect to S3 over HTTPS, you need to make sure that the connection doesn't fail due to failed certificate verification. An SSL certificate must be [installed on the VMS for the S3 service](/docs-5-5/docs/managing-s3-connection-settings#enabling-an-https-connection). The S3 service provides the installed certificate to S3cmd as part of the SSL handshake when establishing an HTTPS connection to the service. By default, S3cmd verifies the certificate using the default local certificate trust store. The local path to the default certificate store is OS dependent. If the default trust store doesn't recognize the CA that signed the certificate, such as with a self signed certificate, the connection will fail with the default configuration. The following are commands you might include in the configuration file to handle SSL certificate verification: - To disable certificate verification: ```plaintext check_ssl_certificate = False ``` - Alternatively, to set a non-default path to the certificate trust store and point it to your certificate: ```plaintext ca-certs = ``` - Often there is a mismatch between the hostname embedded in the certificate and the service endpoint URL, which causes failure when the hostname is verified. To disable checking the hostname: ```plaintext no-check-hostname ``` #### Signature Versions VAST Cluster supports both signature version 4 (default and recommended) and signature version 2. To enforce signature version 2 specify the following in the configuration file: ```plaintext signature_v2=true ``` ## Create a Bucket ```plaintext s3cmd [-c CONFIGFILE] mb s3://BUCKET [--ca-certs=CERTIFICATE] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket | | CERTIFICATE | SSL certificate for HTTPS connections. | ### Example Create a bucket named *bucket1* specifying our configuration file as the non default *vast.s3cfg*. ```plaintext $ s3cmd -c vast.s3cfg mb s3://bucket1 Bucket 's3://bucket1/' created ``` For HTTPS: ```plaintext $ s3cmd mb s3://testbucket --ca-certs=client_cert.crt Bucket 's3://testbucket/' created ``` ## List All Buckets on the Server ```plaintext s3cmd [-c CONFIGFILE] ls s3:// ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | ### Example List all buckets on the server, specifying the root path of all buckets (s3://) and specifying our configuration file as the non default *vast.s3cfg*. ```plaintext $ s3cmd -c vast.s3cfg ls s3:// 2019-06-24 09:55  s3://bucket1 ``` ## List Objects in a Bucket ```plaintext s3cmd [-c CONFIGFILE] ls [s3://BUCKET[/PREFIX]]} ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket | | PREFIX | Filter objects by prefix. Although the S3 structure is flat consisting of buckets containing objects with no hierarchy of subbuckets and subfolders, object keys can include prefixes separated by a delimiter. Hierarchy can be inferred from the prefixes. | ### Example List objects in the bucket *bucket1*. ```plaintext $ s3cmd -c vast.s3cfg ls s3://bucket1 2019-06-24 19:51  205 s3://bucket1/blue.txt 2019-06-24 19:52  93  s3://bucket1/brown.txt 2019-06-24 19:40  176 s3://bucket1/vast.s3cfg ``` ## Retrieve Bucket Metadata ```plaintext $ s3cmd [-c CONFIGFILE]-c info s3://BUCKET ``` | CONFIGFILE | The name of your VAST configuration file | | --- | --- | | BUCKET | The name of the bucket | ### Example Retrieve the metadata of the bucket *bucket1*. ```plaintext $ s3cmd -c vast.s3cfg info s3://bucket1 s3://bucket1/ (bucket):   Location:  vast-1   Payer:     none   Expiration Rule: none   Policy:    none   CORS:      none   ACL:       vastuser: FULL_CONTROL ``` ## Delete a Bucket ```plaintext s3cmd [-c CONFIGFILE] rb s3://BUCKET ``` > [!NOTE] > Note > > This command can be used with the `--recursive` flag which removes the bucket with all of its contents. This form of the command calls the *DeleteObjects* AWS S3 API which is supported from VAST Cluster 3.2.0. | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket | ### Example Delete bucket *bucket1*. ```plaintext $ s3cmd -c vast.s3cfg rb s3://bucket1 ``` ## Set Bucket ACL Before setting ACL permissions, we recommend you read [Managing S3 Access Control Lists (ACLs)](/docs-5-5/docs/managing-s3-access#managing-s3-access-control-lists-acls). ```plaintext s3cmd [-c CONFIGFILE] setacl s3://BUCKET[/OBJECT] [--acl-grant=PERMISSION:{USER|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}] ``` `CONFIGFILE` Configuration file. Defaults to $HOME/s3cf `BUCKET` Name of bucket `--acl-grant` Grant stated permission for stated user or group. `--acl-revoke` Revoke stated permission for stated user or group `PERMISSION` Type of permission to grant: - read - write - read_acp - write_acp - full_control - all `USER` A user to which you want to grant the permission. Can be specified as: . - A principal name in the format *user@domain*, where *user* is the user name and *domain* is configured for an external auth provider on the cluster (LDAP, NIS). > [!NOTE] > Note > > Users on the VAST provider cannot be specified this way. - A VID, which is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the [user query](/docs-5-5/docs/user-query) VAST CLI command and specify *udb* as the context of the query. The output includes the user's VID. `GROUP` A group to which you want to grant the permission. The group can be specified as follows: - Set the type to *GroupLoginName* and specify the group in the format *group@domain*, where *group* is the group name and *domain* is configured for an external auth provider on the cluster (such as LDAP). - Specify the group's VID, which is the VAST ID used in the cluster's internal user database. A group VID can be retrieved by running the [group query](/docs-5-5/docs/group-query) VAST CLI command and specify *udb* as the context of the query. The output includes the group's VID. If you want to grant permissions to an S3 predefined group, specify one of the following: - `http://acs.amazonaws.com/groups/global/AllUsers`, or - `http://acs.amazonaws.com/groups/global/AuthenticatedUsers` ### Examples Grant read permissions for bucket *mybbucket* to group *mygroup@domain.com*: ```plaintext $ s3cmd setacl s3://mybucket --acl-grant=read:'groupLoginName=mygroup@domain.com' ``` Grant full control to the predefined group *AuthenticatedUsers* to access the bucket *mybucket*. ```plaintext $ s3cmd setacl s3://mybucket --acl-grant='full_control:http://acs.amazonaws.com/groups/global/AuthenticatedUsers' ``` Grant read permission to the predefined *AllUsers* group to access the bucket *mybucket*. ```plaintext $ s3cmd setacl s3://mybucket --acl-grant='read:http://acs.amazonaws.com/groups/global/AllUsers' ``` Grant user with VAST ID 14 write permission to the bucket *mybucket*: ```plaintext $ s3cmd setacl s3://mybucket --acl-grant='write:14 ``` ## Return Bucket ACL ```plaintext s3cmd [-c CONFIGFILE] info s3://BUCKET ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket | ### Example Retrieve all metadata of the bucket *bucket1*. The output includes the bucket's ACL. ```plaintext $ s3cmd -c vast.s3cfg info s3://bucket1 s3://bucket1/ (bucket):   Location:  vast-1   Payer:     none   Expiration Rule: none   Policy:    none   CORS:      none   ACL:       vastuser: FULL_CONTROL ``` ## Upload an Object ```plaintext s3cmd [-c CONFIGFILE] [--disable-multipart] put FILE [FILE...] s3://BUCKET[/PREFIX] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | --disable-multipart | Disables multipart upload for files over 15 MB (which by default are loaded as multipart uploads) | | FILE | Name of local file to upload | | BUCKET | Name of bucket to which to upload file | | PREFIX | Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by the delimiter (which is typically '/'). Object listing can be filtered by prefixes and delimiters. See [List Objects in a Bucket](/docs-5-5/docs/s3cmd#list-objects-in-a-bucket). For example, when uploading a file called *file.asp*, if you specify a prefix */dev/misc* the complete S3 object key becomes *dev/misc/file.asp*. A file system hierarchy can be inferred consisting of a folder called *misc* nested under a folder called *dev* in which the file *file.asp* is contained. | ### Example Add file *blue.txt* to bucket *bucket1*. ```plaintext $ s3cmd -c vast.s3cfg put blue.txt s3://bucket1 upload: 'blue.txt' -> 's3://bucket1/blue.txt'  [1 of 1] 205 of 205   100% in  0s  4.33 kB/s  done ``` ## Create a Copy of an Object ```plaintext s3cmd [-c CONFIGFILE] cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]  ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET1 | Bucket from which to copy object | | OBJECT1 | Source object to copy | | BUCKET2 | Bucket in which to create new object | | OBJECT2 | Name of destination object, if different from source object | ### Example Create copy of *file.zip*, which is stored in bucket *mybucket*, store the copy in the same bucket and call it *copyoffile.zip*: ```plaintext $ s3cmd -c vast.s3cfg cp s3://mybucket/file.zip s3://mybucket/copyoffile.zip ``` ## Download an Object ```plaintext s3cmd [-c CONFIGFILE] get s3://BUCKET/OBJECT LOCAL_FILE ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Bucket from which to retrieve stored object | | OBJECT | Object to download | | LOCAL_FILE | The local file path to download to | ### Example Download the file *file.zip* from the bucket *bucket.zip*. ```plaintext $ s3cmd -c vast.s3cfg get s3://mybucket/file.zip file.zip ``` ## Retrieve Object Metadata ```plaintext   s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Bucket in which object is stored | | OBJECT | Object for which to retrieve metadata | ### Example Retrieve metadata of file *vast.s3cfg*. ```plaintext $ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg s3://bucket1/vast.s3cfg (object):   File size: 176   Last mod:  Mon, 24 Jun 2019 19:40:27 GMT   MIME type: text/plain   Storage:   STANDARD   MD5 sum:   9de54b7b7a89d9526b32305a331dea6a   SSE:       none   Policy:    none   CORS:      none   ACL:       vastuser: FULL_CONTROL   x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast                                               data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u                                               name:vastdata ``` ## Delete an Object ```plaintext s3cmd [-c CONFIGFILE] del s3://BUCKET/OBJECT ``` or (alias for del) ```plaintext s3cmd [-c CONFIGFILE] rm s3://BUCKET/OBJECT ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket | | OBJECT | Name of object to delete | ### Example Delete the file *brown.txt* from the bucket *bucket1*. ```plaintext $ s3cmd -c vast.s3cfg rm s3://bucket1/brown.txt delete: 's3://bucket1/brown.txt' ``` ## Set Object ACL Before setting ACL permissions, we recommend you read [Managing S3 Access Control Lists (ACLs)](/docs-5-5/docs/managing-s3-access#managing-s3-access-control-lists-acls). ```plaintext s3cmd [-c CONFIGFILE] setacl s3://BUCKET/OBJECT [--acl-grant=PERMISSION:{USER_VAST_ID|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}] ``` `CONFIGFILE` Configuration file. Defaults to $HOME/s3cf `BUCKET` Name of bucket `--acl-grant` Grant stated permission for stated user or group. `--acl-revoke` Revoke stated permission for stated user or group `PERMISSION` Type of permission to grant: - read - write - read_acp - write_acp - full_control - all `USER` A user to which you want to grant the permission. Can be specified as: . - A principal name in the format *user@domain*, where *user* is the user name and *domain* is configured for an external auth provider on the cluster (LDAP, NIS). > [!NOTE] > Note > > Users on the VAST provider cannot be specified this way. - A VID, which is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the [user query](/docs-5-5/docs/user-query) VAST CLI command and specify *udb* as the context of the query. The output includes the user's VID. `GROUP` A group to which you want to grant the permission. The group can be specified as follows: - Set the type to *GroupLoginName* and specify the group in the format *group@domain*, where *group* is the group name and *domain* is configured for an external auth provider on the cluster (such as LDAP). - Specify the group's VID, which is the VAST ID used in the cluster's internal user database. A group VID can be retrieved by running the [group query](/docs-5-5/docs/group-query) VAST CLI command and specify *udb* as the context of the query. The output includes the group's VID. If you want to grant permissions to an S3 predefined group, specify one of the following: - `http://acs.amazonaws.com/groups/global/AllUsers`, or - `http://acs.amazonaws.com/groups/global/AuthenticatedUsers` ### Example Grant write permission to object *myfile.asp* in bucket *mybucket* to the user whose VID is 6. ```plaintext $ s3cmd -c vast.s3cfg setacl s3://mybucket/myfile.asp --acl-grant=write:6 ``` Grant read permission to object *myfile.asp* in bucket *mybucket* to group *mygroup@domain.com*: ```plaintext $ s3cmd -c vast.s3cfg setacl s3://mybucket/myfile.asp --acl-grant=read:'groupLoginName=mygroup@domain.com' ``` ## Return Object ACL ```plaintext s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket in which object is stored | | OBJECT | Name of object for which to return ACL | ### Example Get ACL of file *vast.s3cfg* as part of full metadata output. ```plaintext $ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg s3://bucket1/vast.s3cfg (object):   File size: 176   Last mod:  Mon, 24 Jun 2019 19:40:27 GMT   MIME type: text/plain   Storage:   STANDARD   MD5 sum:   9de54b7b7a89d9526b32305a331dea6a   SSE:       none   Policy:    none   CORS:      none   ACL:       vastuser: FULL_CONTROL   x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast                                               data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u                                               name:vastdata ``` ## Initiate a Multipart Upload Multipart uploads are automatically used when a file to upload is larger than 15MB. The file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is uploaded separately. The file is reconstructed at the destination when the transfer is completed. If an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded. ```plaintext s3cmd [-c CONFIGFILE] put [--multipart-chunk-size-mb=SIZE] FILE  s3://BUCKET[/PREFIX] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | SIZE | Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded-multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB. | | FILE | A file over 15 MB in size. (If smaller, the file will be uploaded without being split into parts.) | | BUCKET | The bucket to which to upload the file. | | PREFIX | Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by the delimiter (which is typically '/'). Object listing can be filtered by prefixes and delimiters. See [List Objects in a Bucket](/docs-5-5/docs/s3cmd#list-objects-in-a-bucket). For example, when uploading a file called *file.asp*, if you specify a prefix */dev/misc* the complete S3 object key becomes *dev/misc/file.asp*. A file system hierarchy can be inferred consisting of a folder called *misc* nested under a folder called *dev* in which the file *file.asp* is contained. | ### Example Start uploading the large file *Big.zip* in parts of 500 MB: ```plaintext $ s3cmd -c  vast.s3cfg put --multipart-chunk-size-mb=500 Big.zip s3://mybucket/ upload: 'Big.zip' > 's3://mybucket/Big.zip' [part 1 of 16, 500MB] [1 of 1] ``` ## Abort a Multipart Upload After a multipart upload is aborted, no additional parts can be uploaded using the upload ID of the multipart upload. The storage consumed by any previously uploaded parts will be freed. To verify that all parts have been removed, see [List Uploaded Parts for a Specific Multipart Upload](/docs-5-5/docs/s3cmd#list-uploaded-parts-for-a-specific-multipart-upload). ```plaintext s3cmd [-c CONFIGFILE] abortmp s3://BUCKET/OBJECT ID ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | Name of bucket to which object is being uploaded | | OBJECT | Name of object which started uploading as multipart upload | | ID | ID of the multipart upload | ### Example If we kill the upload process mid-upload (ctrl-C), the upload ID is provided: ```plaintext upload: 'Big.zip' -> 's3://mybucket/Big.zip'  [part 5 of 16, 500MB] [1 of 1]  13041664 of 524288000     2% in  109s   116.48 kB/s^CERROR: Upload of 'Big.zip' part 5 failed. Use  s3cmd abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM to abort the upload, or  s3cmd --upload-id Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put ... to continue the upload. ``` Now we can use this upload ID to abort the upload: ```plaintext $ s3cmd -c vast.s3cfg abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y78LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM ``` ## List Uploaded Parts for a Specific Multipart Upload ```plaintext s3cmd [-c CONFIGFILE] listmp s3://BUCKET/OBJECT ID ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | The bucket the object was uploaded to | | OBJECT | The name of the object being uploaded | | ID | The ID of the multipart upload | ### Example List uploaded parts for specific initiated upload of *Big.zip* to *mybucket*. ```plaintext $ s3cmd -c vast.s3cfg listmp s3://mybucket/Big.zip Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM ``` ## List Multipart Uploads ```plaintext s3cmd [-c CONFIGFILE] multipart s3://BUCKET [ID] ``` | CONFIGFILE | Configuration file. Defaults to $HOME/.s3cfg | | --- | --- | | BUCKET | The name of a bucket (multipart uploads to the specified bucket are listed) | | ID | The ID of a specific multipart upload (for filtering list) | ### Example To list all current multipart uploads to bucket *mybucket* in progress ```plaintext $ s3cmd -c vast.s3cfg multipart s3://mybucket ``` ## Configure Bucket Logging To enable bucket logging of access to bucket `mybucket` to a bucket named `destination-bucket` so that log object keys are prefixed with `logs`: ```plaintext s3cmd accesslog --access-logging-target-prefix=s3://destination-bucket/logs/ s3://mybucket ```