Applies to VAST Cluster 5.5.
Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the page with the details. The error texts are quoted as VAST Cluster 5.5 and the clients return them.
Connecting
| Symptom | Likely cause | Fix |
|---|---|---|
SignatureDoesNotMatch with s3cmd or a hand-built URL |
A port in the endpoint (:80, :443). The port isn't configurable, and naming the default port can break Signature Version 4. |
Leave the port out. See the AWS CLI and tools. |
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain |
The cluster's S3 certificate is self-signed or signed by a CA your system doesn't trust. | Point the client at the CA certificate (verify=, AWS_CA_BUNDLE, ca_bundle). See Python clients. |
hostname '<address>' doesn't match either of ... |
The CA is trusted now, but the name or address you connect to isn't in the certificate. | Connect with a name the certificate lists. |
| Connection refused on port 80 (or 443) | Your platform team disabled HTTP (or HTTPS) for S3. | Use the other scheme. |
403 InvalidAccessKeyId: The AWS access key Id you provided does not exist in our records. |
The key doesn't exist in the tenant you reach, was removed or is disabled. Key pairs copied by replication are disabled on the destination by default. | Check the key with your platform team, and that your endpoint serves your key's tenant. |
403 SignatureDoesNotMatch with a key that exists |
The secret key is wrong. | Check the secret; it's shown only once, when the key pair is created. |
AWS CLI: Unable to locate credentials although AWS_ACCESS_KEY_ID is set |
--profile on the command line makes the CLI ignore the environment variables. |
Put the keys in the profile's credentials, or select the profile with AWS_PROFILE instead. See the AWS CLI. |
The client goes to AWS, or fails resolving <bucket>.s3.amazonaws.com |
No endpoint configured for that client or tool. | Set the endpoint explicitly; see Python clients and the AWS CLI and tools. |
Requests to https://<bucket>.<endpoint>/... fail with 501 NotImplemented |
Virtual-hosted-style addressing. | Use path-style addressing (addressing_style = path, force_path_style, fs.s3a.path.style.access). |
STS AssumeRole fails with 403 AccessDenied |
The request went over HTTP; STS answers over HTTPS only. | Use the https:// endpoint. |
STS AssumeRole fails with 400 PackedPolicyTooLarge |
The session policy is over 2,048 characters. | Shorten it; see limits. |
STS AssumeRole fails with 400 ValidationError: request duration exceeds role's MaxSessionDuration |
DurationSeconds is longer than the role allows. |
Ask for less, or ask your platform team for a longer maximum. |
Access denied
| Symptom | Likely cause | Fix |
|---|---|---|
AccessDenied although an identity policy allows the action |
The view doesn't use the S3 Native security flavor, so policies are ignored and file permissions or ACLs decide. | Ask which flavor the view uses; see concepts. |
AccessDenied on objects written by another user |
With S3 Native, a new object's default ACL gives only its owner access. | Ask for an identity or bucket policy that grants your user the actions; see access. |
AccessDenied on a key under a directory created over NFS |
With the NFS flavor, S3 requests need traverse (x) permission on every directory in the path, and write permission on the parent to upload. |
Fix the directories' mode bits over NFS, or ask about the flavor. |
put_bucket_acl or put_object_acl succeeds but nothing changes |
With the NFS flavor, S3 ACL changes are accepted and ignored. | Change permissions over NFS. |
put_bucket_policy is refused |
You're not the bucket owner and lack s3:PutBucketPolicy, or the view isn't S3 Native, or the policy names a resource outside the bucket. Over 20 KB, it gets 400 InvalidPolicyDocument. |
See bucket policies. |
400 MalformedPolicy: Policy is malformed |
The policy isn't in the format the cluster expects, for example an AWS-style principal ({"AWS": ["arn:aws:iam:::user/..."]}) instead of a VAST user or group ({"User": ["name"]}). |
See bucket policies. |
400 MalformedPolicy: Failed to allocate memory |
A bucket policy with many statements (about 100 or more). | Combine statements: list several actions and resources in one statement. |
CreateBucket is refused |
You aren't allowed to create buckets: no matching S3 Endpoint view or permission. Or the name is taken, possibly in another tenant (names are unique across the cluster). An invalid name gets 400 InvalidBucketName. |
Ask your platform team; pick another name. |
Objects and requests
| Symptom | Likely cause | Fix |
|---|---|---|
412 PreconditionFailed: At least one of the pre-conditions you specified did not hold |
An If-Match or If-None-Match condition wasn't met: the object changed, or (with If-None-Match: *) already exists. |
Expected; re-read and retry. See conditional requests. |
If-None-Match: * gets 501 NotImplemented in one bucket but works in another |
Conditional writes aren't supported in versioned buckets. | Use a non-versioned bucket for create-once keys. |
404 NoSuchKey from a conditional delete with If-Match: * |
The object no longer exists. | Expected. |
400 BadDigest: The Content-MD5 you specified did not match what we received. |
The Content-MD5 you sent doesn't match the data. |
Check how your code computes it (base64 of the binary MD5). |
put_bucket_versioning(... "Suspended") gets 501 NotImplemented |
Versioning can't be suspended once enabled. | None; see versioning. |
put_bucket_versioning(... "Enabled") gets 501 NotImplemented |
The bucket's view has a protocol besides S3, such as NFS. Versioning needs an S3-only view. | Ask your platform team for an S3-only view. |
501 NotImplemented: A header you provided implies functionality that is not implemented. on another request |
The request isn't supported, for example bucket encryption, replication or website configuration. | See S3 compatibility. |
Deleting an object version gets 403 AccessDenied |
The version is under a legal hold or retention period. | Remove the legal hold, or wait for the retention date. In governance mode, bypass with s3:BypassGovernanceRetention; in compliance mode, nothing can bypass it. |
| Presigned POST upload fails for some files | The object name isn't ASCII (403 AccessDenied), or the POST policy is too large (400 InvalidRequest). |
Rename, or shorten the policy; see limits. |
400 InvalidTag: Cannot provide multiple Tags with the same key although the keys differ |
A tag key over 128 or a value over 256 characters, or a character that isn't ASCII. | Shorten the tag or use ASCII; see limits. |
400 MetadataTooLarge |
User metadata over 2 KB, keys and values together. | Keep large metadata in the object or a separate object. |
400 KeyTooLong, 400 InvalidObjectName or 400 InvalidArgument: ... Invalid @s3_object_name=... |
The key is over 1,024 characters, has a part over 255 characters, contains // or starts with /, or has more than 198 /. |
See S3 compatibility. |
400 InvalidPart from upload_part |
The part number isn't between 1 and 10,000. | Use bigger parts. |
A home-grown client's CompleteMultipartUpload fails |
The request line uses an absolute URL. | Send the path form (/<bucket>/<key>?uploadId=...). |
| Browser upload fails with a CORS error, but only when the upload itself fails | A failed PutObject response has no Access-Control-Allow-Origin header, so the browser reports CORS instead of the real error. |
Look at the response status in the browser's network log. |
A rename (x-amz-copy-disposition: replace) doesn't apply the new metadata or ACL |
A rename moves the object as it is: x-amz-metadata-directive: REPLACE and ACL headers on it are ignored. |
Rename first, then change metadata with a normal copy, or the ACL with put_object_acl. |
400 InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection |
SSE-C needs HTTPS (and AES256). | Use HTTPS. |
| A multipart object's ETag doesn't match the file's MD5 | Expected: multipart ETags are computed from the parts and end in -<parts>. |
Compare with a multipart ETag computed the same way; see large objects. |
S3 and NFS on the same data
| Symptom | Likely cause | Fix |
|---|---|---|
| An object written over S3 isn't visible over NFS | Its key ends with /, or has . or .. parts; such objects have no file. (Keys that start with / or contain // are refused by default.) |
Use file-like keys; see keys that have no file. |
An object written over S3 can't be read over NFS (Permission denied) |
With S3 Native, only the owner may read new objects, and the NFS user isn't the key's user (a UID with no VAST user is denied; the key's user's UID can read). With NFS flavor, the mode bits from the view policy don't allow it. | See ownership and permissions. |
chmod or chown over NFS succeeds but has no effect |
With S3 Native, permission changes over NFS are ignored. | Change access with policies or ACLs. |
| Files in a bucket are read-only over NFS, and each object is a directory of numbered files | The bucket is versioned: versioned objects are read-only over NFS and SMB, and each object appears as a directory with one file per version. | Write over S3, or use a non-versioned bucket. |
| A file written over NFS has an ETag that isn't its MD5 | Expected: files created or changed over NFS or SMB get an ETag computed from metadata. | Don't use those ETags as checksums. rclone check reports them as md5 differ; use --size-only or --download. |
| An NFSv4 client doesn't see a change made over S3 straight away | NFSv4 file delegations: close-to-open consistency holds only between NFSv4 clients. | Ask whether delegations are on for the view; reopen the file. |
Files show owner nobody |
NFSv3: root squash or all squash for your host. NFSv4: the cluster can't resolve the user (BADOWNER), usually because ID mapping isn't set up. |
Ask about squash settings; set up ID mapping. |
NFS
| Symptom | Likely cause | Fix |
|---|---|---|
mount.nfs: access denied by server while mounting ... |
Your client's address isn't in the view policy's host rules for that protocol. | Ask your platform team to add it. |
An NFSv4 mount fails with mount.nfs: Operation not permitted although the view has NFSv4 |
A view above it in the path, such as a view on /, doesn't have NFSv4. |
Ask your platform team; every view above an NFSv4 view needs NFSv4. |
mount.nfs: Protocol not supported with vers=4.0 |
NFSv4.0 isn't supported. | Mount with vers=4.1 or vers=4.2, or vers=3. |
setxattr fails with ENOTSUP / EOPNOTSUPP |
Not an NFSv4.2 mount, NFSv4.2 isn't enabled for the tenant, or a non-user. namespace. |
Mount with vers=4.2; see extended attributes. |
File name too long |
Over 255 characters with the default name-length setting. | Shorten it, or ask about the view policy's name length limit. |
nconnect is ignored, or the mount fails with mount.nfs: an incorrect mount option was specified (on a 3.10 kernel) |
The client kernel doesn't support it. | Use a newer distribution, or the VAST-NFS driver. |
| Locks aren't seen by other clients (NFSv3) | The mount uses nolock, or the client's lock service isn't running. |
Remount without nolock. |
See also
- S3 compatibility and limits.
- Access: check what your key can do.