---
title: "Troubleshooting VAST DataStore"
slug: "dev-datastore-troubleshooting-index"
updated: 2026-10-28T02:15:03Z
published: 2026-10-28T02:15:03Z
canonical: "docs.vastdata.com/dev-datastore-troubleshooting-index"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting VAST DataStore

*Applies to VAST Cluster 5.5.*

Find what you're seeing in the tables below. Each row gives the likely cause and the fix, and links to the page with the details. The error texts are quoted as VAST Cluster 5.5 and the clients return them.

## Connecting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `SignatureDoesNotMatch` with s3cmd or a hand-built URL | A port in the endpoint (`:80`, `:443`). The port isn't configurable, and naming the default port can break Signature Version 4. | Leave the port out. See [the AWS CLI and tools](/develop/docs/dev-datastore-integrations-aws-cli-and-tools#s3cmd). |
| `[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain` | The cluster's S3 certificate is self-signed or signed by a CA your system doesn't trust. | Point the client at the CA certificate (`verify=`, `AWS_CA_BUNDLE`, `ca_bundle`). See [Python clients](/develop/docs/dev-datastore-integrations-python#https-and-certificates). |
| `hostname '<address>' doesn't match either of ...` | The CA is trusted now, but the name or address you connect to isn't in the certificate. | Connect with a name the certificate lists. |
| Connection refused on port 80 (or 443) | Your platform team disabled HTTP (or HTTPS) for S3. | Use the other scheme. |
| `403 InvalidAccessKeyId: The AWS access key Id you provided does not exist in our records.` | The key doesn't exist in the tenant you reach, was removed or is disabled. Key pairs copied by replication are disabled on the destination by default. | Check the key with your platform team, and that your endpoint serves your key's tenant. |
| `403 SignatureDoesNotMatch` with a key that exists | The secret key is wrong. | Check the secret; it's shown only once, when the key pair is created. |
| AWS CLI: `Unable to locate credentials` although `AWS_ACCESS_KEY_ID` is set | `--profile` on the command line makes the CLI ignore the environment variables. | Put the keys in the profile's credentials, or select the profile with `AWS_PROFILE` instead. See [the AWS CLI](/develop/docs/dev-datastore-integrations-aws-cli-and-tools#aws-cli). |
| The client goes to AWS, or fails resolving `<bucket>.s3.amazonaws.com` | No endpoint configured for that client or tool. | Set the endpoint explicitly; see [Python clients](/develop/docs/dev-datastore-integrations-python) and [the AWS CLI and tools](/develop/docs/dev-datastore-integrations-aws-cli-and-tools). |
| Requests to `https://<bucket>.<endpoint>/...` fail with `501 NotImplemented` | Virtual-hosted-style addressing. | Use path-style addressing (`addressing_style = path`, `force_path_style`, `fs.s3a.path.style.access`). |
| STS `AssumeRole` fails with `403 AccessDenied` | The request went over HTTP; STS answers over HTTPS only. | Use the `https://` endpoint. |
| STS `AssumeRole` fails with `400 PackedPolicyTooLarge` | The session policy is over 2,048 characters. | Shorten it; see [limits](/develop/docs/dev-datastore-reference-limits#s3). |
| STS `AssumeRole` fails with `400 ValidationError: request duration exceeds role's MaxSessionDuration` | `DurationSeconds` is longer than the role allows. | Ask for less, or ask your platform team for a longer maximum. |

## Access denied

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `AccessDenied` although an identity policy allows the action | The view doesn't use the S3 Native security flavor, so policies are ignored and file permissions or ACLs decide. | Ask which flavor the view uses; see [concepts](/develop/docs/dev-datastore-concepts-index#what-decides-whether-a-request-is-allowed). |
| `AccessDenied` on objects written by another user | With S3 Native, a new object's default ACL gives only its owner access. | Ask for an identity or bucket policy that grants your user the actions; see [access](/develop/docs/dev-datastore-operations-access#identity-policies). |
| `AccessDenied` on a key under a directory created over NFS | With the NFS flavor, S3 requests need traverse (`x`) permission on every directory in the path, and write permission on the parent to upload. | Fix the directories' mode bits over NFS, or ask about the flavor. |
| `put_bucket_acl` or `put_object_acl` succeeds but nothing changes | With the NFS flavor, S3 ACL changes are accepted and ignored. | Change permissions over NFS. |
| `put_bucket_policy` is refused | You're not the bucket owner and lack `s3:PutBucketPolicy`, or the view isn't S3 Native, or the policy names a resource outside the bucket. Over 20 KB, it gets `400 InvalidPolicyDocument`. | See [bucket policies](/develop/docs/dev-datastore-operations-access#bucket-policies). |
| `400 MalformedPolicy: Policy is malformed` | The policy isn't in the format the cluster expects, for example an AWS-style principal (`{"AWS": ["arn:aws:iam:::user/..."]}`) instead of a VAST user or group (`{"User": ["name"]}`). | See [bucket policies](/develop/docs/dev-datastore-operations-access#bucket-policies). |
| `400 MalformedPolicy: Failed to allocate memory` | A bucket policy with many statements (about 100 or more). | Combine statements: list several actions and resources in one statement. |
| `CreateBucket` is refused | You aren't allowed to create buckets: no matching S3 Endpoint view or permission. Or the name is taken, possibly in another tenant (names are unique across the cluster). An invalid name gets `400 InvalidBucketName`. | Ask your platform team; pick another name. |

## Objects and requests

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `412 PreconditionFailed: At least one of the pre-conditions you specified did not hold` | An `If-Match` or `If-None-Match` condition wasn't met: the object changed, or (with `If-None-Match: *`) already exists. | Expected; re-read and retry. See [conditional requests](/develop/docs/dev-datastore-guides-s3-objects#conditional-writes-and-reads). |
| `If-None-Match: *` gets `501 NotImplemented` in one bucket but works in another | Conditional writes aren't supported in versioned buckets. | Use a non-versioned bucket for create-once keys. |
| `404 NoSuchKey` from a conditional delete with `If-Match: *` | The object no longer exists. | Expected. |
| `400 BadDigest: The Content-MD5 you specified did not match what we received.` | The `Content-MD5` you sent doesn't match the data. | Check how your code computes it (base64 of the binary MD5). |
| `put_bucket_versioning(... "Suspended")` gets `501 NotImplemented` | Versioning can't be suspended once enabled. | None; see [versioning](/develop/docs/dev-datastore-guides-versioning-and-object-lock#whats-different-from-aws). |
| `put_bucket_versioning(... "Enabled")` gets `501 NotImplemented` | The bucket's view has a protocol besides S3, such as NFS. Versioning needs an S3-only view. | Ask your platform team for an S3-only view. |
| `501 NotImplemented: A header you provided implies functionality that is not implemented.` on another request | The request isn't supported, for example bucket encryption, replication or website configuration. | See [S3 compatibility](/develop/docs/dev-datastore-reference-s3-compatibility#headers). |
| Deleting an object version gets `403 AccessDenied` | The version is under a legal hold or retention period. | Remove the legal hold, or wait for the retention date. In governance mode, bypass with `s3:BypassGovernanceRetention`; in compliance mode, nothing can bypass it. |
| Presigned POST upload fails for some files | The object name isn't ASCII (`403 AccessDenied`), or the POST policy is too large (`400 InvalidRequest`). | Rename, or shorten the policy; see [limits](/develop/docs/dev-datastore-reference-limits#s3). |
| `400 InvalidTag: Cannot provide multiple Tags with the same key` although the keys differ | A tag key over 128 or a value over 256 characters, or a character that isn't ASCII. | Shorten the tag or use ASCII; see [limits](/develop/docs/dev-datastore-reference-limits#s3). |
| `400 MetadataTooLarge` | User metadata over 2 KB, keys and values together. | Keep large metadata in the object or a separate object. |
| `400 KeyTooLong`, `400 InvalidObjectName` or `400 InvalidArgument: ... Invalid @s3_object_name=...` | The key is over 1,024 characters, has a part over 255 characters, contains `//` or starts with `/`, or has more than 198 `/`. | See [S3 compatibility](/develop/docs/dev-datastore-reference-s3-compatibility#protocol-basics). |
| `400 InvalidPart` from `upload_part` | The part number isn't between 1 and 10,000. | Use bigger parts. |
| A home-grown client's `CompleteMultipartUpload` fails | The request line uses an absolute URL. | Send the path form (`/<bucket>/<key>?uploadId=...`). |
| Browser upload fails with a CORS error, but only when the upload itself fails | A failed `PutObject` response has no `Access-Control-Allow-Origin` header, so the browser reports CORS instead of the real error. | Look at the response status in the browser's network log. |
| A rename (`x-amz-copy-disposition: replace`) doesn't apply the new metadata or ACL | A rename moves the object as it is: `x-amz-metadata-directive: REPLACE` and ACL headers on it are ignored. | Rename first, then change metadata with a normal copy, or the ACL with `put_object_acl`. |
| `400 InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection` | SSE-C needs HTTPS (and AES256). | Use HTTPS. |
| A multipart object's ETag doesn't match the file's MD5 | Expected: multipart ETags are computed from the parts and end in `-<parts>`. | Compare with a multipart ETag computed the same way; see [large objects](/develop/docs/dev-datastore-guides-large-objects-and-multipart#the-multipart-calls). |

## S3 and NFS on the same data

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| An object written over S3 isn't visible over NFS | Its key ends with `/`, or has `.` or `..` parts; such objects have no file. (Keys that start with `/` or contain `//` are refused by default.) | Use file-like keys; see [keys that have no file](/develop/docs/dev-datastore-guides-multiprotocol#keys-that-have-no-file). |
| An object written over S3 can't be read over NFS (`Permission denied`) | With S3 Native, only the owner may read new objects, and the NFS user isn't the key's user (a UID with no VAST user is denied; the key's user's UID can read). With NFS flavor, the mode bits from the view policy don't allow it. | See [ownership and permissions](/develop/docs/dev-datastore-guides-multiprotocol#ownership-and-permissions). |
| `chmod` or `chown` over NFS succeeds but has no effect | With S3 Native, permission changes over NFS are ignored. | Change access with policies or ACLs. |
| Files in a bucket are read-only over NFS, and each object is a directory of numbered files | The bucket is versioned: versioned objects are read-only over NFS and SMB, and each object appears as a directory with one file per version. | Write over S3, or use a non-versioned bucket. |
| A file written over NFS has an ETag that isn't its MD5 | Expected: files created or changed over NFS or SMB get an ETag computed from metadata. | Don't use those ETags as checksums. `rclone check` reports them as `md5 differ`; use `--size-only` or `--download`. |
| An NFSv4 client doesn't see a change made over S3 straight away | NFSv4 file delegations: close-to-open consistency holds only between NFSv4 clients. | Ask whether delegations are on for the view; reopen the file. |
| Files show owner `nobody` | NFSv3: root squash or all squash for your host. NFSv4: the cluster can't resolve the user (`BADOWNER`), usually because ID mapping isn't set up. | Ask about squash settings; set up ID mapping. |

## NFS

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `mount.nfs: access denied by server while mounting ...` | Your client's address isn't in the view policy's host rules for that protocol. | Ask your platform team to add it. |
| An NFSv4 mount fails with `mount.nfs: Operation not permitted` although the view has NFSv4 | A view above it in the path, such as a view on `/`, doesn't have NFSv4. | Ask your platform team; every view above an NFSv4 view needs NFSv4. |
| `mount.nfs: Protocol not supported` with `vers=4.0` | NFSv4.0 isn't supported. | Mount with `vers=4.1` or `vers=4.2`, or `vers=3`. |
| `setxattr` fails with `ENOTSUP` / `EOPNOTSUPP` | Not an NFSv4.2 mount, NFSv4.2 isn't enabled for the tenant, or a non-`user.` namespace. | Mount with `vers=4.2`; see [extended attributes](/develop/docs/dev-datastore-guides-nfs-for-applications#tag-files-with-extended-attributes). |
| `File name too long` | Over 255 characters with the default name-length setting. | Shorten it, or ask about the view policy's name length limit. |
| `nconnect` is ignored, or the mount fails with `mount.nfs: an incorrect mount option was specified` (on a 3.10 kernel) | The client kernel doesn't support it. | Use a newer distribution, or the VAST-NFS driver. |
| Locks aren't seen by other clients (NFSv3) | The mount uses `nolock`, or the client's lock service isn't running. | Remount without `nolock`. |

## See also

- [S3 compatibility](/develop/docs/dev-datastore-reference-s3-compatibility) and [limits](/develop/docs/dev-datastore-reference-limits).
- [Access](/develop/docs/dev-datastore-operations-access): check what your key can do.