--- title: "STS for S3 Troubleshooting Guide" slug: "sts-for-s3-troubleshooting-guide" description: "Diagnose OIDC-based STS federation issues on VAST with our checklist, covering common errors like InvalidIdentityToken and AccessDenied for effective troubleshooting." updated: 2026-10-28T20:02:03Z published: 2026-10-28T20:02:03Z canonical: "docs.vastdata.com/sts-for-s3-troubleshooting-guide" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # STS for S3 Troubleshooting Guide This guide helps diagnose and resolve issues with OIDC-based STS federation on VAST. --- ## Quick Diagnostic Checklist Before diving into specific errors, verify these common configuration points: | Component | How to Check | What to Look For | | --- | --- | --- | | OIDC Provider | `vcli: oidc list` | State = `OK` | | Tenant Association | `vcli: tenant list` | OIDC provider ID column populated | | VIP Assignment | `vcli: vippool list --tenant-id <ID>` | At least one VIP pool assigned | | IAM Role | `vcli: iamrole list --tenant-id <ID>` | Role exists in correct tenant | | Identity Policy | `vcli: identitypolicy list --tenant-id <ID>` | Policy exists and attached to role | | JWT Token | Decode and inspect | `iss`, `aud`, `exp` claims valid | --- ## Error Reference ### STS Errors (AssumeRoleWithWebIdentity) | Error | HTTP | Meaning | Common Causes | | --- | --- | --- | --- | | `InvalidIdentityToken` | 400 | JWT validation failed | Wrong token type, signature invalid, claims mismatch | | `ExpiredToken` | 400 | JWT has expired | Token `exp` claim is in the past | | `InvalidToken` | 400 | Token malformed or untrusted | Malformed JWT, unknown issuer | | `AccessDenied` | 403 | Not authorized to assume role | Role not found, wrong tenant, trust policy denial | ### S3 Errors (after successful STS) | Error | HTTP | Meaning | Common Causes | | --- | --- | --- | --- | | `InvalidAccessKeyId` | 403 | Credentials not recognized | Expired STS creds, wrong VIP, revoked keys | | `AccessDenied` | 403 | Operation not permitted | Identity policy doesn't allow this action | | `SignatureDoesNotMatch` | 403 | Request signature invalid | Corrupted credentials, clock skew | | `NoSuchBucket` | 404 | Bucket doesn't exist | Wrong bucket name, bucket not created | --- ## Symptom-Based Troubleshooting ### "AssumeRoleWithWebIdentity fails with InvalidIdentityToken" **Possible causes:** 1. **Using** `access_token` instead of `id_token` The `access_token` from Azure is meant for Microsoft Graph API, not VAST. Always use the `id_token`. **How to verify:** ```bash # Decode your token and check the audience cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.aud' ``` - If `aud` is `<https://graph.microsoft.com`> or similar → wrong token - If `aud` is your Azure App ID → correct token 2. **Audience (**`aud`) mismatch The `aud` claim must match the condition in your trust policy. **How to verify:** ```bash # Check JWT audience cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.aud' ``` This value must exactly match the `StringEquals` condition in your trust policy: ```json "Condition": {  "StringEquals": {    "login.microsoftonline.com//v2.0:aud": ""  } } ``` 3. **JWT signature cannot be verified** VAST fetches public keys from the OIDC provider to verify JWT signatures. If this fails: ```bash vcli: admin> oidc list # Check State column vcli: admin> oidc refresh_keys --id # Force a key refresh ``` --- ### "AssumeRoleWithWebIdentity fails with ExpiredToken" The JWT has passed its expiration time. **How to verify:** ```bash # Check expiration timestamp cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.exp' # Compare to current time date +%s ``` If current time > `exp` value, the token is expired. **Resolution:** Get a new JWT from your OIDC provider (re-authenticate). **Note:** Azure `id_token` typically expires in ~1 hour. Plan your credential renewal accordingly. --- ### "AssumeRoleWithWebIdentity fails with AccessDenied" **Possible causes:** 1. **Role doesn't exist** ```bash vcli: admin> iamrole list --tenant-id ``` Verify the role name matches what you're requesting. 2. **Wrong tenant name in role ARN** The role ARN format is: `arn:vast::<TENANT_NAME>:role/<ROLE_NAME>` Verify `TENANT_NAME` matches exactly (case-sensitive). 3. **Trust policy denies based on claims** If your trust policy has conditions on claims like `sub`, verify the JWT contains matching values: ```bash cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.sub' ``` --- ### "S3 operations fail with InvalidAccessKeyId" **Possible causes:** 1. **STS credentials have expired** STS credentials expire after the `DurationSeconds` specified (default: 900 seconds = 15 minutes). **Resolution:** Call `AssumeRoleWithWebIdentity` again to get fresh credentials. 2. **Using wrong VIP** STS credentials are tenant-scoped. You must send S3 requests to a VIP assigned to that tenant. ```bash vcli: admin> vippool list --tenant-id ``` Use a VIP from this list. 3. **Access keys were revoked** An administrator may have revoked all access keys for the role: ```bash vcli: admin> iamrole revoke_access_keys --id ``` **Resolution:** Call `AssumeRoleWithWebIdentity` again to get new credentials. 4. **Credentials not exported correctly** Verify environment variables are set: ```bash echo $AWS_ACCESS_KEY_ID echo $AWS_SECRET_ACCESS_KEY echo $AWS_SESSION_TOKEN ``` All three must be non-empty. The `AWS_SESSION_TOKEN` is required for STS-based credentials. --- ### "S3 operations fail with AccessDenied" The STS call succeeded, but the identity policy doesn't permit this operation. **How to diagnose:** 1. Check what policy is attached to the role: ```bash vcli: admin> iamrole show --id --identity-policies ``` 2. Review the policy actions: ```bash vcli: admin> identitypolicy show --id ``` 3. Common issues: - Policy allows `s3:GetObject` but not `s3:PutObject` - Policy restricts to specific bucket but you're accessing a different one - Policy restricts to specific object prefix **Example:** Policy allows only GET operations: ```json {  "Statement": [{    "Effect": "Allow",    "Action": ["s3:GetObject", "s3:ListBucket"],    "Resource": "*"  }] } ``` This policy will deny PUT, DELETE, etc. --- ### "OIDC provider shows PROVIDER_ACCESS_FAILURE" VAST cannot reach the OIDC discovery URL or fetch public keys. **How to diagnose:** 1. Check the discovery URL is correct: ```bash vcli: admin> oidc list # Note the discovery URL ``` 2. Test connectivity from VAST cluster (or a machine with similar network access): ```bash curl -s "https://login.microsoftonline.com//v2.0/.well-known/openid-configuration" | jq . ``` 3. Common causes: - Firewall blocking outbound HTTPS to `login.microsoftonline.com` - DNS resolution failure - Typo in tenant ID - Proxy required but not configured 4. Force a key refresh after fixing connectivity: ```bash vcli: admin> oidc refresh_keys --id ``` --- ### "OIDC provider creation fails" **Possible causes:** 1. **Invalid discovery URL** The URL must be reachable and return valid OIDC metadata. Test it: ```bash curl -s "" | jq . ``` Should return JSON with `issuer`, `jwks_uri`, `authorization_endpoint`, etc. 2. **Discovery URL returns an unexpected format** Some identity providers have non-standard configurations. Ensure the response includes required fields. --- ## OIDC Provider Scenarios ### Azure AD v1.0 vs v2.0 Endpoints Azure AD has two endpoint versions with different behaviors: | Aspect | v1.0 | v2.0 | | --- | --- | --- | | Discovery URL | `<https://login.microsoftonline.com/<TENANT>>/.well-known/openid-configuration` | `<https://login.microsoftonline.com/<TENANT>>/v2.0/.well-known/openid-configuration` | | `iss` claim format | `<https://sts.windows.net/<TENANT>>/` | `<https://login.microsoftonline.com/<TENANT>>/v2.0` | | Token endpoint | `oauth2/token` | `oauth2/v2.0/token` | | Scopes | Uses `resource` parameter | Uses `scope` parameter | **Important:** Your trust policy `Principal.Federated` must match the `iss` claim format for your chosen endpoint version. If you're getting `InvalidIdentityToken` errors, verify you're using consistent endpoint versions between: - OIDC provider discovery URL on VAST - Token request endpoint in your client - Trust policy principal value --- ## Credential Lifecycle Issues ### Understanding Expiration ```plaintext JWT Lifetime (~1 hour) ├── STS Creds #1 (15 min) ─► Expires ─► Get new STS creds with same JWT ├── STS Creds #2 (15 min) ─► Expires ─► Get new STS creds with same JWT ├── STS Creds #3 (15 min) ─► Expires ─► Get new STS creds with same JWT └── STS Creds #4 (15 min) ─► JWT expires ─► Must re-authenticate ``` ### Renewal Strategy **For long-running applications:** 1. Track both JWT expiration (`exp` claim) and STS credential expiration (`Expiration` field) 2. Renew STS credentials before they expire (e.g., at 80% of duration) 3. When JWT expires, trigger re-authentication flow 4. Consider using refresh tokens if your OIDC provider supports them ### After Key Revocation If an administrator revokes access keys for a role: ```bash vcli: admin> iamrole revoke_access_keys --id ``` - All existing STS credentials for that role become invalid immediately - New `AssumeRoleWithWebIdentity` calls will succeed and return new, valid credentials - This is useful for emergency credential rotation --- ## Diagnostic Commands Reference ### OIDC Provider ```bash # List all OIDC providers vcli: admin> oidc list # Refresh public keys (useful after connectivity issues) vcli: admin> oidc refresh_keys --id ``` ### Tenant Configuration ```bash # List tenants and their OIDC provider associations vcli: admin> tenant list # Check VIP pools for a tenant vcli: admin> vippool list --tenant-id ``` ### IAM Roles and Policies ```bash # List IAM roles in a tenant vcli: admin> iamrole list --tenant-id # Show role details including trust policy vcli: admin> iamrole show --id # Show role with attached identity policies vcli: admin> iamrole show --id --identity-policies # List identity policies vcli: admin> identitypolicy list --tenant-id # Show policy details vcli: admin> identitypolicy show --id # Revoke all access keys for a role (emergency) vcli: admin> iamrole revoke_access_keys --id ``` ### JWT Inspection (Client-Side) ```bash # Decode JWT header cat /tmp/jwt_token.txt | cut -d'.' -f1 | base64 -d 2>/dev/null | jq . # Decode JWT payload (claims) cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq . # Check specific claims cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '{iss, aud, sub, exp, email}' # Check if token is expired EXP=$(cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq -r '.exp') NOW=$(date +%s) if [ "$NOW" -ge "$EXP" ]; then echo "Token EXPIRED"; else echo "Token valid for $((EXP - NOW)) seconds"; fi ``` ### AWS CLI Debugging ```bash # Enable debug output for AWS CLI aws s3 ls s3://mybucket/ \  --endpoint-url "https://" \  --no-verify-ssl \  --debug ``` ### boto3 Debugging (Python) ```plaintext import boto3 import logging # Enable debug logging boto3.set_stream_logger('', logging.DEBUG) ``` --- ## Common Configuration Mistakes | Mistake | Symptom | Fix | | --- | --- | --- | | Using `access_token` instead of `id_token` | `InvalidIdentityToken` | Use `id_token` from OIDC response | | Mismatched `iss` in trust policy | `InvalidIdentityToken` | Remove `<https://`,> prepend `oidc-provider/` | | Wrong `aud` in trust policy condition | `InvalidIdentityToken` | Use exact Azure App ID | | OIDC provider not associated with tenant | `AccessDenied` on STS | Associate in Web UI | | S3 requests to wrong VIP | `InvalidAccessKeyId` | Use VIP assigned to tenant | | Missing `AWS_SESSION_TOKEN` | `InvalidAccessKeyId` | Export all three credential env vars | | Identity policy too restrictive | `AccessDenied` on S3 ops | Review policy actions and resources | | v1.0 vs v2.0 endpoint mismatch | `InvalidIdentityToken` | Use consistent endpoint version |