--- title: "Integrating boto3, AWSCLI, and Hadoop with VAST STS" slug: "integrating-boto3-awscli-and-hadoop-with-vast-sts" description: "Learn to configure AWS applications with STS for secure, temporary credentials, enhancing security without permanent access keys. Step-by-step guide included." updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/integrating-boto3-awscli-and-hadoop-with-vast-sts" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Integrating boto3, AWSCLI, and Hadoop with VAST STS This guide explains how to configure AWS-compatible applications to authenticate with temporary credentials from the **Security Token Service (STS)** instead of long-lived access keys. STS provides short-lived, automatically expiring credentials that improve security by eliminating the need to distribute or store permanent credentials in applications. Temporary credentials can be issued on demand, scoped to specific permissions, and integrated with existing identity providers and authentication workflows. Many applications and tools that support the Amazon S3 API - including **Boto3, AWS CLI**, **Apache Hadoop**, and other S3-compatible clients - can use STS-issued credentials with little or no application changes. In most cases, the integration consists of obtaining temporary credentials from an STS endpoint and configuring the application to use the resulting access key, secret key, and session token. This document provides step-by-step instructions for configuring common applications to authenticate with STS. Each application is described, including the required configuration, how to supply temporary credentials, and any application-specific considerations or limitations. If you want to set up STS on your VAST cluster: [STS AssumeRole with a Permanent Access Key Pair (No OIDC)](/kb/docs/sts-assumerole-with-a-permanent-access-key-pair-no-oidc) ## Applications Usage For this section, we assume you have the application set up and ready for use. > [!NOTE] > If you're installing for the first time, refer to the [appendix](/develop/docs/integrating-boto3-awscli-and-hadoop-with-vast-sts#appendix) ## boto3 **boto3.py** ```python #!/usr/bin/env python3 """VAST STS AssumeRole smoke test (boto3).""" import os import boto3 from botocore.config import Config ROLE_ARN = os.environ.get(    "VAST_ROLE_ARN", "arn:vast::default:role/" ) STS_ENDPOINT = os.environ.get("VAST_STS_ENDPOINT", "https://vip") S3_ENDPOINT = os.environ.get("VAST_S3_ENDPOINT", "http://vip") CA_BUNDLE = os.environ.get("AWS_CA_BUNDLE", "/path/to/you/ca/budnle") BUCKET = os.environ.get("VAST_TEST_BUCKET", "") def main():    session = boto3.session.Session(region_name="us-east-1") #Needed    sts = session.client(        "sts",        endpoint_url=STS_ENDPOINT,        verify=CA_BUNDLE,        config=Config(signature_version="v4"),    )    print(f"Calling AssumeRole on {STS_ENDPOINT} ...")    out = sts.assume_role(        RoleArn=ROLE_ARN,        RoleSessionName="boto3-vast-tutorial",        DurationSeconds=3600, #15 minutes minimum to 36 hours maximum        )    creds = out["Credentials"]    print("OK assume_role")    print("  AccessKeyId :", creds["AccessKeyId"])    print("  Expiration  :", creds["Expiration"])    print("  SessionToken:", creds["SessionToken"][:24] + "...")    # Optional: prove temp creds work on S3    s3 = boto3.client(        "s3",        endpoint_url=S3_ENDPOINT,        aws_access_key_id=creds["AccessKeyId"],        aws_secret_access_key=creds["SecretAccessKey"],        aws_session_token=creds["SessionToken"],        region_name="us-east-1",        config=Config(            s3={"addressing_style": "path"},            signature_version="s3v4",        ),    )    print(f"Listing s3://{BUCKET}/tables/ via {S3_ENDPOINT} ...")    resp = s3.list_objects_v2(        Bucket=BUCKET, Prefix="...", Delimiter="/", MaxKeys=20    )    prefixes = [p["Prefix"] for p in resp.get("CommonPrefixes", [])]    print("  Prefixes:", prefixes)    if not prefixes:        print("  WARNING: empty listing — check role IAM policy for this bucket")    else:        print("OK S3 list with temporary credentials") if __name__ == "__main__":    main() ``` ### Output example **Details** ```plaintext Calling AssumeRole on https://vip ... OK assume_role  AccessKeyId : TEMP................  Expiration  : 2026-07-21 ...  SessionToken: AQAAAFRFTVA................ Listing s3://<>/tables/ via http://vip ...  Prefixes: ['tables/revenue_sts/', 'tables/sample_sts/', ...] OK S3 list with temporary credentials ``` ## aws cli **Details** ```plaintext export AWS_ACCESS_KEY_ID='' export AWS_SECRET_ACCESS_KEY='' export AWS_DEFAULT_REGION=us-east-1 export AWS_CA_BUNDLE=/path/to/your/ca/bundle aws sts assume-role \  --role-arn arn:vast::default:role/ \  --role-session-name cli-check \  --duration-seconds 3600 \ #15 minutes minimum to 36 hours maximum  --endpoint-url https://vip ``` ### Output example **Details** ```plaintext OK assume_role AccessKeyId: TEMP... Expiration: 2026-07-21 12:35:23+00:00 S3 prefixes: tables/revenue_by_region/, revenue_sts/, sample/, sample_sts/ ``` ## Spark (Hadoop) ### Prerequisites - Before you begin, ensure that your VAST cluster is configured with a Java KeyStore (JKS) certificate to enable secure HTTPS communication. > [!NOTE] > In case your Spark context is longer then the sts expiration time, Spark (Hadoop) will ask for new keys (by default, 5 minutes before the key is expiried) **spark_sts.py** ```python #!/usr/bin/env python3 """ Spark job using Hadoop AssumedRoleCredentialProvider with VAST STS. With proper SSL/TLS configuration for HTTPS. """ from botocore.exceptions import ClientError from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType from datetime import datetime import urllib3 urllib3.disable_warnings() # Configuration VAST_ENDPOINT = "" BUCKET_NAME = "" BASE_ACCESS_KEY = "" BASE_SECRET_KEY = "" CA_CERT = "/opt/spark/conf/ssl/vast_cert/RootCA.crt" def create_spark_session():    """    Create Spark session with AssumedRoleCredentialProvider.    Uses SSL truststore for HTTPS STS endpoint.    """    print("=" * 60)    print("STEP 2: Creating Spark with AssumedRoleCredentialProvider")    print("=" * 60)    print("Configuration:")    print("  credentials.provider = AssumedRoleCredentialProvider")    print("  assumed.role.sts.endpoint = " + VAST_ENDPOINT)    print("  SSL: Using RootCA.crt truststore")    print("")        spark = SparkSession.builder \        .appName("VAST-STS-HTTPS-Demo") \        .config("spark.hadoop.fs.s3a.endpoint", VAST_ENDPOINT) \        .config("spark.hadoop.fs.s3a.path.style.access", "true") \        .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \        .config("spark.hadoop.fs.s3a.connection.ssl.enabled", "true") \        .config("spark.hadoop.fs.s3a.aws.credentials.provider",                "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider") \        .config("spark.hadoop.fs.s3a.assumed.role.arn",                "arn:vast::default:role/") \        .config("spark.hadoop.fs.s3a.assumed.role.sts.endpoint", VAST_ENDPOINT) \        .config("spark.hadoop.fs.s3a.assumed.role.sts.endpoint.region", "us-east-1") \        .config("spark.hadoop.fs.s3a.assumed.role.session.duration", "1h") \        .config("spark.hadoop.fs.s3a.assumed.role.session.name", "spark-vast-only") \        .config("spark.hadoop.fs.s3a.assumed.role.credentials.provider",                "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") \        .config("spark.hadoop.fs.s3a.access.key", BASE_ACCESS_KEY) \        .config("spark.hadoop.fs.s3a.secret.key", BASE_SECRET_KEY) \        .getOrCreate()        spark.sparkContext.setLogLevel("WARN")    print("✅ Spark session created with SSL")    return spark def write_parquet(spark):    """Write sample Parquet data via S3A."""    print("=" * 60)    print("STEP 3: Writing Parquet to VAST via S3A (HTTPS)")    print("=" * 60)        data = [        ("Alice", 34, "Engineering", 85000.0),        ("Bob", 45, "Marketing", 72000.0),        ("Charlie", 29, "Engineering", 78000.0),        ("Diana", 38, "Sales", 92000.0),        ("Eve", 41, "Engineering", 95000.0),    ]        schema = StructType([        StructField("name", StringType(), False),        StructField("age", IntegerType(), False),        StructField("department", StringType(), False),        StructField("salary", DoubleType(), False),    ])        df = spark.createDataFrame(data, schema)    df.show()        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")    output_path = f"s3a://{BUCKET_NAME}/employees_{timestamp}"        print(f"Writing to: {output_path}")    df.write.mode("overwrite").parquet(output_path)    print("✅ Parquet written successfully")        return output_path def read_parquet(spark, path):    """Read back and verify."""    print("=" * 60)    print("STEP 4: Reading Parquet back")    print("=" * 60)        df = spark.read.parquet(path)    print(f"✅ Read {df.count()} rows")        df.createOrReplaceTempView("employees")    spark.sql("""        SELECT department, COUNT(*) as count, ROUND(AVG(salary), 2) as avg_salary        FROM employees GROUP BY department ORDER BY avg_salary DESC    """).show() def main():    print("=" * 60)    print("SPARK + VAST STS with HTTPS")    print("Using Hadoop AssumedRoleCredentialProvider")    print("=" * 60)    print("")    spark = create_spark_session()        try:        output_path = write_parquet(spark)        read_parquet(spark, output_path)                print("")        print("=" * 60)        print("✅ SUCCESS - VAST STS works with HTTPS!")        print(f"   Output: {output_path}")        print("=" * 60)    finally:        spark.stop() if __name__ == "__main__":    main() ``` **docker-compose.yml** ```yaml services:  spark-job:    build: .    container_name: spark-vast-ssl    environment:      AWS_REGION: "us-east-1"      AWS_DEFAULT_REGION: "us-east-1"    command: >      /opt/spark/bin/spark-submit      --master local[*]      --conf "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true -Djavax.net.ssl.trustStore=/opt/spark/conf/ssl/truststore.jks -Djavax.net.ssl.trustStorePassword=password"      /app/spark_vast.py    volumes:      - ./spark_vast_only.py:/app/spark_vast_only.py:ro      - ./https:/opt/spark/conf/ssl:ro ``` ### Output example **Details** ```plaintext docker compose run --rm spark-job WARN[0000] Found orphan containers ([minio]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. ============================================================ SPARK + VAST STS with HTTPS Using Hadoop AssumedRoleCredentialProvider ============================================================ ============================================================ STEP 1: Checking bucket exists (HTTPS with CA cert) ============================================================ ============================================================ STEP 2: Creating Spark with AssumedRoleCredentialProvider ============================================================ Configuration:  credentials.provider = AssumedRoleCredentialProvider  assumed.role.sts.endpoint = https://172.27.117.1  SSL: Using RootCA.crt truststore 26/01/05 09:20:37 INFO SparkContext: Running Spark version 3.5.3 26/01/05 09:20:37 INFO SparkContext: OS info Linux, 4.18.0-372.26.1.el8.lb.x86_64, amd64 26/01/05 09:20:37 INFO SparkContext: Java version 11.0.24 26/01/05 09:20:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 26/01/05 09:20:37 INFO ResourceUtils: ============================================================== 26/01/05 09:20:37 INFO ResourceUtils: No custom resources configured for spark.driver. 26/01/05 09:20:37 INFO ResourceUtils: ============================================================== 26/01/05 09:20:37 INFO SparkContext: Submitted application: VAST-STS-HTTPS-Demo 26/01/05 09:20:37 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 26/01/05 09:20:37 INFO ResourceProfile: Limiting resource is cpu 26/01/05 09:20:37 INFO ResourceProfileManager: Added ResourceProfile id: 0 26/01/05 09:20:37 INFO SecurityManager: Changing view acls to: spark 26/01/05 09:20:37 INFO SecurityManager: Changing modify acls to: spark 26/01/05 09:20:37 INFO SecurityManager: Changing view acls groups to: 26/01/05 09:20:37 INFO SecurityManager: Changing modify acls groups to: 26/01/05 09:20:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY 26/01/05 09:20:37 INFO Utils: Successfully started service 'sparkDriver' on port 33435. 26/01/05 09:20:37 INFO SparkEnv: Registering MapOutputTracker 26/01/05 09:20:37 INFO SparkEnv: Registering BlockManagerMaster 26/01/05 09:20:37 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 26/01/05 09:20:37 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 26/01/05 09:20:37 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 26/01/05 09:20:37 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-32072331-a150-460c-b1d6-4551e2b525b1 26/01/05 09:20:37 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB 26/01/05 09:20:37 INFO SparkEnv: Registering OutputCommitCoordinator 26/01/05 09:20:37 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI 26/01/05 09:20:37 INFO Utils: Successfully started service 'SparkUI' on port 4040. 26/01/05 09:20:38 INFO Executor: Starting executor ID driver on host 8a85850b5ba8 26/01/05 09:20:38 INFO Executor: OS info Linux, 4.18.0-372.26.1.el8.lb.x86_64, amd64 26/01/05 09:20:38 INFO Executor: Java version 11.0.24 26/01/05 09:20:38 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): '' 26/01/05 09:20:38 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@746b2070 for default. 26/01/05 09:20:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38647. 26/01/05 09:20:38 INFO NettyBlockTransferService: Server created on 8a85850b5ba8:38647 26/01/05 09:20:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 26/01/05 09:20:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 8a85850b5ba8, 38647, None) 26/01/05 09:20:38 INFO BlockManagerMasterEndpoint: Registering block manager 8a85850b5ba8:38647 with 434.4 MiB RAM, BlockManagerId(driver, 8a85850b5ba8, 38647, None) 26/01/05 09:20:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 8a85850b5ba8, 38647, None) 26/01/05 09:20:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 8a85850b5ba8, 38647, None) ✅ Spark session created with SSL ============================================================ STEP 3: Writing Parquet to VAST via S3A (HTTPS) ============================================================ +-------+---+-----------+-------+ |   name|age| department| salary| +-------+---+-----------+-------+ |  Alice| 34|Engineering|85000.0| |    Bob| 45|  Marketing|72000.0| |Charlie| 29|Engineering|78000.0| |  Diana| 38|      Sales|92000.0| |    Eve| 41|Engineering|95000.0| +-------+---+-----------+-------+ Writing to: s3a://test-view2/employees_20260105_092041 26/01/05 09:20:41 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties ✅ Parquet written successfully ============================================================ STEP 4: Reading Parquet back ============================================================ ✅ Read 5 rows +-----------+-----+----------+ | department|count|avg_salary| +-----------+-----+----------+ |      Sales|    1|   92000.0| |Engineering|    3|   86000.0| |  Marketing|    1|   72000.0| +-----------+-----+----------+ ============================================================ ✅ SUCCESS - VAST STS works with HTTPS!   Output: s3a://test-view2/employees_20260105_092041 ============================================================ ``` ## Appendix ### Boto3 Installation ```plaintext pip install boto3 ``` ### AWS CLI installation [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ### Spark Installation [vastdataorg/spark-vast](https://hub.docker.com/r/vastdataorg/spark-vast)