Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Integrations

Prev Next

Settings for the clients and tools that reach VAST S3. Start with Python clients for boto3, s3fs and PyArrow; use Spark and Hadoop for S3A jobs, and AWS CLI, s3cmd and rclone for the command line.

  • Python clients: Configure boto3, s3fs (for pandas and Dask) and PyArrow's S3FileSystem for a VAST S3 endpoint, including HTTPS certificates.
  • Spark and Hadoop: Configure Hadoop's S3A connector for a VAST endpoint, tune its throughput, and commit Spark job output safely without atomic renames.
  • AWS CLI, s3cmd and rclone: The endpoint configuration that AWS CLI, s3cmd and rclone each need to move data to and from VAST S3.