Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Spark and Hadoop

Prev Next

Applies to VAST Cluster 5.5.

Spark, Hadoop MapReduce and other Hadoop-based tools reach VAST S3 through Hadoop's S3A connector (s3a:// paths). This page gives the S3A settings for a VAST endpoint, and explains how to commit job output safely, since S3 has no atomic directory rename.

This page is about files and objects. To query VAST DataBase tables from Spark, use the VAST connector instead: see Spark.

The settings on this page were run against VAST Cluster 5.5 with Spark 3.5.3 and its Hadoop 3.3.4 client (hadoop-aws 3.3.4): a Parquet write and read through s3a://, with the classic and the magic committer, and a job that fails halfway.

Connection settings

Set these in core-site.xml, in spark-defaults.conf with the spark.hadoop. prefix, or on the command line with --conf spark.hadoop.<property>=<value> (these six properties are enough for Spark to write and read Parquet on VAST S3):

# spark-defaults.conf (not captured)
spark.hadoop.fs.s3a.endpoint                 http://s3.example.com
spark.hadoop.fs.s3a.path.style.access        true
spark.hadoop.fs.s3a.connection.ssl.enabled   false
spark.hadoop.fs.s3a.access.key               <access key>
spark.hadoop.fs.s3a.secret.key               <secret key>
spark.hadoop.fs.s3a.endpoint.region          us-east-1
Property Value for VAST Why
fs.s3a.endpoint http://<name or VIP> or https://... The cluster's S3 endpoint. No port.
fs.s3a.path.style.access true Path-style URLs. A virtual-hosted-style request (<bucket>.<endpoint>) got 501 NotImplemented.
fs.s3a.connection.ssl.enabled true for HTTPS, false for HTTP Must match the endpoint's scheme.
fs.s3a.endpoint.region Any region name, for example us-east-1 Recent S3A versions need a region for signing; the cluster accepts any.
fs.s3a.access.key, fs.s3a.secret.key Your key pair Better: a credentials provider or Hadoop credential store, so keys don't sit in plain config files.

For HTTPS with a certificate the JVM doesn't trust, add the certificate (or its CA) to the truststore your Spark driver and executors use.

With temporary credentials from the cluster's STS (access), use S3A's temporary-credentials provider and also set fs.s3a.session.token.

Throughput settings

S3A uploads large files as multipart uploads and keeps a pool of connections and threads. The settings that matter, all S3A's own:

Property What it controls
fs.s3a.multipart.size Part size for uploads. Keep it at least 5 MiB.
fs.s3a.multipart.threshold File size above which S3A switches to multipart.
fs.s3a.connection.maximum Connections per filesystem instance. Raise it with the thread count.
fs.s3a.threads.max Upload and copy threads per filesystem instance.
fs.s3a.fast.upload.buffer Where parts are buffered before upload: disk, array or bytebuffer.

How these choices affect speed on VAST is explained in performance. Executors spread over many hosts, each resolving the endpoint name, naturally spread over the cluster's virtual IPs.

Committing job output

Hadoop and Spark commit output by writing to a temporary directory and renaming it into place. On S3, a "directory" rename is a copy and delete of every object, so it's slow and not atomic: a failed job can leave partial output, and two jobs can see each other's files. That is true on VAST as on AWS. S3A doesn't use the VAST rename request.

So:

  • Don't use the classic FileOutputCommitter (v1 or v2) for s3a:// output.
  • Use one of the S3A committers, which write with multipart uploads and make files visible only when the job commits: the directory and partitioned staging committers, or the magic committer. The magic committer needs no local staging disk but depends on the store's multipart behaviour.
  • For overwrite and concurrent writers, use a table format (Iceberg, Delta Lake or Hudi) rather than overwriting directories. Dynamic partition overwrite isn't supported by the magic committer.

To use the magic committer from Spark:

# spark-defaults.conf (not captured)
spark.hadoop.fs.s3a.committer.name                              magic
spark.hadoop.fs.s3a.committer.magic.enabled                     true
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a       org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
spark.sql.sources.commitProtocolClass                           org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class                        org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter

The two Spark classes come from Spark's spark-hadoop-cloud module, which must be on the classpath. With these settings, a Spark 3.5 job wrote Parquet to VAST S3 through the magic committer, and its _SUCCESS file is the committer's JSON summary, which names the magic committer and the files it committed. A job that failed halfway left no output files and no incomplete multipart uploads behind. Test it with your own jobs, including a failing one, before you rely on it.

Check the setup

A quick check from a shell with the Hadoop client configured (these shell commands weren't run for these docs; the same file-system calls made from Spark worked):

# not captured
hadoop fs -mkdir -p s3a://<bucket>/docs-spark-and-hadoop/
hadoop fs -put localfile.csv s3a://<bucket>/docs-spark-and-hadoop/
hadoop fs -ls s3a://<bucket>/docs-spark-and-hadoop/
hadoop fs -rm -r s3a://<bucket>/docs-spark-and-hadoop/

If a setting doesn't seem to apply, check where it comes from: hdfs getconf -confKey fs.s3a.path.style.access prints the value the Hadoop client sees.

See also

Related articles