apache/hadoop · error · PathCommitException

Multipart uploads are disabled for the FileSystem, the commi

Error message

Multipart uploads are disabled for the FileSystem, the committer can't proceed.

What it means

All S3A committers (staging, magic, directory) stage their work as S3 multipart uploads, so at committer construction AbstractS3ACommitter checks S3AFileSystem.isMultipartUploadEnabled() and aborts with PathCommitException if multipart uploads are disabled. The check is deliberately early: without multipart, pending commits could never be completed.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java:222

  protected AbstractS3ACommitter(
      Path outputPath,
      TaskAttemptContext context) throws IOException {
    super(outputPath, context);
    setOutputPath(outputPath);
    this.jobContext = requireNonNull(context, "null job context");
    this.role = "Task committer " + context.getTaskAttemptID();
    setConf(context.getConfiguration());
    Pair<String, JobUUIDSource> id = buildJobUUID(
        conf, context.getJobID());
    this.uuid = id.getLeft();
    this.uuidSource = id.getRight();
    LOG.info("Job UUID {} source {}", getUUID(), getUUIDSource().getText());
    initOutput(outputPath);
    LOG.debug("{} instantiated for job \"{}\" ID {} with destination {}",
        role, jobName(context), jobIdString(context), outputPath);
    S3AFileSystem fs = getDestS3AFS();
    if (!fs.isMultipartUploadEnabled()) {
      throw new PathCommitException(outputPath, "Multipart uploads are disabled for the FileSystem,"
          + " the committer can't proceed.");
    }
    // set this thread's context with the job ID.
    // audit spans created in this thread will pick
    // up this value., including the commit operations instance
    // soon to be created.
    new AuditContextUpdater(jobContext)
        .updateCurrentAuditContext();

    // the filesystem is the span source, always.
    this.auditSpanSource = fs.getAuditSpanSource();
    this.createJobMarker = context.getConfiguration().getBoolean(
        CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER,
        DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER);
    // the statistics are shared between this committer and its operations.
    this.committerStatistics = fs.newCommitterStatistics();
    this.commitOperations = new CommitOperations(fs, committerStatistics,
        outputPath.toString());

View on GitHub (pinned to 2add963021)

Solutions

Remove the override or set fs.s3a.multipart.uploads.enabled=true (default) for jobs using S3A committers
If multipart must stay disabled on that filesystem, do not route its jobs to S3A committers: unset mapreduce.outputcommitter.factory.scheme.s3a so the classic FileOutputCommitter is used
Check both cluster core-site.xml and job/spark.hadoop.* overrides; the task-side config is the one that counts

Example fix

<!-- before -->
<property><name>fs.s3a.multipart.uploads.enabled</name><value>false</value></property>

<!-- after (or just remove the property; default is true) -->
<property><name>fs.s3a.multipart.uploads.enabled</name><value>true</value></property>

Defensive patterns

Strategy: validation

Validate before calling

Configuration conf = job.getConfiguration();
if (!conf.getBoolean("fs.s3a.multipart.uploads.enabled", true)) {
  throw new IOException("S3A committers require fs.s3a.multipart.uploads.enabled=true");
}

Prevention

Validate the multipart flag at job submission, not at task time
Config-lint clusters for fs.s3a.multipart.uploads.enabled=false when committers are in use
When multipart must be disabled, also unbind the S3A committer factory for that path

When it happens

Trigger: Task or job committer initialization runs against a destination whose configuration sets fs.s3a.multipart.uploads.enabled=false (default is true). Magic committer integration is also disabled by the same flag, so magic-committed writes fail identically.

Common situations: An operator disabled multipart uploads cluster-wide (e.g. to work around an S3 or proxy issue) while Spark/Hive jobs still select an S3A committer via mapreduce.outputcommitter.factory.scheme.s3a; a job-level override sets the flag false for one path; copy-pasted 'tuning' configs carrying the false value.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/784ce52b9acf6590. Report an issue: GitHub.