apache/hadoop · error · PathCommitException

Filesystem not supported by this committer

Error message

Filesystem not supported by this committer

What it means

AbstractS3ACommitterFactory is bound through mapreduce.outputcommitter.factory.scheme.s3a (or a broader factory binding) and only accepts S3AFileSystem destinations: createOutputCommitter checks fs instanceof S3AFileSystem and otherwise throws PathCommitException 'Filesystem not supported by this committer'. The S3A committers stage data via S3 multipart uploads, so they cannot serve other filesystems.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java:51

/**
 * Dynamically create the output committer based on subclass type and settings.
 */
public abstract class AbstractS3ACommitterFactory
    extends PathOutputCommitterFactory {
  public static final Logger LOG = LoggerFactory.getLogger(
      AbstractS3ACommitterFactory.class);

  @Override
  public PathOutputCommitter createOutputCommitter(Path outputPath,
      TaskAttemptContext context) throws IOException {
    FileSystem fs = getDestinationFileSystem(outputPath, context);
    PathOutputCommitter outputCommitter;
    if (fs instanceof S3AFileSystem) {
      outputCommitter = createTaskCommitter((S3AFileSystem)fs,
          outputPath, context);
    } else {
      throw new PathCommitException(outputPath,
          "Filesystem not supported by this committer");
    }
    LOG.info("Using Committer {} for {} created by {}",
        outputCommitter,
        outputPath,
        this);
    return outputCommitter;
  }

  /**
   * Get the destination filesystem, returning null if there is none.
   * Code using this must explicitly or implicitly look for a null value
   * in the response.
   * @param outputPath output path
   * @param context job/task context
   * @return the destination filesystem, if it can be determined
   * @throws IOException if the FS cannot be instantiated
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Bind the S3A committer factory per scheme only: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory, leaving other schemes on their default committers
2. Remove any global factory override so non-s3a paths use the classic FileOutputCommitter
3. Point job outputs that must use S3A committers at s3a:// URIs (not legacy s3n:// or viewfs-wrapped paths)

Example fix

<!-- before: global factory binding breaks non-S3A outputs -->
<property><name>mapreduce.outputcommitter.factory.class</name>
  <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value></property>

<!-- after: bind per scheme -->
<property><name>mapreduce.outputcommitter.factory.scheme.s3a</name>
  <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value></property>

FileSystem fs = outputPath.getFileSystem(conf);
if (!(fs instanceof S3AFileSystem)) {
  conf.unset("mapreduce.outputcommitter.factory.scheme." + outputPath.toUri().getScheme());
  LOG.warn("Output {} on {} cannot use S3A committers", outputPath, fs.getScheme());
}

static boolean outputSupportsS3ACommitter(Path out, Configuration conf) throws IOException {
  return out.getFileSystem(conf) instanceof S3AFileSystem;
}

When it happens

Trigger: The output path of a job resolved through this factory lands on a non-S3A filesystem: local file tests (file://), HDFS, s3n/s3 (legacy), or viewfs; typically because the factory was bound globally instead of per-scheme.

Common situations: Setting a global mapreduce.outputcommitter.factory binding (or spark.sql.sources.outputCommitterClass in older setups) to gain S3A committers, then running a job whose output is local/HDFS; unit tests with output on LocalFileSystem inheriting the production committer config.

Related errors

• Multipart uploads are disabled for the FileSystem, the commi
• Task attempt {attemptID} has a self-generated job UUID
• Multipart IO request {sdkRequest} rejected {header}
• {component}: Invalid AWS credentials in {credentials} requir
• Class {className} {e} (configuration key fs.s3a.http.signer.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/485c179383df4577. Report an issue: GitHub.