{"record":{"id":"485c179383df4577","repo":"apache/hadoop","slug":"filesystem-not-supported-by-this-committer","errorCode":null,"errorMessage":"Filesystem not supported by this committer","messagePattern":"Filesystem not supported by this committer","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java","lineNumber":51,"sourceCode":"\n/**\n * Dynamically create the output committer based on subclass type and settings.\n */\npublic abstract class AbstractS3ACommitterFactory\n    extends PathOutputCommitterFactory {\n  public static final Logger LOG = LoggerFactory.getLogger(\n      AbstractS3ACommitterFactory.class);\n\n  @Override\n  public PathOutputCommitter createOutputCommitter(Path outputPath,\n      TaskAttemptContext context) throws IOException {\n    FileSystem fs = getDestinationFileSystem(outputPath, context);\n    PathOutputCommitter outputCommitter;\n    if (fs instanceof S3AFileSystem) {\n      outputCommitter = createTaskCommitter((S3AFileSystem)fs,\n          outputPath, context);\n    } else {\n      throw new PathCommitException(outputPath,\n          \"Filesystem not supported by this committer\");\n    }\n    LOG.info(\"Using Committer {} for {} created by {}\",\n        outputCommitter,\n        outputPath,\n        this);\n    return outputCommitter;\n  }\n\n  /**\n   * Get the destination filesystem, returning null if there is none.\n   * Code using this must explicitly or implicitly look for a null value\n   * in the response.\n   * @param outputPath output path\n   * @param context job/task context\n   * @return the destination filesystem, if it can be determined\n   * @throws IOException if the FS cannot be instantiated\n   */","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java#L33-L69","documentation":"AbstractS3ACommitterFactory is bound through mapreduce.outputcommitter.factory.scheme.s3a (or a broader factory binding) and only accepts S3AFileSystem destinations: createOutputCommitter checks fs instanceof S3AFileSystem and otherwise throws PathCommitException 'Filesystem not supported by this committer'. The S3A committers stage data via S3 multipart uploads, so they cannot serve other filesystems.","triggerScenarios":"The output path of a job resolved through this factory lands on a non-S3A filesystem: local file tests (file://), HDFS, s3n/s3 (legacy), or viewfs; typically because the factory was bound globally instead of per-scheme.","commonSituations":"Setting a global mapreduce.outputcommitter.factory binding (or spark.sql.sources.outputCommitterClass in older setups) to gain S3A committers, then running a job whose output is local/HDFS; unit tests with output on LocalFileSystem inheriting the production committer config.","solutions":["Bind the S3A committer factory per scheme only: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory, leaving other schemes on their default committers","Remove any global factory override so non-s3a paths use the classic FileOutputCommitter","Point job outputs that must use S3A committers at s3a:// URIs (not legacy s3n:// or viewfs-wrapped paths)"],"exampleFix":"<!-- before: global factory binding breaks non-S3A outputs -->\n<property><name>mapreduce.outputcommitter.factory.class</name>\n  <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value></property>\n\n<!-- after: bind per scheme -->\n<property><name>mapreduce.outputcommitter.factory.scheme.s3a</name>\n  <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value></property>","handlingStrategy":"type-guard","validationCode":"FileSystem fs = outputPath.getFileSystem(conf);\nif (!(fs instanceof S3AFileSystem)) {\n  conf.unset(\"mapreduce.outputcommitter.factory.scheme.\" + outputPath.toUri().getScheme());\n  LOG.warn(\"Output {} on {} cannot use S3A committers\", outputPath, fs.getScheme());\n}","typeGuard":"static boolean outputSupportsS3ACommitter(Path out, Configuration conf) throws IOException {\n  return out.getFileSystem(conf) instanceof S3AFileSystem;\n}","tryCatchPattern":null,"preventionTips":["Bind committer factories per scheme (mapreduce.outputcommitter.factory.scheme.s3a), never globally","Guard test/local outputs from inheriting production committer config","Check the output filesystem scheme before selecting a committer in job builders"],"tags":["aws","s3a","committer","filesystem","scheme","configuration"],"backgroundTag":"filesystem-scheme-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}