{"record":{"id":"ffec94ad5de6cc45","repo":"apache/hadoop","slug":"this-committer-does-not-work-with-the-filesystem-o","errorCode":null,"errorMessage":"This committer does not work with the filesystem of type {scheme}","messagePattern":"This committer does not work with the filesystem of type (.+?)","errorType":"validation","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/ManifestCommitterFactory.java","lineNumber":51,"sourceCode":" * This is the committer factory to register as the source of committers\n * for the job/filesystem schema.\n */\n@InterfaceAudience.Public\n@InterfaceStability.Evolving\npublic class ManifestCommitterFactory extends PathOutputCommitterFactory {\n\n  /**\n   * Name of this factory.\n   */\n  public static final String NAME = ManifestCommitterFactory.class.getName();\n\n  @Override\n  public ManifestCommitter createOutputCommitter(final Path outputPath,\n      final TaskAttemptContext context) throws IOException {\n    // safety check. S3A does not support this, so fail fast.\n    final String scheme = outputPath.toUri().getScheme();\n    if (UNSUPPORTED_FS_SCHEMAS.contains(scheme)) {\n      throw new PathIOException(outputPath.toString(),\n          \"This committer does not work with the filesystem of type \" + scheme);\n    }\n    return new ManifestCommitter(outputPath, context);\n  }\n\n}\n","sourceCodeStart":33,"sourceCodeEnd":58,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/ManifestCommitterFactory.java#L33-L58","documentation":"ManifestCommitterFactory.createOutputCommitter() fail-fast checks the output path's URI scheme against a hard-coded blocklist (InternalConstants.UNSUPPORTED_FS_SCHEMAS = {\"s3a\", \"wasb\"}) and throws PathIOException('This committer does not work with the filesystem of type <scheme>'). The manifest committer's rename-based algorithm is unsafe on S3A and WASB, so binding it there is rejected before any work is written.","triggerScenarios":"Output path starting with s3a:// or wasb:// while the manifest committer factory is bound for that scheme (mapreduce.outputcommitter.factory.scheme.s3a=...ManifestCommitterFactory) or globally, so ManifestCommitterFactory.createOutputCommitter receives the S3A/WASB output path.","commonSituations":"Setting the manifest committer as the default factory for all schemes to 'simplify' config; migrating from HDFS/ABFS where the manifest committer works onto S3; leftover site config after moving workloads to object storage.","solutions":["For S3A use the S3A committers instead: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.BindingCommitterFactory plus fs.s3a.committer.name=magic (or staging/directory).","Remove the manifest committer binding for s3a/wasb so PathOutputCommitterFactory falls back to FileOutputCommitter.","If the manifest committer is intended, point the output at a supported filesystem (hdfs://, abfs://, gs://) rather than S3A/WASB.","Check for a global default binding (mapreduce.outputcommitter.factory.classname or .scheme.*) in core-site/mapred-site that sweeps S3A paths into the manifest committer."],"exampleFix":"// before: manifest committer bound to S3A\nconf.set(\"mapreduce.outputcommitter.factory.scheme.s3a\",\n    \"org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory\");\n\n// after: S3A paths use the S3A committer binding\nconf.set(\"mapreduce.outputcommitter.factory.scheme.s3a\",\n    \"org.apache.hadoop.fs.s3a.commit.BindingCommitterFactory\");\nconf.set(\"fs.s3a.committer.name\", \"magic\");","handlingStrategy":"validation","validationCode":"// client-side scheme gate before submit\nSet<String> blocked = new HashSet<>(Arrays.asList(\"s3a\", \"wasb\"));\nString scheme = outputPath.toUri().getScheme();\nif (scheme != null && blocked.contains(scheme)\n    && conf.get(\"mapreduce.outputcommitter.factory.scheme.\" + scheme, \"\").contains(\"ManifestCommitter\")) {\n  throw new IllegalArgumentException(\n      \"Manifest committer cannot be used with \" + scheme + \" output: \" + outputPath);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Bind committer factories per scheme explicitly; never set the manifest factory as a global default.","For S3A prefer the S3A committers via BindingCommitterFactory and fs.s3a.committer.name.","Assert the output scheme against the committer's supported list in job scaffolding."],"tags":["hadoop","s3a","wasb","manifest-committer","filesystem-scheme","configuration"],"backgroundTag":"filesystem-scheme-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}