apache/hadoop · error · PathIOException

This committer does not work with the filesystem of type {sc

Error message

This committer does not work with the filesystem of type {scheme}

What it means

ManifestCommitterFactory.createOutputCommitter() fail-fast checks the output path's URI scheme against a hard-coded blocklist (InternalConstants.UNSUPPORTED_FS_SCHEMAS = {"s3a", "wasb"}) and throws PathIOException('This committer does not work with the filesystem of type <scheme>'). The manifest committer's rename-based algorithm is unsafe on S3A and WASB, so binding it there is rejected before any work is written.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/ManifestCommitterFactory.java:51

 * This is the committer factory to register as the source of committers
 * for the job/filesystem schema.
 */
@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ManifestCommitterFactory extends PathOutputCommitterFactory {

  /**
   * Name of this factory.
   */
  public static final String NAME = ManifestCommitterFactory.class.getName();

  @Override
  public ManifestCommitter createOutputCommitter(final Path outputPath,
      final TaskAttemptContext context) throws IOException {
    // safety check. S3A does not support this, so fail fast.
    final String scheme = outputPath.toUri().getScheme();
    if (UNSUPPORTED_FS_SCHEMAS.contains(scheme)) {
      throw new PathIOException(outputPath.toString(),
          "This committer does not work with the filesystem of type " + scheme);
    }
    return new ManifestCommitter(outputPath, context);
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. For S3A use the S3A committers instead: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.BindingCommitterFactory plus fs.s3a.committer.name=magic (or staging/directory).
2. Remove the manifest committer binding for s3a/wasb so PathOutputCommitterFactory falls back to FileOutputCommitter.
3. If the manifest committer is intended, point the output at a supported filesystem (hdfs://, abfs://, gs://) rather than S3A/WASB.
4. Check for a global default binding (mapreduce.outputcommitter.factory.classname or .scheme.*) in core-site/mapred-site that sweeps S3A paths into the manifest committer.

Example fix

// before: manifest committer bound to S3A
conf.set("mapreduce.outputcommitter.factory.scheme.s3a",
    "org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory");

// after: S3A paths use the S3A committer binding
conf.set("mapreduce.outputcommitter.factory.scheme.s3a",
    "org.apache.hadoop.fs.s3a.commit.BindingCommitterFactory");
conf.set("fs.s3a.committer.name", "magic");

// client-side scheme gate before submit
Set<String> blocked = new HashSet<>(Arrays.asList("s3a", "wasb"));
String scheme = outputPath.toUri().getScheme();
if (scheme != null && blocked.contains(scheme)
    && conf.get("mapreduce.outputcommitter.factory.scheme." + scheme, "").contains("ManifestCommitter")) {
  throw new IllegalArgumentException(
      "Manifest committer cannot be used with " + scheme + " output: " + outputPath);
}

When it happens

Trigger: Output path starting with s3a:// or wasb:// while the manifest committer factory is bound for that scheme (mapreduce.outputcommitter.factory.scheme.s3a=...ManifestCommitterFactory) or globally, so ManifestCommitterFactory.createOutputCommitter receives the S3A/WASB output path.

Common situations: Setting the manifest committer as the default factory for all schemes to 'simplify' config; migrating from HDFS/ABFS where the manifest committer works onto S3; leftover site config after moving workloads to object storage.

Related errors

• Unsupported block buffer "{name}"
• Failed to create Store Operations from configuration option
• E_NORMAL_FS
• E_WRONG_FS
• No more entry in " + f

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ffec94ad5de6cc45. Report an issue: GitHub.