apache/hadoop · error · PathCommitException

E_WRONG_FS

E_WRONG_FS

Error message

Output path is not on an S3A Filesystem

What it means

Thrown by CommitUtils.verifyIsS3AFS when the filesystem bound to an output path is not an S3AFileSystem instance. S3A committers can only work against s3a:// URLs because they rely on S3A-specific commit machinery; any other filesystem implementation (s3n, s3b, hdfs, local) fails this instanceof check. The method exists to fail fast before committer setup touches S3A-only APIs.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java:88

      // dump out details to console for support diagnostics
      String fsUri = fs.getUri().toString();
      LOG.error("{}: {}:\n{}", E_NORMAL_FS, fsUri, fs);
      // then fail
      throw new PathCommitException(fsUri, E_NORMAL_FS);
    }
  }

  /**
   * Verify that an FS is an S3A FS.
   * @param fs filesystem
   * @param path path to to use in exception
   * @return the typecast FS.
   * @throws PathCommitException if the FS is not an S3A FS.
   */
  public static S3AFileSystem verifyIsS3AFS(FileSystem fs, Path path)
      throws PathCommitException {
    if (!(fs instanceof S3AFileSystem)) {
      throw new PathCommitException(path, E_WRONG_FS);
    }
    return (S3AFileSystem) fs;
  }

  /**
   * Get the S3A FS of a path.
   * @param path path to examine
   * @param conf config
   * @param magicCommitRequired is magic complete required in the FS?
   * @return the filesystem
   * @throws PathCommitException output path isn't to an S3A FS instance, or
   * if {@code magicCommitRequired} is set, if doesn't support these commits.
   * @throws IOException failure to instantiate the FS
   */
  public static S3AFileSystem getS3AFileSystem(Path path,
      Configuration conf,
      boolean magicCommitRequired)
      throws PathCommitException, IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Change the output path to use the s3a:// scheme, e.g. s3a://bucket/output
2. If the path is relative or scheme-less, qualify it against an explicit s3a:// base path before passing it to the committer
3. Confirm no stray fs.s3a.impl aliasing maps s3a:// to another FileSystem class in core-site.xml
4. If the output is genuinely not on S3, use the standard FileOutputCommitter instead of an S3A committer

Example fix

// before
JobConf conf = new JobConf();
FileOutputFormat.setOutputPath(job, new Path("s3n://bucket/output"));

// after
FileOutputFormat.setOutputPath(job, new Path("s3a://bucket/output"));

Path qualified = path.makeQualified(fs.getUri(), fs.getWorkingDirectory());
if (!"s3a".equals(qualified.toUri().getScheme())) {
  throw new IOException("Output path must use s3a://, got: " + qualified);
}

public static boolean isS3AFileSystem(FileSystem fs) {
  return fs instanceof S3AFileSystem;
}

try {
  S3AFileSystem s3a = CommitUtils.verifyIsS3AFS(fs, outputPath);
} catch (PathCommitException e) {
  // scheme problem: rewrite the output path with the s3a:// scheme and retry once
  LOG.error("Output path not on S3A: {}", outputPath, e);
}

When it happens

Trigger: Calling verifyIsS3AFS or getOutputFS (which delegates to it) when mapreduce.output.fileoutputformat.outputdir (or Spark's spark.sql.sources.output.pathCommitterClass path) uses a non-s3a scheme, or when FileSystem.get returned a cached instance for an alias like s3://.

Common situations: The job's output path is written as s3n://bucket/out or s3b://bucket/out (legacy connectors); output goes to hdfs:// or file:// while the code assumes S3; the output path has no scheme at all and resolves against the default filesystem (fs.defaultFS).

Related errors

• This committer does not work with the filesystem of type {sc
• Unsupported block buffer "{name}"
• Multipart IO request {sdkRequest} rejected {header}
• Multipart uploads are disabled for the FileSystem, the commi
• Unable to recover task %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f0d328f942b7fab3. Report an issue: GitHub.