apache/hadoop · error · PathCommitException

E_NORMAL_FS

E_NORMAL_FS

Error message

Filesystem does not have support for 'magic' committer enabled in configuration option fs.s3a.committer.magic.enabled

What it means

Thrown by CommitUtils.verifyIsMagicCommitFS when the S3AFileSystem instance was created without magic-commit support (fs.isMagicCommitEnabled() is false). Before throwing, the code LOG.error's the error code, the filesystem URI and the filesystem's own toString() diagnostics, so the log already contains the support details. It means the 'fs.s3a.committer.magic.enabled' option was not true in the configuration used when the filesystem instance was constructed.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java:74

    verifyIsMagicCommitFS(fs);
    if (!fs.isMagicCommitPath(path)) {
      throw new PathCommitException(path, E_BAD_PATH);
    }
  }

  /**
   * Verify that an S3A FS instance is a magic commit FS.
   * @param fs filesystem
   * @throws PathCommitException if the FS isn't a magic commit FS.
   */
  public static void verifyIsMagicCommitFS(S3AFileSystem fs)
      throws PathCommitException {
    if (!fs.isMagicCommitEnabled()) {
      // dump out details to console for support diagnostics
      String fsUri = fs.getUri().toString();
      LOG.error("{}: {}:\n{}", E_NORMAL_FS, fsUri, fs);
      // then fail
      throw new PathCommitException(fsUri, E_NORMAL_FS);
    }
  }

  /**
   * Verify that an FS is an S3A FS.
   * @param fs filesystem
   * @param path path to to use in exception
   * @return the typecast FS.
   * @throws PathCommitException if the FS is not an S3A FS.
   */
  public static S3AFileSystem verifyIsS3AFS(FileSystem fs, Path path)
      throws PathCommitException {
    if (!(fs instanceof S3AFileSystem)) {
      throw new PathCommitException(path, E_WRONG_FS);
    }
    return (S3AFileSystem) fs;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.s3a.committer.magic.enabled=true together with fs.s3a.committer.name=magic in core-site.xml on every node that will create the S3AFileSystem (or pass both in the job Configuration)
2. Check the LOG.error line emitted just before the exception: it prints the FS URI and full filesystem state showing which config was actually in effect
3. Verify no per-bucket override (fs.s3a.bucket.<bucket>.*) disables the feature
4. If magic commit is not intended, stop calling the magic commit APIs and use another committer

Example fix

<!-- before -->
<property><name>fs.s3a.committer.name</name><value>magic</value></property>

<!-- after -->
<property><name>fs.s3a.committer.name</name><value>magic</value></property>
<property><name>fs.s3a.committer.magic.enabled</name><value>true</value></property>

FileSystem fs = path.getFileSystem(conf);
if (!(fs instanceof S3AFileSystem)
    || !((S3AFileSystem) fs).isMagicCommitEnabled()) {
  throw new IOException("Magic commits not enabled for " + path
      + " -- set fs.s3a.committer.magic.enabled=true");
}

try {
  CommitUtils.verifyIsMagicCommitFS(s3aFs);
} catch (PathCommitException e) {
  // read the LOG.error dump of the FS state emitted just before the throw
  throw new IOException("FS lacks magic commit support; fix core-site.xml", e);
}

When it happens

Trigger: Any code path that calls verifyIsMagicCommitFS or verifyIsMagicCommitPath against an S3AFileSystem whose configuration lacks fs.s3a.committer.magic.enabled=true, typically when initializing or using the magic committer.

Common situations: fs.s3a.committer.name=magic is set in the job config but fs.s3a.committer.magic.enabled=true is missing; the option is set only in the client's core-site.xml while the FileSystem is instantiated on nodes with a different core-site.xml; a typo in the option name; a per-bucket override (fs.s3a.bucket.<bucket>.committer.magic.enabled) is set to false and wins over the base key.

Related errors

• Unsupported block buffer "{name}"
• This committer does not work with the filesystem of type {sc
• E_BAD_PATH
• No more entry in " + f
• f + " is a directory"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b9655e76263edb54. Report an issue: GitHub.