apache/hadoop · error · PathCommitException

E_BAD_PATH

E_BAD_PATH

Error message

Path does not represent a magic-commit path

What it means

Thrown by CommitUtils.verifyIsMagicCommitPath when a path that is about to take part in a magic commit does not contain the hidden '__magic_job-*' directory element. The S3A magic committer has tasks write into that magic directory so the final publish is a metadata-only rename inside S3. This check runs after verifyIsMagicCommitFS succeeds, so the filesystem is magic-enabled but the specific path is not a magic path.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java:58

 */
public final class CommitUtils {
  private static final Logger LOG =
      LoggerFactory.getLogger(CommitUtils.class);

  private CommitUtils() {
  }

  /**
   * Verify that the path is a magic one.
   * @param fs filesystem
   * @param path path
   * @throws PathCommitException if the path isn't a magic commit path
   */
  public static void verifyIsMagicCommitPath(S3AFileSystem fs,
      Path path) throws PathCommitException {
    verifyIsMagicCommitFS(fs);
    if (!fs.isMagicCommitPath(path)) {
      throw new PathCommitException(path, E_BAD_PATH);
    }
  }

  /**
   * Verify that an S3A FS instance is a magic commit FS.
   * @param fs filesystem
   * @throws PathCommitException if the FS isn't a magic commit FS.
   */
  public static void verifyIsMagicCommitFS(S3AFileSystem fs)
      throws PathCommitException {
    if (!fs.isMagicCommitEnabled()) {
      // dump out details to console for support diagnostics
      String fsUri = fs.getUri().toString();
      LOG.error("{}: {}:\n{}", E_NORMAL_FS, fsUri, fs);
      // then fail
      throw new PathCommitException(fsUri, E_NORMAL_FS);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Write task output only through the committer-provided task attempt path so the __magic_job-<jobId> element is present (e.g. s3a://bucket/output/__magic_job-job_xxx/_temporary/attempt_xxx/...)
2. Pre-check with fs.isMagicCommitPath(path) and fail with an explicit message before invoking any commit operation
3. List the output directory (s3a://bucket/output/__magic_job-*) to confirm the magic directory still exists and was not removed by another job or lifecycle rule
4. If you do not need magic commits, switch to the directory, partitioned or staging committer via fs.s3a.committer.name

Example fix

// before: writing directly to the final destination
FileSystem fs = path.getFileSystem(conf);
fs.create(new Path("s3a://bucket/output/part-00000"));

// after: write inside the magic task-attempt directory the committer created
Path attemptPath = context.getTaskAttemptPath(workDir);
fs.create(new Path(attemptPath, "part-00000"));

S3AFileSystem s3a = (S3AFileSystem) fs;
if (!s3a.isMagicCommitPath(path)) {
  // not a __magic_job-* path: handle as a regular path instead of committing
  LOG.warn("Skipping magic commit for non-magic path {}", path);
}

try {
  CommitUtils.verifyIsMagicCommitPath(s3aFs, path);
} catch (PathCommitException e) {
  // fail the task loudly; re-running requires writing into the magic attempt dir
  throw new IOException("Not a magic commit path: " + path, e);
}

When it happens

Trigger: Calling verifyIsMagicCommitPath (directly or through S3AFileSystem rename/commit code paths guarded by it) with a path for which fs.isMagicCommitPath(path) returns false, e.g. 's3a://bucket/output/part-00000' with no __magic_job-<jobId> element anywhere in the path.

Common situations: A job runs with fs.s3a.committer.name=magic but application code writes straight to the final output path instead of the committer's task attempt directory; the magic directory was deleted or renamed by a concurrent cleanup process mid-job; a custom OutputCommitter or InputFormat constructs destination paths by hand and bypasses the magic layout.

Related errors

• E_NORMAL_FS
• E_NO_MAGIC_PATH_ELEMENT
• Rename to subdir is forbidden
• Wrong FS {} -expected {}
• Cannot rename {absoluteSrc} under itself : {absoluteDst}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ec52db240fdcd67b. Report an issue: GitHub.