{"record":{"id":"ec52db240fdcd67b","repo":"apache/hadoop","slug":"e-bad-path","errorCode":"E_BAD_PATH","errorMessage":"Path does not represent a magic-commit path","messagePattern":"Path does not represent a magic-commit path","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java","lineNumber":58,"sourceCode":" */\npublic final class CommitUtils {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(CommitUtils.class);\n\n  private CommitUtils() {\n  }\n\n  /**\n   * Verify that the path is a magic one.\n   * @param fs filesystem\n   * @param path path\n   * @throws PathCommitException if the path isn't a magic commit path\n   */\n  public static void verifyIsMagicCommitPath(S3AFileSystem fs,\n      Path path) throws PathCommitException {\n    verifyIsMagicCommitFS(fs);\n    if (!fs.isMagicCommitPath(path)) {\n      throw new PathCommitException(path, E_BAD_PATH);\n    }\n  }\n\n  /**\n   * Verify that an S3A FS instance is a magic commit FS.\n   * @param fs filesystem\n   * @throws PathCommitException if the FS isn't a magic commit FS.\n   */\n  public static void verifyIsMagicCommitFS(S3AFileSystem fs)\n      throws PathCommitException {\n    if (!fs.isMagicCommitEnabled()) {\n      // dump out details to console for support diagnostics\n      String fsUri = fs.getUri().toString();\n      LOG.error(\"{}: {}:\\n{}\", E_NORMAL_FS, fsUri, fs);\n      // then fail\n      throw new PathCommitException(fsUri, E_NORMAL_FS);\n    }\n  }","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java#L40-L76","documentation":"Thrown by CommitUtils.verifyIsMagicCommitPath when a path that is about to take part in a magic commit does not contain the hidden '__magic_job-*' directory element. The S3A magic committer has tasks write into that magic directory so the final publish is a metadata-only rename inside S3. This check runs after verifyIsMagicCommitFS succeeds, so the filesystem is magic-enabled but the specific path is not a magic path.","triggerScenarios":"Calling verifyIsMagicCommitPath (directly or through S3AFileSystem rename/commit code paths guarded by it) with a path for which fs.isMagicCommitPath(path) returns false, e.g. 's3a://bucket/output/part-00000' with no __magic_job-<jobId> element anywhere in the path.","commonSituations":"A job runs with fs.s3a.committer.name=magic but application code writes straight to the final output path instead of the committer's task attempt directory; the magic directory was deleted or renamed by a concurrent cleanup process mid-job; a custom OutputCommitter or InputFormat constructs destination paths by hand and bypasses the magic layout.","solutions":["Write task output only through the committer-provided task attempt path so the __magic_job-<jobId> element is present (e.g. s3a://bucket/output/__magic_job-job_xxx/_temporary/attempt_xxx/...)","Pre-check with fs.isMagicCommitPath(path) and fail with an explicit message before invoking any commit operation","List the output directory (s3a://bucket/output/__magic_job-*) to confirm the magic directory still exists and was not removed by another job or lifecycle rule","If you do not need magic commits, switch to the directory, partitioned or staging committer via fs.s3a.committer.name"],"exampleFix":"// before: writing directly to the final destination\nFileSystem fs = path.getFileSystem(conf);\nfs.create(new Path(\"s3a://bucket/output/part-00000\"));\n\n// after: write inside the magic task-attempt directory the committer created\nPath attemptPath = context.getTaskAttemptPath(workDir);\nfs.create(new Path(attemptPath, \"part-00000\"));","handlingStrategy":"validation","validationCode":"S3AFileSystem s3a = (S3AFileSystem) fs;\nif (!s3a.isMagicCommitPath(path)) {\n  // not a __magic_job-* path: handle as a regular path instead of committing\n  LOG.warn(\"Skipping magic commit for non-magic path {}\", path);\n}","typeGuard":null,"tryCatchPattern":"try {\n  CommitUtils.verifyIsMagicCommitPath(s3aFs, path);\n} catch (PathCommitException e) {\n  // fail the task loudly; re-running requires writing into the magic attempt dir\n  throw new IOException(\"Not a magic commit path: \" + path, e);\n}","preventionTips":["Always derive output paths from the committer's task attempt path rather than building s3a URLs by string concatenation","Assert fs.isMagicCommitPath(path) in unit tests for any custom committer subclass that manipulates paths","Keep the __magic_job-* directory out of lifecycle/cleanup jobs until job commit finishes"],"tags":["s3a","magic-committer","path-validation","hadoop"],"backgroundTag":"path-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}