apache/hadoop · error · IllegalArgumentException

E_NO_MAGIC_PATH_ELEMENT

E_NO_MAGIC_PATH_ELEMENT

Error message

No __magic_job- element in path

What it means

Unchecked IllegalArgumentException from MagicCommitPaths.magicElementIndex when no element of the supplied path element list starts with '__magic_job-'. This utility backs the magic-committer path arithmetic (finding the magic element, splitting parents/children, extracting the job id) and assumes the caller already knows the path is a magic path. When the magic element is absent there is no meaningful index to return, so it throws rather than returning -1 or Optional.

Solutions

  1. Gate every call with S3AFileSystem.isMagicCommitPath(path) or CommitUtils.verifyIsMagicCommitPath before using MagicCommitPaths helpers
  2. Scan the element list yourself for a component starting with '__magic_job-' and skip the magic handling when absent
  3. Because this is an unchecked exception, wrap committer-adjacent code that touches paths in a boundary that converts it into a reported job error

Example fix

// before
List<String> elements = MagicCommitPaths.pathToElements(path);
int i = MagicCommitPaths.magicElementIndex(elements); // throws for non-magic paths

// after
if (!fs.isMagicCommitPath(path)) {
  // regular path handling
} else {
  int i = MagicCommitPaths.magicElementIndex(MagicCommitPaths.pathToElements(path));
}
Defensive patterns

Strategy: validation

Validate before calling

List<String> elements = MagicCommitPaths.pathToElements(path);
boolean hasMagic = elements.stream()
    .anyMatch(e -> e.startsWith(CommitConstants.MAGIC_PATH_PREFIX));
if (!hasMagic) {
  // plain path -- take the non-magic code route
}

Try / catch

try {
  int idx = MagicCommitPaths.magicElementIndex(elements);
} catch (IllegalArgumentException e) {
  // unchecked: guard at the API boundary so it cannot escape as a raw runtime error
  throw new IOException("Path has no magic element: " + elements, e);
}

Prevention

When it happens

Trigger: Calling magicElementIndex or the methods built on it (magicPathParents, magicPathChildren, pathToJobId) with the element list of a path like s3a://bucket/output/part-00000 that contains no __magic_job-* element.

Common situations: Calling MagicCommitPaths helpers on arbitrary user paths without first checking isMagicCommitPath; refactoring that routes non-magic paths into magic-path code; paths constructed from strings that lost the magic element during joining or normalization.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/304c9f86a307d602. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/MagicCommitPaths.java:110

    return elements.contains(BASE);
  }

  /**
   * Get the index of the magic path element.
   * @param elements full path element list
   * @return the index.
   * @throws IllegalArgumentException if there is no magic element
   */
  public static int magicElementIndex(List<String> elements) {
    Optional<Integer> index = IntStream.range(0, elements.size())
        .filter(i -> elements.get(i).startsWith(MAGIC_PATH_PREFIX))
        .boxed()
        .findFirst();

    if (index.isPresent()) {
      return index.get();
    } else {
      throw new IllegalArgumentException(E_NO_MAGIC_PATH_ELEMENT);
    }
  }

  /**
   * Get the parent path elements of the magic path.
   * The list may be immutable or may be a view of the underlying list.
   * Both the parameter list and the returned list MUST NOT be modified.
   * @param elements full path element list
   * @return the parent elements; may be empty
   */
  public static List<String> magicPathParents(List<String> elements) {
    return elements.subList(0, magicElementIndex(elements));
  }

  /**
   * Get the child path elements under the magic path.
   * The list may be immutable or may be a view of the underlying list.
   * Both the parameter list and the returned list MUST NOT be modified.

View on GitHub (pinned to 2add963021)