apache/hadoop · error · IllegalArgumentException

E_NO_MAGIC_PATH_ELEMENT

E_NO_MAGIC_PATH_ELEMENT

Error message

No __magic_job- element in path

What it means

Unchecked IllegalArgumentException from MagicCommitPaths.magicElementIndex when no element of the supplied path element list starts with '__magic_job-'. This utility backs the magic-committer path arithmetic (finding the magic element, splitting parents/children, extracting the job id) and assumes the caller already knows the path is a magic path. When the magic element is absent there is no meaningful index to return, so it throws rather than returning -1 or Optional.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/MagicCommitPaths.java:110

    return elements.contains(BASE);
  }

  /**
   * Get the index of the magic path element.
   * @param elements full path element list
   * @return the index.
   * @throws IllegalArgumentException if there is no magic element
   */
  public static int magicElementIndex(List<String> elements) {
    Optional<Integer> index = IntStream.range(0, elements.size())
        .filter(i -> elements.get(i).startsWith(MAGIC_PATH_PREFIX))
        .boxed()
        .findFirst();

    if (index.isPresent()) {
      return index.get();
    } else {
      throw new IllegalArgumentException(E_NO_MAGIC_PATH_ELEMENT);
    }
  }

  /**
   * Get the parent path elements of the magic path.
   * The list may be immutable or may be a view of the underlying list.
   * Both the parameter list and the returned list MUST NOT be modified.
   * @param elements full path element list
   * @return the parent elements; may be empty
   */
  public static List<String> magicPathParents(List<String> elements) {
    return elements.subList(0, magicElementIndex(elements));
  }

  /**
   * Get the child path elements under the magic path.
   * The list may be immutable or may be a view of the underlying list.
   * Both the parameter list and the returned list MUST NOT be modified.

View on GitHub (pinned to 2add963021)

Solutions

1. Gate every call with S3AFileSystem.isMagicCommitPath(path) or CommitUtils.verifyIsMagicCommitPath before using MagicCommitPaths helpers
2. Scan the element list yourself for a component starting with '__magic_job-' and skip the magic handling when absent
3. Because this is an unchecked exception, wrap committer-adjacent code that touches paths in a boundary that converts it into a reported job error

Example fix

// before
List<String> elements = MagicCommitPaths.pathToElements(path);
int i = MagicCommitPaths.magicElementIndex(elements); // throws for non-magic paths

// after
if (!fs.isMagicCommitPath(path)) {
  // regular path handling
} else {
  int i = MagicCommitPaths.magicElementIndex(MagicCommitPaths.pathToElements(path));
}

List<String> elements = MagicCommitPaths.pathToElements(path);
boolean hasMagic = elements.stream()
    .anyMatch(e -> e.startsWith(CommitConstants.MAGIC_PATH_PREFIX));
if (!hasMagic) {
  // plain path -- take the non-magic code route
}

try {
  int idx = MagicCommitPaths.magicElementIndex(elements);
} catch (IllegalArgumentException e) {
  // unchecked: guard at the API boundary so it cannot escape as a raw runtime error
  throw new IOException("Path has no magic element: " + elements, e);
}

When it happens

Trigger: Calling magicElementIndex or the methods built on it (magicPathParents, magicPathChildren, pathToJobId) with the element list of a path like s3a://bucket/output/part-00000 that contains no __magic_job-* element.

Common situations: Calling MagicCommitPaths helpers on arbitrary user paths without first checking isMagicCommitPath; refactoring that routes non-magic paths into magic-path code; paths constructed from strings that lost the magic element during joining or normalization.

Related errors

• E_BAD_PATH
• GCS path must not have consecutive '/' characters: '%s'
• [{0}] must be an absolute path [{1}]
• E_NORMAL_FS
• Invalid client side encryption algorithm. Only CSE-KMS and C

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/304c9f86a307d602. Report an issue: GitHub.