apache/hadoop · error · IOException

Invalid path URI: {} - cannot contain both a URI fragment an

Error message

Invalid path URI: {} - cannot contain both a URI fragment and a wildcard

What it means

While building LocalResources from cache URIs, MRApps resolves each path (resolving the parent when a '*' wildcard is present) and then applies the same mutual-exclusion rule as the submitter path: a URI may carry a wildcard or a fragment, not both, throwing IOException('Invalid path URI: ... cannot contain both a URI fragment and a wildcard'). With a wildcard, link names come from the resolved directory listing; a fragment would try to name one link for many files.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:320

      HashMap<Path, String> linkLookup = new HashMap<Path, String>();
      if (withLinks != null) {
        for (URI u: withLinks) {
          Path p = new Path(u);
          FileSystem remoteFS = p.getFileSystem(conf);
          String name = p.getName();
          String wildcard = null;

          // If the path is wildcarded, resolve its parent directory instead
          if (name.equals(DistributedCache.WILDCARD)) {
            wildcard = name;
            p = p.getParent();
          }

          p = remoteFS.resolvePath(p.makeQualified(remoteFS.getUri(),
              remoteFS.getWorkingDirectory()));

          if ((wildcard != null) && (u.getFragment() != null)) {
            throw new IOException("Invalid path URI: " + p + " - cannot "
                + "contain both a URI fragment and a wildcard");
          } else if (wildcard != null) {
            name = p.getName() + Path.SEPARATOR + wildcard;
          } else if (u.getFragment() != null) {
            name = u.getFragment();
          }

          // If it's not a JAR, add it to the link lookup.
          if (!StringUtils.toLowerCase(name).endsWith(".jar")) {
            String old = linkLookup.put(p, name);

            if ((old != null) && !name.equals(old)) {
              LOG.warn("The same path is included more than once "
                  + "with different links or wildcards: " + p + " [" +
                  name + ", " + old + "]");
            }
          }
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Remove the fragment from wildcard entries: 'hdfs://nn/libs/*' localizes every file with its own name
2. Or expand the wildcard yourself and list explicit files, each optionally with its own fragment
3. Sanitize cache URI configs to reject entries containing both '*' and '#' before submission

Example fix

# before
job.addCacheFile(new URI("hdfs://nn/libs/*#libs")); // throws IOException at setup

# after
job.addCacheFile(new URI("hdfs://nn/libs/*")); // every jar linked under its own name
// or list files explicitly if a specific link name is needed:
job.addCacheFile(new URI("hdfs://nn/libs/a.jar#a.jar"));

static boolean hasWildcardFragmentConflict(URI u) {
  Path p = new Path(u);
  return DistributedCache.WILDCARD.equals(p.getName()) && u.getFragment() != null;
}
// strip the fragment from wildcarded entries before conf reaches MR setup

try {
  MRApps.setupDistributedCache(conf, localResources);
} catch (IOException e) {
  if (e.getMessage().contains("fragment and a wildcard")) {
    // rewrite the offending cache URI without the fragment and retry setup once
  }
  throw e;
}

When it happens

Trigger: A job conf cache entry such as 'hdfs://nn/libs/*#libs' processed during job setup/AM localization; archives with the same combination ('hdfs://nn/arch/*#a').

Common situations: Config templates that append '#name' to every cache/archive path including wildcarded ones; user attempts to control the symlink name of a directory of jars; migration scripts combining two previously-separate patterns.

Related errors

• Invalid path URI: {} - cannot contain both a URI fragment an
• Resource name must be relative
• Failed to create a URI (URISyntaxException) for the remote p
• Failed to create a URI (URISyntaxException) for the remote p
• Failed to create a URI (URISyntaxException) for the remote p

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1d46216b8d3e277. Report an issue: GitHub.