apache/hadoop · error · IllegalArgumentException

Invalid path URI: {} - cannot contain both a URI fragment an

Error message

Invalid path URI: {} - cannot contain both a URI fragment and a wildcard

What it means

LocalResourceBuilder.createLocalResources rejects cache URIs that combine a wildcard path ('dir/*') with a URI fragment ('#name'): the wildcard case resolves the parent directory and derives link names from actual file names, while the fragment names exactly one link — the two mechanisms are mutually exclusive, so the builder throws IllegalArgumentException. The fragment is only honored for non-wildcard entries.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java:135

        // If there's no wildcard, try using the fragment for the link
        if (linkName == null) {
          linkName = u.getFragment();

          // Because we don't know what's in the fragment, we have to handle
          // it with care.
          if (linkName != null) {
            Path linkPath = new Path(linkName);

            if (linkPath.isAbsolute()) {
              throw new IllegalArgumentException("Resource name must be "
                  + "relative");
            }

            linkName = linkPath.toUri().getPath();
          }
        } else if (u.getFragment() != null) {
          throw new IllegalArgumentException("Invalid path URI: " + p +
              " - cannot contain both a URI fragment and a wildcard");
        }

        // If there's no wildcard or fragment, just link to the file name
        if (linkName == null) {
          linkName = p.getName();
        }

        LocalResource orig = localResources.get(linkName);
        if(orig != null && !orig.getResource().equals(URL.fromURI(p.toUri()))) {
          LOG.warn(getResourceDescription(orig.getType()) + orig.getResource()
              + " conflicts with " + getResourceDescription(type) + u);
          continue;
        }
        Boolean sharedCachePolicy = sharedCacheUploadPolicies.get(u.toString());
        sharedCachePolicy =
            sharedCachePolicy == null ? Boolean.FALSE : sharedCachePolicy;
        localResources.put(linkName, LocalResource.newInstance(URL.fromURI(p

View on GitHub (pinned to 2add963021)

Solutions

1. Use the wildcard alone ('hdfs://nn/jars/*') — each file gets its own link named after the file
2. Or point at the specific file with the fragment ('hdfs://nn/jars/one.jar#one.jar')
3. Strip fragments from wildcard entries in whatever generates your cache configuration

Example fix

# before
mapreduce.job.cache.files = hdfs://nn/jars/*#myjars   # throws

# after (choose one)
mapreduce.job.cache.files = hdfs://nn/jars/*            # wildcard: per-file default names
# or
mapreduce.job.cache.files = hdfs://nn/jars/one.jar#one.jar  # explicit file + fragment

static boolean isValidCacheUri(URI u) {
  Path p = new Path(u);
  boolean wildcard = p.getName().equals(DistributedCache.WILDCARD);
  return !(wildcard && u.getFragment() != null);
}
for (URI u : cacheUris) { if (!isValidCacheUri(u)) throw new IOException("wildcard+fragment: " + u); }

try {
  job.addCacheFile(u);
} catch (IllegalArgumentException e) {
  if (u.getFragment() != null && u.toString().contains("*")) {
    job.addCacheFile(UriBuilder.fromUri(u).fragment(null).build()); // drop fragment
  } else throw e;
}

When it happens

Trigger: A distributed-cache entry like 'hdfs://nn/jars/*#myjars' (or '#myjars.jar') set via job.addCacheFile, DistributedCache.addCacheFile, or the mapreduce.job.cache.files conf key.

Common situations: Trying to name a whole directory of jars with one symlink; copy-pasting a single-file fragment pattern onto a wildcard entry; generated job configs that concatenate a fragment onto every cache URI.

Related errors

• Invalid path URI: {} - cannot contain both a URI fragment an
• Resource name must be relative
• Failed to create a URI (URISyntaxException) for the remote p
• Failed to create a URI (URISyntaxException) for the remote p
• Failed to create a URI (URISyntaxException) for the remote p

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0c8b02b1ccab5d28. Report an issue: GitHub.