apache/hadoop · error · IOException

Failed to create a URI (URISyntaxException) for the remote p

Error message

Failed to create a URI (URISyntaxException) for the remote path {}. This was based on the libjar parameter: {}

What it means

The libjars counterpart of the defensive post-copy check: after copyRemoteFiles places the jar in the staging libjars directory, getPathURI(newPath, tmpURI.getFragment()) rebuilds the final URI. The URI constructor throws URISyntaxException when the fragment taken from your original -libjars entry is malformed, and the code wraps it as IOException('Failed to create a URI (URISyntaxException) ... based on the libjar parameter'). The comment says it 'should not throw', so this indicates a bad '#alias' fragment, not a copy failure.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:320

        boolean uploadToSharedCache = false;
        boolean fromSharedCache = false;
        if (scConfig.isSharedCacheLibjarsEnabled()) {
          newURI = useSharedCache(tmpURI, tmp.getName(), statCache, conf, true);
          if (newURI == null) {
            uploadToSharedCache = true;
          } else {
            fromSharedCache = true;
          }
        }

        if (newURI == null) {
          Path newPath =
              copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
          try {
            newURI = getPathURI(newPath, tmpURI.getFragment());
          } catch (URISyntaxException ue) {
            // should not throw a uri exception
            throw new IOException(
                "Failed to create a URI (URISyntaxException) for the"
                    + " remote path " + newPath
                    + ". This was based on the libjar parameter: " + tmpjars,
                ue);
          }
        }

        if (!foundFragment) {
          // We do not count shared cache paths containing fragments as a
          // "foundFragment." This is because these resources are not in the
          // staging directory and will be added to the distributed cache
          // separately.
          foundFragment = (newURI.getFragment() != null) && !fromSharedCache;
        }
        Job.addFileToClassPath(new Path(newURI.getPath()), conf, jtFs, false);
        if (fromSharedCache) {
          // We simply add this URI to the distributed cache. It will not come
          // from the staging directory (it is in the shared cache), so we

View on GitHub (pinned to 2add963021)

Solutions

1. Use identifier-safe aliases: 'a.jar#depjar' instead of 'a.jar#dep jar'
2. Percent-encode spaces in aliases as %20
3. Omit the fragment when a custom name is unnecessary — the file name is used as the link name
4. Pre-check: new URI(null, null, null, fragment) must not throw before you submit

Example fix

# before
hadoop jar app.jar Driver -libjars "hdfs://nn/libs/guava.jar#guava lib" in out
# IOException: Failed to create a URI (URISyntaxException) ... libjar parameter

# after
hadoop jar app.jar Driver -libjars "hdfs://nn/libs/guava.jar#guava-lib" in out

// reject libjar entries with unsafe '#alias' fragments
for (String jar : StringUtils.getStrings(conf.get("tmpjars"))) {
  int i = jar.indexOf('#');
  if (i >= 0) {
    try { new URI(null, null, null, jar.substring(i + 1)); }
    catch (URISyntaxException e) {
      throw new IllegalArgumentException("Bad alias in -libjars entry: " + jar, e);
    }
  }
}

static boolean hasSafeFragment(String entry) {
  int i = entry.indexOf('#');
  if (i < 0) return true;
  try { new URI(null, null, null, entry.substring(i + 1)); return true; }
  catch (URISyntaxException e) { return false; }
}

When it happens

Trigger: -libjars 'hdfs://nn/l/a.jar#dep jar' (space in the fragment alias); fragments with reserved characters ('?', backslash, quotes); an entry whose fragment came from an unescaped shell variable.

Common situations: Adding mnemonic link names to dependency jars; CI pipelines interpolating version strings containing '+' or spaces into the alias; documentation examples that show '#name' without restricting the character set.

Related errors

• Resource name must be relative
• Invalid path URI: {} - cannot contain both a URI fragment an
• Invalid path URI: {} - cannot contain both a URI fragment an
• Failed to create a URI (URISyntaxException) for the remote p
• Failed to create a URI (URISyntaxException) for the remote p

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2f86a5e8b31f2451. Report an issue: GitHub.