apache/hadoop · error · IOException

Failed to create a URI (URISyntaxException) for the remote p

Error message

Failed to create a URI (URISyntaxException) for the remote path {}. This was based on the files parameter: {}

What it means

After copying a remote -files resource into the staging directory, JobResourceUploader rebuilds the final cache URI via getPathURI(newPath, tmpURI.getFragment()). The multi-argument URI constructor percent-encodes nothing and throws URISyntaxException when the fragment (the '#alias' part of your original entry) contains characters illegal in that position. The code comments this as 'should not throw' and wraps it in IOException, so hitting it means the original -files entry carried a malformed fragment.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:263

        }
        Path tmp = new Path(tmpURI);
        URI newURI = null;
        boolean uploadToSharedCache = false;
        if (scConfig.isSharedCacheFilesEnabled()) {
          newURI = useSharedCache(tmpURI, tmp.getName(), statCache, conf, true);
          if (newURI == null) {
            uploadToSharedCache = true;
          }
        }

        if (newURI == null) {
          Path newPath =
              copyRemoteFiles(filesDir, tmp, conf, submitReplication);
          try {
            newURI = getPathURI(newPath, tmpURI.getFragment());
          } catch (URISyntaxException ue) {
            // should not throw a uri exception
            throw new IOException(
                "Failed to create a URI (URISyntaxException) for the"
                    + " remote path " + newPath
                    + ". This was based on the files parameter: " + tmpFile,
                ue);
          }
        }

        job.addCacheFile(newURI);
        if (scConfig.isSharedCacheFilesEnabled()) {
          fileSCUploadPolicies.put(newURI.toString(), uploadToSharedCache);
        }
      }
    }
  }

  // Suppress warning for use of DistributedCache (it is everywhere).
  @SuppressWarnings("deprecation")
  @VisibleForTesting

View on GitHub (pinned to 2add963021)

Solutions

1. Keep fragment aliases to plain identifiers: letters, digits, dash, underscore, dot
2. Encode any space in the alias as %20: 'a.jar#my%20alias' (the fragment is preserved and decoded by the localization layer)
3. Drop the fragment entirely when you don't need a custom symlink name
4. Validate the fragment with new URI(null, null, null, fragment) before submitting

Example fix

# before
hadoop jar app.jar Driver -files "hdfs://nn/lib/a.jar#my alias" in out
# IOException: Failed to create a URI (URISyntaxException) ... based on the files parameter

# after
hadoop jar app.jar Driver -files "hdfs://nn/lib/a.jar#my-alias" in out

// validate the '#alias' fragment of each -files entry
static void checkFragment(String entry) throws URISyntaxException {
  String s = entry.substring(entry.indexOf('#') + 1);
  new URI(null, null, null, s); // throws if fragment is illegal
}

static boolean hasSafeFragment(String entry) {
  int i = entry.indexOf('#');
  if (i < 0) return true;
  try { new URI(null, null, null, entry.substring(i + 1)); return true; }
  catch (URISyntaxException e) { return false; }
}

When it happens

Trigger: -files 'hdfs://nn/lib/a.jar#my alias' (raw space in the symlink alias); fragments containing '?', backslashes, or other reserved characters; aliases copied from shell history containing quotes or trailing punctuation.

Common situations: Users adding friendly distributed-cache link names with spaces or special characters; generated command lines that interpolate unescaped variables into the '#alias' portion; copy-paste of paths that already contain a '#'.

Related errors

• Resource name must be relative
• Invalid path URI: {} - cannot contain both a URI fragment an
• Invalid path URI: {} - cannot contain both a URI fragment an
• Error parsing files argument. Argument must be a valid URI:
• Failed to create a URI (URISyntaxException) for the remote p

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c85f7847a0c24a97. Report an issue: GitHub.