apache/hadoop · error · IllegalArgumentException

Error processing URI

Error message

Error processing URI

What it means

In copyRemoteFiles, a source URI ending in '/' would make originalPath.getName() return an empty string, so the code strips the trailing slash by re-parsing the shortened string with new URI(...). If that shortened string is still syntactically invalid, the URISyntaxException is rethrown as IllegalArgumentException('Error processing URI'). Like its sibling checks, this is a defensive parse on input you supplied via -files/-libjars/-archives.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:693

    if (FileUtil.compareFs(remoteFs, jtFs)) {
      return originalPath;
    }

    boolean root = false;
    if (ROOT_PATH.equals(originalPath.toUri().getPath())) {
      // "/" needs special treatment
      root = true;
    } else {
      // If originalPath ends in a "/", then remove it so
      // that originalPath.getName() does not return an empty string
      String uriString = originalPath.toUri().toString();
      if (uriString.endsWith("/")) {
        try {
          URI strippedURI =
              new URI(uriString.substring(0, uriString.length() - 1));
          originalPath = new Path(strippedURI);
        } catch (URISyntaxException e) {
          throw new IllegalArgumentException("Error processing URI", e);
        }
      }
    }

    // this might have name collisions. copy will throw an exception
    // parse the original path to create new path
    Path newPath = root ?
        parentDir : new Path(parentDir, originalPath.getName());
    FileUtil.copy(remoteFs, originalPath, jtFs, newPath, false, conf);
    jtFs.setReplication(newPath, replication);
    jtFs.makeQualified(newPath);
    return newPath;
  }

  /**
   * Checksum a local resource file and call use for that resource with the scm.
   */
  private URI useSharedCache(URI sourceFile, String resourceName,

View on GitHub (pinned to 2add963021)

Solutions

1. Drop the trailing '/' from distributed-cache entries — point at the file itself, not a directory-style path
2. Build URIs with the API (new URI(scheme, authority, path, fragment)) or Path instead of concatenation
3. Sanitize entries in a pre-submit validation loop: reject values ending in '/' or failing new URI(...)
4. Log the exact -files/-libjars/-archives values when this fires; the message omits the original string

Example fix

# before
hadoop jar app.jar Driver -files "hdfs://nn/data/#" in out
# IllegalArgumentException: Error processing URI

# after
hadoop jar app.jar Driver -files "hdfs://nn/data/side-data.txt" in out

// reject slash-terminated or otherwise pathological cache entries
static void checkEntry(String s) {
  if (s.endsWith("/")) throw new IllegalArgumentException("Cache entry must name a file, not end in '/': " + s);
  try { new URI(s); } catch (URISyntaxException e) {
    throw new IllegalArgumentException("Cache entry is not a valid URI: " + s, e);
  }
}

static boolean isValidCacheUri(String s) {
  if (s == null || s.endsWith("/")) return false;
  try { new URI(s); return true; }
  catch (URISyntaxException e) { return false; }
}

When it happens

Trigger: A -files/-archives entry ending in '/' whose stem is still illegal (e.g. 'hdfs://nn/dir/#', double slashes, an entry that is only a fragment); hand-concatenated URIs like "hdfs://" + host + "/dir/" producing malformed schemes when combined with trailing slashes; paths from templating systems that leave stray characters before the final '/'.

Common situations: Scripts building paths by string concatenation; configuration templates with a trailing slash plus placeholder substitution that empties a component; rare — most malformed entries fail earlier at new URI(entry), so this specific message signals a slash-suffixed pathological value.

Related errors

• Error parsing files argument. Argument must be a valid URI:
• Error parsing archives argument. Argument must be a valid UR
• Error parsing argument. Argument must be a valid URI: {}
• Error parsing local resource path. Path was not able to be c
• Invalid specification for distributed-cache artifacts of typ

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8bce2d25cc23dc1a. Report an issue: GitHub.