apache/hadoop · error · IOException

Unexpected URISyntaxException: ${e}

Error message

Unexpected URISyntaxException: ${e}

What it means

IOException('Unexpected URISyntaxException: ...') thrown by AppendToFile.expandArgument (CopyCommands.java:370) for 'hadoop fs -appendToFile'. A source argument that is not '-' (stdin) is parsed with 'new URI(arg)'; when that fails the code retries with PathData(String) only on Windows (Path.WINDOWS) because PathData understands drive-letter paths. On Linux/macOS the IOException is fatal and includes the URISyntaxException text.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java:370

    public void setAppendToNewBlock(boolean appendToNewBlock) {
      this.appendToNewBlock = appendToNewBlock;
    }

    // commands operating on local paths have no need for glob expansion
    @Override
    protected List<PathData> expandArgument(String arg) throws IOException {
      List<PathData> items = new LinkedList<PathData>();
      if (arg.equals("-")) {
        readStdin = true;
      } else {
        try {
          items.add(new PathData(new URI(arg), getConf()));
        } catch (URISyntaxException e) {
          if (Path.WINDOWS) {
            // Unlike URI, PathData knows how to parse Windows drive-letter paths.
            items.add(new PathData(arg, getConf()));
          } else {
            throw new IOException("Unexpected URISyntaxException: " + e.toString());
          }
        }
      }
      return items;
    }

    @Override
    protected void processOptions(LinkedList<String> args)
        throws IOException {

      if (args.size() < 2) {
        throw new IOException("missing destination argument");
      }

      CommandFormat cf = new CommandFormat(2, Integer.MAX_VALUE, "n");
      cf.parse(args);
      appendToNewBlock = cf.getOpt("n");
      getRemoteDestination(args);

View on GitHub (pinned to 2add963021)

Solutions

1. Rename the file to avoid URI-reserved characters (spaces, brackets, %) before appending
2. Percent-encode the illegal characters in the argument (space -> %20) so new URI() parses it
3. Pipe the content instead: 'cat "my file.txt" | hadoop fs -appendToFile - /dst' ('-' bypasses URI parsing entirely)
4. On Windows-only flows no change is needed; on Linux avoid passing drive-letter paths

Example fix

# before (Linux)
hadoop fs -appendToFile 'run [3].log' /logs/all    # Unexpected URISyntaxException

# after
cat 'run [3].log' | hadoop fs -appendToFile - /logs/all

if (!arg.equals("-")) {
  try {
    new URI(arg);
  } catch (URISyntaxException e) {
    if (!Path.WINDOWS) {
      // pipe the file instead: cat file | hadoop fs -appendToFile - dst
      throw new IOException("Unexpected URISyntaxException: " + e.toString());
    }
  }
}

static boolean isAppendableArgument(String arg) {
  if (arg.equals("-")) return true;
  try { new URI(arg); return true; } catch (URISyntaxException e) { return false; }
}

try {
  items.add(new PathData(new URI(arg), getConf()));
} catch (URISyntaxException e) {
  if (Path.WINDOWS) {
    items.add(new PathData(arg, getConf()));
  } else {
    throw new IOException("Unexpected URISyntaxException: " + e.toString());
  }
}

When it happens

Trigger: 'hadoop fs -appendToFile "my file.txt" /dst' on Linux (space in name); a source containing '[', ']', or a stray '%'; Windows-style 'C:\logs\a.txt' passed on a Linux host; a glob the shell did not expand (quoted) containing braces/brackets.

Common situations: Log-ingestion scripts appending files with timestamps/spaces in names; running a script written and tested on Windows against a Linux cluster; quoted globs reaching the command unexpanded.

Related errors

• unexpected URISyntaxException
• missing destination argument
• stdin (-) must be the sole input argument when present
• GCS path supports only '%s' scheme, instead got '%s' from '%
• Bad configuration of hadoop.security.key.provider.path at ${

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/41511a1bb2ddfe1b. Report an issue: GitHub.