What causes "Too many arguments: expected {} but got {}" in apache/hadoop?

Invoke the command once per path (loop) instead of passing all paths at once to single-path commands like test

apache/hadoop · error · TooManyArgumentsException

Too many arguments: expected {} but got {}

Error message

Too many arguments: expected {} but got {}

What it means

Thrown by CommandFormat.parse() when more positional arguments remain after option stripping than the command's declared maximum (maxPar). TooManyArgumentsException extends IllegalArgumentException and reports 'Too many arguments: expected <max> but got <n>'. Only commands constructed with a finite maxPar (e.g. 'test' with maxPar=1) can raise it; commands declared with Integer.MAX_VALUE never do.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandFormat.java:138

          args.remove(pos);
        } else {
          arg = "";
        }
        if (!arg.startsWith("-") || arg.equals("-")) {
          optionsWithValue.put(opt, arg);
        }
      } else if (ignoreUnknownOpts) {
        pos++;
      } else {
        throw new UnknownOptionException(arg);
      }
    }
    int psize = args.size();
    if (psize < minPar) {
      throw new NotEnoughArgumentsException(minPar, psize);
    }
    if (psize > maxPar) {
      throw new TooManyArgumentsException(maxPar, psize);
    }
  }
  
  /** Return if the option is set or not
   * 
   * @param option String representation of an option
   * @return true is the option is set; false otherwise
   */
  public boolean getOpt(String option) {
    return options.containsKey(option) ? options.get(option) : false;
  }

  /**
   * get the option's value
   *
   * @param option option name
   * @return option value
   * if option exists, but no value assigned, return ""

View on GitHub (pinned to 2add963021)

Solutions

Invoke the command once per path (loop) instead of passing all paths at once to single-path commands like test
Quote or pre-validate globs so they expand to the expected number of arguments before the call
Re-read 'hdfs dfs -help <command>' to see the accepted argument shape

Example fix

# before
hdfs dfs -test -e /data/a /data/b
# after
hdfs dfs -test -e /data/a && hdfs dfs -test -e /data/b

Defensive patterns

Strategy: validation

Validate before calling

// single-path commands: assert exactly one positional before running
String[] positional = Arrays.stream(argv).filter(a -> !a.startsWith("-")).toArray(String[]::new);
if (positional.length > 1) {
  throw new IllegalArgumentException("command takes exactly one path, got " + positional.length);
}

Try / catch

try {
  cf.parse(args);
} catch (CommandFormat.TooManyArgumentsException e) {
  // split the workload and run the command once per path
  for (String p : extraPaths) runCommand(singlePathCmd(p));
}

Prevention

Quote globs you want passed literally and count matches you want expanded
Read the command's USAGE - single-path commands (test) never accept a list
In wrappers, validate argument count ranges before delegating to FsShell

When it happens

Trigger: 'hdfs dfs -test -e /a /b' (test accepts at most one path), passing multiple paths to any single-path command, or a shell glob expanding ('*.csv' to many files) into a command with maxPar=1.

Common situations: Unquoted globs expanding unexpectedly in scripts; users assuming 'test' accepts multiple paths like 'ls'; wrapping loops that feed an array where a single path is expected; misreading an option as a value so a stray token lands in the positional list.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2f5d19d305ef6562. Report an issue: GitHub.