apache/hadoop · error · IllegalArgumentException

Both source file listing and source paths present

Error message

Both source file listing and source paths present

What it means

Sources can be specified two ways: a -f file listing, or positional source paths in front of the target. parseSourceAndTargetPaths found both present - -f was given AND at least one extra positional path besides the target - and refuses to guess which set to copy.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:275

    String[] leftOverArgs = command.getArgs();
    if (leftOverArgs == null || leftOverArgs.length < 1) {
      throw new IllegalArgumentException("Target path not specified");
    }

    //Last Argument is the target path
    targetPath = new Path(leftOverArgs[leftOverArgs.length - 1].trim());

    //Copy any source paths in the arguments to the list
    for (int index = 0; index < leftOverArgs.length - 1; index++) {
      sourcePaths.add(new Path(leftOverArgs[index].trim()));
    }

    /* If command has source file listing, use it else, fall back on source
       paths in args.  If both are present, throw exception and bail */
    if (command.hasOption(
        DistCpOptionSwitch.SOURCE_FILE_LISTING.getSwitch())) {
      if (!sourcePaths.isEmpty()) {
        throw new IllegalArgumentException("Both source file listing and " +
            "source paths present");
      }
      return new DistCpOptions.Builder(new Path(getVal(command,
          DistCpOptionSwitch.SOURCE_FILE_LISTING.getSwitch())), targetPath);
    } else {
      if (sourcePaths.isEmpty()) {
        throw new IllegalArgumentException("Neither source file listing nor " +
            "source paths present");
      }
      return new DistCpOptions.Builder(sourcePaths, targetPath);
    }
  }

  private static String getVal(CommandLine command, String swtch) {
    if (swtch == null) {
      return null;
    }
    String optionValue = command.getOptionValue(swtch.trim());

View on GitHub (pinned to 2add963021)

Solutions

1. Pick one source mechanism: remove -f and keep inline paths, or keep -f and remove all inline sources so only the target remains.
2. Re-run: hadoop distcp -f /tmp/list.txt hdfs://nn/tgt or hadoop distcp hdfs://nn/src hdfs://nn/tgt.
3. Audit wrapper scripts for unconditional source-path appends.

Example fix

# before: both -f and an inline source (plus target)
hadoop distcp -f /tmp/list.txt hdfs://nn/src hdfs://nn/tgt

# after: sources from exactly one place
hadoop distcp -f /tmp/list.txt hdfs://nn/tgt

// Wrapper validation: -f and inline source paths are mutually exclusive
boolean hasF = false;
List<String> positional = new ArrayList<>();
for (int i = 0; i < args.length; i++) {
  if ("-f".equals(args[i])) { hasF = true; i++; } // skip its value
  else if (!args[i].startsWith("-")) positional.add(args[i]);
}
if (hasF && positional.size() > 1) {
  throw new IllegalArgumentException(
      "-f given plus inline sources " + positional.subList(0, positional.size() - 1));
}

try {
  OptionsParser.parse(args);
} catch (IllegalArgumentException e) {
  if ("Both source file listing and source paths present".equals(e.getMessage())) {
    // drop either -f or the inline sources, then re-run
    throw new IllegalArgumentException("Pick one source mechanism: -f OR inline paths", e);
  }
  throw e;
}

When it happens

Trigger: hadoop distcp -f /tmp/list.txt hdfs://nn/src hdfs://nn/tgt - two positional paths means hdfs://nn/src became a source path while -f also supplied a listing.

Common situations: incrementally editing a working one-liner by adding -f but forgetting to remove the inline source; scripts appending a default source path unconditionally; operators assuming -f is an additional filter rather than an alternative.

Related errors

• Neither source file listing nor source paths present
• Nothing to process. Source paths::EMPTY
• Unable to parse arguments. {args}
• Bandwidth specified is invalid: {value}
• Number of liststatus threads is invalid: {value}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef18a6ba870a28d3. Report an issue: GitHub.