apache/hadoop · error · IOException

The number of source paths is less than 2. <target path> <sr

Error message

The number of source paths is less than 2. <target path> <src path> <src path> ...

What it means

Thrown by Concat.processArguments() when fewer than 3 total arguments remain (target plus fewer than 2 sources). FileSystem.concat() with fewer than two parts is meaningless or unsupported, so the shell pre-empts it with IOException('The number of source paths is less than 2. <target path> <src path> <src path> ...').

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Concat.java:56

  public static void registerCommands(CommandFactory factory) {
    factory.addClass(Concat.class, "-concat");
  }

  public static final String NAME = "concat";
  public static final String USAGE = "<target path> <src path> <src path> ...";
  public static final String DESCRIPTION = "Concatenate existing source files"
      + " into the target file. Target file and source files should be in the"
      + " same directory.";
  private static FileSystem testFs; // test only.

  @Override
  protected void processArguments(LinkedList<PathData> args)
      throws IOException {
    if (args.size() < 1) {
      throw new IOException("Target path not specified. " + USAGE);
    }
    if (args.size() < 3) {
      throw new IOException(
          "The number of source paths is less than 2. " + USAGE);
    }
    PathData target = args.removeFirst();
    LinkedList<PathData> srcList = args;
    if (!target.exists || !target.stat.isFile()) {
      throw new FileNotFoundException(String
          .format("Target path %s does not exist or is" + " not file.",
              target.path));
    }
    Path[] srcArray = new Path[srcList.size()];
    for (int i = 0; i < args.size(); i++) {
      PathData src = srcList.get(i);
      if (!src.exists || !src.stat.isFile()) {
        throw new FileNotFoundException(
            String.format("%s does not exist or is not file.", src.path));
      }
      srcArray[i] = src.path;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Pass at least two existing source files along with the target
2. Check the number of matches first (e.g. count 'hdfs dfs -ls /data/part-*' output) and skip concat when < 2
3. If only one part exists, copy/append it with a different command instead of concat

Example fix

# before
hdfs dfs concat /data/target /data/part-0
# after
hdfs dfs concat /data/target /data/part-0 /data/part-1

// require target + >=2 sources
if (positionalArgs.size() < 3) {
  throw new IllegalArgumentException(
    "need at least 3 paths (target + 2 sources), got " + positionalArgs.size());
}

try {
  shellRun(concatArgs);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("less than 2")) {
    // not enough parts to justify a concat; fall back to a plain copy for the single part
    shellRun("-cp", singleSource, target);
  } else throw e;
}

When it happens

Trigger: 'hdfs dfs concat /data/target /data/part-0' (only one source); glob for sources expanding to a single file; source list built by a loop that appended fewer entries than expected.

Common situations: Compaction jobs whose input filter matched only one file for the period; hand-testing concat with one part before the real run.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Target path not specified. <target path> <src path> <src pat
• Illegal option {}
• Not enough arguments: expected {} but got {}
• Too many arguments: expected {} but got {}
• Target path %s does not exist or is not file.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/473328c1ebf52732. Report an issue: GitHub.