What causes "Nothing to process. Source paths::EMPTY" in apache/hadoop?

If using -f, verify the listing has content: hadoop fs -cat | grep -c '[^[:space:]]' must return > 0.

apache/hadoop · error · InvalidInputException

Nothing to process. Source paths::EMPTY

Error message

Nothing to process. Source paths::EMPTY

What it means

GlobbedCopyListing expands glob patterns in DistCp source paths before the copy listing is built. If the DistCpContext carries zero source paths, there is nothing to enumerate and it aborts immediately with InvalidInputException, before any MR job is submitted. The usual root cause is an empty -f source-listing file (its lines are loaded into sourcePaths) or a programmatic caller that built options without any source.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/GlobbedCopyListing.java:72

  protected void validatePaths(DistCpContext context)
      throws IOException, InvalidInputException {
  }

  /**
   * Implementation of CopyListing::buildListing().
   * Creates the copy listing by "globbing" all source-paths.
   * @param pathToListingFile The location at which the copy-listing file
   *                           is to be created.
   * @param context The distcp context with associated input options.
   * @throws IOException if unable to construct the fileList
   */
  @Override
  public void doBuildListing(Path pathToListingFile, DistCpContext context)
      throws IOException {

    List<Path> globbedPaths = new ArrayList<Path>();
    if (context.getSourcePaths().isEmpty()) {
      throw new InvalidInputException("Nothing to process. Source paths::EMPTY");  
    }

    for (Path p : context.getSourcePaths()) {
      FileSystem fs = p.getFileSystem(getConf());
      FileStatus[] inputs = fs.globStatus(p);

      if(inputs != null && inputs.length > 0) {
        for (FileStatus onePath: inputs) {
          globbedPaths.add(onePath.getPath());
        }
      } else {
        throw new InvalidInputException(p + " doesn't exist");        
      }
    }

    context.setSourcePaths(globbedPaths);
    simpleListing.buildListing(pathToListingFile, context);
  }

View on GitHub (pinned to 2add963021)

Solutions

If using -f, verify the listing has content: hadoop fs -cat <file> | grep -c '[^[:space:]]' must return > 0.
If passing sources positionally, echo the expanded command before running to catch unset/empty variables.
Programmatic callers: assert context.getSourcePaths() is non-empty before calling buildListing/execute.
Fix the upstream producer so it fails loudly when it would emit an empty list, then regenerate and re-run.

Example fix

# before: /tmp/list.txt is empty, distcp aborts
hadoop distcp -f /tmp/list.txt hdfs://nn/tgt

# after: only run when the listing has real entries
if hadoop fs -cat /tmp/list.txt | grep -q '[^[:space:]]'; then
  hadoop distcp -f /tmp/list.txt hdfs://nn/tgt
else
  echo "source list empty; nothing to copy" >&2
fi

Defensive patterns

Strategy: validation

Validate before calling

// Before constructing/running DistCp
Path listFile = options.getSourceFileListing(); // set by -f
if (listFile != null) {
  FileSystem lfs = listFile.getFileSystem(conf);
  int lines = 0;
  try (BufferedReader r = new BufferedReader(new InputStreamReader(
          lfs.open(listFile), StandardCharsets.UTF_8))) {
    while (r.readLine() != null) { lines++; }
  }
  if (lines == 0) {
    throw new InvalidInputException("Source listing " + listFile + " is empty");
  }
} else if (options.getSourcePaths().isEmpty()) {
  throw new InvalidInputException("No source paths given");
}

Try / catch

try {
  distCp.run(args);
} catch (InvalidInputException e) {
  if (e.getMessage().contains("Source paths::EMPTY")) {
    // configuration/data problem - fail the pipeline loudly, do not retry
    throw new IllegalStateException("Upstream produced an empty source list", e);
  }
  throw e;
}

Prevention

Generate the -f listing and run distcp in the same pipeline step so emptiness is detected immediately.
Fail upstream jobs when they would emit an empty file list instead of writing an empty file.
Log the source-path count before every distcp invocation.

When it happens

Trigger: hadoop distcp -f <file> <target> where the listing file is empty or contains only blank lines; a wrapper script that expands an unset shell variable so no positional source reaches the command line together with -f; a Java caller constructing DistCpOptions/DistCpContext with an empty sourcePaths list.

Common situations: cron jobs that copy whatever an upstream feed produced, and the feed produced nothing that night; the upstream job that writes the -f file failed silently or wrote an empty file; scripts passing $SOURCES when SOURCES is unset; programmatic DistCp use where setSourcePaths was forgotten.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f5ce12a70a5c2f00. Report an issue: GitHub.