apache/hadoop · error · InvalidInputException

{p} doesn't exist

Error message

{p} doesn't exist

What it means

During listing, GlobbedCopyListing resolves every source path through FileSystem.globStatus(). A null or zero-length result means the pattern matched nothing on that FileSystem - a nonexistent path, a typo, or a legal glob that happens to match zero files. DistCp aborts listing, so the job never submits.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/GlobbedCopyListing.java:84

  @Override
  public void doBuildListing(Path pathToListingFile, DistCpContext context)
      throws IOException {

    List<Path> globbedPaths = new ArrayList<Path>();
    if (context.getSourcePaths().isEmpty()) {
      throw new InvalidInputException("Nothing to process. Source paths::EMPTY");  
    }

    for (Path p : context.getSourcePaths()) {
      FileSystem fs = p.getFileSystem(getConf());
      FileStatus[] inputs = fs.globStatus(p);

      if(inputs != null && inputs.length > 0) {
        for (FileStatus onePath: inputs) {
          globbedPaths.add(onePath.getPath());
        }
      } else {
        throw new InvalidInputException(p + " doesn't exist");        
      }
    }

    context.setSourcePaths(globbedPaths);
    simpleListing.buildListing(pathToListingFile, context);
  }

  /** {@inheritDoc} */
  @Override
  protected long getBytesToCopy() {
    return simpleListing.getBytesToCopy();
  }

  /** {@inheritDoc} */
  @Override
  protected long getNumberOfPaths() {
    return simpleListing.getNumberOfPaths();
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Run the same pattern through ls first: hadoop fs -ls 'hdfs://nn/data/2026-08-22-*.csv' - if it shows nothing, fix the pattern or paths.
2. Use fully-qualified URIs with the correct scheme and nameservice authority instead of relative paths.
3. Escape glob metacharacters that are literal parts of filenames (\*, \?, \[, \{).
4. If missing sources are expected, pre-filter them out of the -f listing instead of letting distcp abort.

Example fix

# before: glob matches nothing, listing aborts
hadoop distcp 'hdfs://nn/inbox/2026-08-22-*.avro' hdfs://nn2/archive/

# after: verify the expansion, then run
hadoop fs -ls 'hdfs://nn/inbox/2026-08-22-*.avro'
hadoop distcp 'hdfs://nn/inbox/2026-08-22-*.avro' hdfs://nn2/archive/

// Pre-flight: every source pattern must expand to >=1 entry
for (Path p : context.getSourcePaths()) {
  FileSystem fs = p.getFileSystem(conf);
  FileStatus[] matches = fs.globStatus(p);
  if (matches == null || matches.length == 0) {
    throw new InvalidInputException("Source matches nothing: " + p);
  }
}

try {
  distCp.run(args);
} catch (InvalidInputException e) {
  // message embeds the failing path, e.g. "hdfs://nn/x/*.csv doesn't exist"
  if (e.getMessage().endsWith("doesn't exist")) {
    reportAndFixSource(e.getMessage()); // fix pattern/authority, re-run - no point retrying as-is
  } else {
    throw e;
  }
}

When it happens

Trigger: A source glob with no matches, e.g. 'hdfs://nn/data/2026-08-22-*.csv' on a day with no data; a path on a different nameservice than the authority given; a source deleted between run scheduling and listing; a literal filename containing glob metacharacters (?, *, {, [) unescaped, so it is treated as a pattern matching nothing.

Common situations: date-stamped globs for periods with no data; typos in long HDFS paths; relative paths resolving against a different fs.defaultFS; viewfs/HA authorities that mount the path differently; filenames with literal glob characters.

Related errors

• No such file or directory
• {path} doesn't exist
• The source path '{path}' starts with /.reserved/raw but the
• The target path '{targetPath}' starts with /.reserved/raw bu
• No such file or directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/336f4133ea6c4b49. Report an issue: GitHub.