apache/hadoop · error · InvalidInputException

{path} doesn't exist

Error message

{path} doesn't exist

What it means

SimpleCopyListing validates every source path with fs.exists() during listing; this path resolved to nothing. It is the same class of failure as the glob check in GlobbedCopyListing, but typically surfaces for entries of a -f file listing or programmatically supplied paths, and can also fire when a source disappears between glob expansion and the existence pass.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java:181

      }

      Path srcPath = context.getSourcePaths().get(0);
      FileSystem sourceFS = srcPath.getFileSystem(getConf());
      if (!sourceFS.isFile(srcPath)) {
        throw new InvalidInputException("Cannot copy " + srcPath +
            ", which is not a file to " + targetPath);
      }
    }

    if (context.shouldAtomicCommit() && targetExists) {
      throw new InvalidInputException("Target path for atomic-commit already exists: " +
        targetPath + ". Cannot atomic-commit to pre-existing target-path.");
    }

    for (Path path: context.getSourcePaths()) {
      FileSystem fs = path.getFileSystem(getConf());
      if (!fs.exists(path)) {
        throw new InvalidInputException(path + " doesn't exist");
      }
      if (Path.getPathWithoutSchemeAndAuthority(path).toString().
          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {
        if (!targetIsReservedRaw) {
          final String msg = "The source path '" + path + "' starts with " +
              HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the target path '" +
              targetPath + "' does not. Either all or none of the paths must " +
              "have this prefix.";
          throw new InvalidInputException(msg);
        }
      } else if (targetIsReservedRaw) {
        final String msg = "The target path '" + targetPath + "' starts with " +
                HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the source path '" +
                path + "' does not. Either all or none of the paths must " +
                "have this prefix.";
        throw new InvalidInputException(msg);
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Check the exact path from the message: hadoop fs -ls <path>.
  2. Regenerate the listing file immediately before the copy, filtering to existing paths.
  3. Fix scheme/authority: use fully-qualified URIs valid on the cluster running distcp.
  4. Pre-filter: while read p; do hadoop fs -test -e "$p" && echo "$p"; done < list > fresh-list.

Example fix

# before: list.txt contains deleted paths
hadoop distcp -f list.txt hdfs://nn/tgt

# after: filter the listing to existing paths first
while read -r p; do hadoop fs -test -e "$p" && echo "$p"; done < list.txt | hadoop fs -put -f - /tmp/fresh.txt
hadoop distcp -f /tmp/fresh.txt hdfs://nn/tgt
Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight every path from a -f listing immediately before the run
FileSystem fs = targetPath.getFileSystem(conf);
List<String> valid = new ArrayList<>();
for (String line : Files.readAllLines(listFileLocal)) {
  String p = line.trim();
  if (p.isEmpty()) continue;
  if (fs.exists(new Path(p))) valid.add(p);
  else LOG.warn("Dropping missing source: " + p);
}
if (valid.isEmpty()) throw new InvalidInputException("All listed sources are missing");

Try / catch

try {
  copyListing.doBuildListing(listingFile, context);
} catch (InvalidInputException e) {
  if (e.getMessage().endsWith("doesn't exist")) {
    // message names the path; regenerate the listing without it and re-run
    regenerateListingWithout(e.getMessage());
  } else {
    throw e;
  }
}

Prevention

When it happens

Trigger: A -f listing file containing stale paths deleted since the listing was generated; a path on the wrong authority (different nameservice, viewfs mount mismatch); source removed by a concurrent job between listing generation and the distcp run.

Common situations: nightly -f lists built hours before the copy; paths valid on the build host's cluster but not the distcp cluster's view; upstream retention jobs deleting sources mid-pipeline.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f2456df51846fae. Report an issue: GitHub.