{"record":{"id":"6f2456df51846fae","repo":"apache/hadoop","slug":"path-doesn-t-exist","errorCode":null,"errorMessage":"{path} doesn't exist","messagePattern":"(.+?) doesn't exist","errorType":"exception","errorClass":"InvalidInputException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java","lineNumber":181,"sourceCode":"      }\n\n      Path srcPath = context.getSourcePaths().get(0);\n      FileSystem sourceFS = srcPath.getFileSystem(getConf());\n      if (!sourceFS.isFile(srcPath)) {\n        throw new InvalidInputException(\"Cannot copy \" + srcPath +\n            \", which is not a file to \" + targetPath);\n      }\n    }\n\n    if (context.shouldAtomicCommit() && targetExists) {\n      throw new InvalidInputException(\"Target path for atomic-commit already exists: \" +\n        targetPath + \". Cannot atomic-commit to pre-existing target-path.\");\n    }\n\n    for (Path path: context.getSourcePaths()) {\n      FileSystem fs = path.getFileSystem(getConf());\n      if (!fs.exists(path)) {\n        throw new InvalidInputException(path + \" doesn't exist\");\n      }\n      if (Path.getPathWithoutSchemeAndAuthority(path).toString().\n          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {\n        if (!targetIsReservedRaw) {\n          final String msg = \"The source path '\" + path + \"' starts with \" +\n              HDFS_RESERVED_RAW_DIRECTORY_NAME + \" but the target path '\" +\n              targetPath + \"' does not. Either all or none of the paths must \" +\n              \"have this prefix.\";\n          throw new InvalidInputException(msg);\n        }\n      } else if (targetIsReservedRaw) {\n        final String msg = \"The target path '\" + targetPath + \"' starts with \" +\n                HDFS_RESERVED_RAW_DIRECTORY_NAME + \" but the source path '\" +\n                path + \"' does not. Either all or none of the paths must \" +\n                \"have this prefix.\";\n        throw new InvalidInputException(msg);\n      }\n    }","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L163-L199","documentation":"SimpleCopyListing validates every source path with fs.exists() during listing; this path resolved to nothing. It is the same class of failure as the glob check in GlobbedCopyListing, but typically surfaces for entries of a -f file listing or programmatically supplied paths, and can also fire when a source disappears between glob expansion and the existence pass.","triggerScenarios":"A -f listing file containing stale paths deleted since the listing was generated; a path on the wrong authority (different nameservice, viewfs mount mismatch); source removed by a concurrent job between listing generation and the distcp run.","commonSituations":"nightly -f lists built hours before the copy; paths valid on the build host's cluster but not the distcp cluster's view; upstream retention jobs deleting sources mid-pipeline.","solutions":["Check the exact path from the message: hadoop fs -ls <path>.","Regenerate the listing file immediately before the copy, filtering to existing paths.","Fix scheme/authority: use fully-qualified URIs valid on the cluster running distcp.","Pre-filter: while read p; do hadoop fs -test -e \"$p\" && echo \"$p\"; done < list > fresh-list."],"exampleFix":"# before: list.txt contains deleted paths\nhadoop distcp -f list.txt hdfs://nn/tgt\n\n# after: filter the listing to existing paths first\nwhile read -r p; do hadoop fs -test -e \"$p\" && echo \"$p\"; done < list.txt | hadoop fs -put -f - /tmp/fresh.txt\nhadoop distcp -f /tmp/fresh.txt hdfs://nn/tgt","handlingStrategy":"validation","validationCode":"// Pre-flight every path from a -f listing immediately before the run\nFileSystem fs = targetPath.getFileSystem(conf);\nList<String> valid = new ArrayList<>();\nfor (String line : Files.readAllLines(listFileLocal)) {\n  String p = line.trim();\n  if (p.isEmpty()) continue;\n  if (fs.exists(new Path(p))) valid.add(p);\n  else LOG.warn(\"Dropping missing source: \" + p);\n}\nif (valid.isEmpty()) throw new InvalidInputException(\"All listed sources are missing\");","typeGuard":null,"tryCatchPattern":"try {\n  copyListing.doBuildListing(listingFile, context);\n} catch (InvalidInputException e) {\n  if (e.getMessage().endsWith(\"doesn't exist\")) {\n    // message names the path; regenerate the listing without it and re-run\n    regenerateListingWithout(e.getMessage());\n  } else {\n    throw e;\n  }\n}","preventionTips":["Generate the -f listing as close to the copy as possible.","Use fully-qualified URIs valid on the cluster that runs distcp.","Filter listings through hadoop fs -test -e before submission."],"tags":["distcp","path-not-found","input-validation","file-listing"],"backgroundTag":"path-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}