{"record":{"id":"f5ce12a70a5c2f00","repo":"apache/hadoop","slug":"nothing-to-process-source-paths-empty","errorCode":null,"errorMessage":"Nothing to process. Source paths::EMPTY","messagePattern":"Nothing to process\\. Source paths::EMPTY","errorType":"exception","errorClass":"InvalidInputException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/GlobbedCopyListing.java","lineNumber":72,"sourceCode":"  protected void validatePaths(DistCpContext context)\n      throws IOException, InvalidInputException {\n  }\n\n  /**\n   * Implementation of CopyListing::buildListing().\n   * Creates the copy listing by \"globbing\" all source-paths.\n   * @param pathToListingFile The location at which the copy-listing file\n   *                           is to be created.\n   * @param context The distcp context with associated input options.\n   * @throws IOException if unable to construct the fileList\n   */\n  @Override\n  public void doBuildListing(Path pathToListingFile, DistCpContext context)\n      throws IOException {\n\n    List<Path> globbedPaths = new ArrayList<Path>();\n    if (context.getSourcePaths().isEmpty()) {\n      throw new InvalidInputException(\"Nothing to process. Source paths::EMPTY\");  \n    }\n\n    for (Path p : context.getSourcePaths()) {\n      FileSystem fs = p.getFileSystem(getConf());\n      FileStatus[] inputs = fs.globStatus(p);\n\n      if(inputs != null && inputs.length > 0) {\n        for (FileStatus onePath: inputs) {\n          globbedPaths.add(onePath.getPath());\n        }\n      } else {\n        throw new InvalidInputException(p + \" doesn't exist\");        \n      }\n    }\n\n    context.setSourcePaths(globbedPaths);\n    simpleListing.buildListing(pathToListingFile, context);\n  }","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/GlobbedCopyListing.java#L54-L90","documentation":"GlobbedCopyListing expands glob patterns in DistCp source paths before the copy listing is built. If the DistCpContext carries zero source paths, there is nothing to enumerate and it aborts immediately with InvalidInputException, before any MR job is submitted. The usual root cause is an empty -f source-listing file (its lines are loaded into sourcePaths) or a programmatic caller that built options without any source.","triggerScenarios":"hadoop distcp -f <file> <target> where the listing file is empty or contains only blank lines; a wrapper script that expands an unset shell variable so no positional source reaches the command line together with -f; a Java caller constructing DistCpOptions/DistCpContext with an empty sourcePaths list.","commonSituations":"cron jobs that copy whatever an upstream feed produced, and the feed produced nothing that night; the upstream job that writes the -f file failed silently or wrote an empty file; scripts passing $SOURCES when SOURCES is unset; programmatic DistCp use where setSourcePaths was forgotten.","solutions":["If using -f, verify the listing has content: hadoop fs -cat <file> | grep -c '[^[:space:]]' must return > 0.","If passing sources positionally, echo the expanded command before running to catch unset/empty variables.","Programmatic callers: assert context.getSourcePaths() is non-empty before calling buildListing/execute.","Fix the upstream producer so it fails loudly when it would emit an empty list, then regenerate and re-run."],"exampleFix":"# before: /tmp/list.txt is empty, distcp aborts\nhadoop distcp -f /tmp/list.txt hdfs://nn/tgt\n\n# after: only run when the listing has real entries\nif hadoop fs -cat /tmp/list.txt | grep -q '[^[:space:]]'; then\n  hadoop distcp -f /tmp/list.txt hdfs://nn/tgt\nelse\n  echo \"source list empty; nothing to copy\" >&2\nfi","handlingStrategy":"validation","validationCode":"// Before constructing/running DistCp\nPath listFile = options.getSourceFileListing(); // set by -f\nif (listFile != null) {\n  FileSystem lfs = listFile.getFileSystem(conf);\n  int lines = 0;\n  try (BufferedReader r = new BufferedReader(new InputStreamReader(\n          lfs.open(listFile), StandardCharsets.UTF_8))) {\n    while (r.readLine() != null) { lines++; }\n  }\n  if (lines == 0) {\n    throw new InvalidInputException(\"Source listing \" + listFile + \" is empty\");\n  }\n} else if (options.getSourcePaths().isEmpty()) {\n  throw new InvalidInputException(\"No source paths given\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  distCp.run(args);\n} catch (InvalidInputException e) {\n  if (e.getMessage().contains(\"Source paths::EMPTY\")) {\n    // configuration/data problem - fail the pipeline loudly, do not retry\n    throw new IllegalStateException(\"Upstream produced an empty source list\", e);\n  }\n  throw e;\n}","preventionTips":["Generate the -f listing and run distcp in the same pipeline step so emptiness is detected immediately.","Fail upstream jobs when they would emit an empty file list instead of writing an empty file.","Log the source-path count before every distcp invocation."],"tags":["distcp","input-validation","source-paths","empty-input"],"backgroundTag":"empty-input-list","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}