apache/hadoop · error · IllegalArgumentException

Listing file doesn't exist at: {listingFilePath}

Error message

Listing file doesn't exist at: {listingFilePath}

What it means

UniformSizeInputFormat (the listing-based split strategy) reads the copy-listing sequence file whose path is stored in distcp.listing.file.path. During split generation, if the listing file does not exist on its filesystem, splits cannot be created and this IllegalArgumentException is thrown. The listing is normally produced by CopyListing during DistCp setup before job submission.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/UniformSizeInputFormat.java:148

    return splits;
  }

  private static Path getListingFilePath(Configuration configuration) {
    final String listingFilePathString =
            configuration.get(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, "");

    assert !listingFilePathString.equals("")
              : "Couldn't find listing file. Invalid input.";
    return new Path(listingFilePathString);
  }

  private SequenceFile.Reader getListingFileReader(Configuration configuration) {

    final Path listingFilePath = getListingFilePath(configuration);
    try {
      final FileSystem fileSystem = listingFilePath.getFileSystem(configuration);
      if (!fileSystem.exists(listingFilePath))
        throw new IllegalArgumentException("Listing file doesn't exist at: "
                                           + listingFilePath);

      return new SequenceFile.Reader(configuration,
                                     SequenceFile.Reader.file(listingFilePath));
    }
    catch (IOException exception) {
      LOG.error("Couldn't find listing file at: " + listingFilePath, exception);
      throw new IllegalArgumentException("Couldn't find listing-file at: "
                                         + listingFilePath, exception);
    }
  }

  /**
   * Implementation of InputFormat::createRecordReader().
   * @param split The split for which the RecordReader is sought.
   * @param context The context of the current task-attempt.
   * @return A SequenceFileRecordReader instance, (since the copy-listing is a
   * simple sequence-file.)

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the printed path exists and is readable by the job user (hdfs dfs -ls <listingFilePath>)
2. Re-run the distcp job so the copy-listing is regenerated
3. Protect the distcp meta folder from deletion for the lifetime of the MapReduce job
4. When building jobs manually, generate the listing with CopyListing and set distcp.listing.file.path to the result

Path listing = new Path(conf.get("distcp.listing.file.path"));
if (!listing.getFileSystem(conf).exists(listing)) {
  throw new FileNotFoundException("copy-listing missing: " + listing);
}

When it happens

Trigger: The distcp meta/staging folder containing the listing was deleted or cleaned between listing generation and task execution; a hand-built job pointing at a stale or mistyped listing path; the listing sits on a filesystem currently unreachable from the tasks.

Common situations: Automated cleanup of _distcp* staging dirs while the job is alive; a job retried long after submission with an expired listing; cross-cluster jobs where the listing path is on a nameservice the mappers cannot reach.

Related errors

• Couldn't find listing-file at: {listingFilePath}
• key + ": No such file or directory."
• Path points to dir not a file
• Configuration file not found at {confFile}
• Index file for the log of " + taskid + " doesn't exist.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b8d2c95db494a14a. Report an issue: GitHub.