apache/hadoop · error · OutputValidationException

Expected a file renamed from {sourcePath}; found {destStatus

Error message

Expected a file renamed from {sourcePath}; found {destStatus}

What it means

ValidateRenamedFilesStage is the optional post-rename validation phase of the manifest committer job commit (enabled by mapreduce.manifest.committer.validate.output, default false). For each manifest entry it requires the destination to be a regular file; if getFileStatus() succeeds but destStatus.isFile() is false it throws OutputValidationException('Expected a file renamed from <sourcePath>; found <destStatus>') - typically a directory now occupies the destination.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java:133

   * @throws IOException IO problem.
   * @throws OutputValidationException if the entry is not valid
   */
  private void validateOneFile(FileEntry entry) throws IOException {
    updateAuditContext(OP_STAGE_JOB_VALIDATE_OUTPUT);

    // report progress back
    progress();
    // look validate the file.
    // raising an FNFE if the file isn't there.
    FileStatus destStatus;
    final Path sourcePath = entry.getSourcePath();
    Path destPath = entry.getDestPath();
    try {
      destStatus = getFileStatus(destPath);

      // it must be a file
      if (!destStatus.isFile()) {
        throw new OutputValidationException(destPath,
            "Expected a file renamed from " + sourcePath
                + "; found " + destStatus);
      }
      final long sourceSize = entry.getSize();
      final long destSize = destStatus.getLen();

      // etags, if the source had one.
      final String sourceEtag = entry.getEtag();
      if (getOperations().storePreservesEtagsThroughRenames(destStatus.getPath())
          && isNotBlank(sourceEtag)) {
        final String destEtag = ManifestCommitterSupport.getEtag(destStatus);
        if (!sourceEtag.equals(destEtag)) {
          LOG.warn("Etag of dest file {}: {} does not match that of manifest entry {}",
              destPath, destStatus, entry);
          throw new OutputValidationException(destPath,
              String.format("Expected the file"
                      + " renamed from %s"
                      + " with etag %s and length %s"

View on GitHub (pinned to 2add963021)

Solutions

1. Identify the conflicting directory from the printed status and remove/relocate it, then re-commit or rerun.
2. Guarantee exclusive access to the output tree - one committing job at a time.
3. If validation is not required for this workflow, leave mapreduce.manifest.committer.validate.output at its default false.
4. Review the destPath pattern in the manifest entry against your partition layout to find the structural collision.

Example fix

// before: validation enabled on a shared output tree
conf.setBoolean("mapreduce.manifest.committer.validate.output", true);

// after: only validate when the output tree is exclusively yours
conf.setBoolean("mapreduce.manifest.committer.validate.output", false);

try {
  job.waitForCompletion(true);
} catch (OutputValidationException e) {
  // validation-only failure with validate.output=true:
  // inspect the entry's destPath, remove the conflicting directory, re-commit or rerun
}

When it happens

Trigger: Validation stage runs, and for some entry the destination path resolves to a directory: another job committed a partition directory over the file destination, or the dest path structure collides with a directory created between task commit and job commit.

Common situations: Concurrent jobs writing the same output tree/partitions; partition layout conflicts where a file and a directory claim the same path; running with validate.output=true while sharing output with other writers.

Related errors

• {operation}: Path is not a directory; its status is :{status
• Failed to write manifest data for task {taskId}to local file
• Not a valid manifest file; file status = {status}
• Expected the file renamed from %s with etag %s and length %s
• Expected the file renamed from %s with length %d but found a

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d45f33e80dd0a15. Report an issue: GitHub.