What causes "Expected a file, but it was not found" in apache/hadoop?

Check whether the dest path from the exception exists now; if a human/process deleted it, stop that and rerun the job.

apache/hadoop · error · OutputValidationException

Expected a file, but it was not found

Error message

Expected a file, but it was not found

What it means

During the manifest committer's optional validation stage, getFileStatus(destPath) for a manifest entry can throw FileNotFoundException; the stage converts it into OutputValidationException('Expected a file, but it was not found', e). The file the manifest says was renamed into the final output is simply absent at validation time - deleted after task commit, or the rename never actually landed.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java:177

        }
      }
      // check the expected length after any etag validation
      if (destSize != sourceSize) {
        LOG.warn("Length of dest file {}: {} does not match that of manifest entry {}",
            destPath, destStatus, entry);
        throw new OutputValidationException(destPath,
            String.format("Expected the file"
                    + " renamed from %s"
                    + " with length %d"
                    + " but found a file of length %d",
                sourcePath,
                sourceSize,
                destSize));
      }

    } catch (FileNotFoundException e) {
      // file didn't exist
      throw new OutputValidationException(destPath,
          "Expected a file, but it was not found", e);
    }
    addFileCommitted(entry);
  }

}

View on GitHub (pinned to 2add963021)

Solutions

Check whether the dest path from the exception exists now; if a human/process deleted it, stop that and rerun the job.
Review task-side commit logs for rename failures for the affected entries - a missing file usually traces back to a task that 'succeeded' with a lost write.
Keep the output tree untouched until commitJob finishes.
If the store is eventually consistent, verify the store-operations implementation is one built for that filesystem.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  job.waitForCompletion(true);
} catch (OutputValidationException e) {
  if (e.getMessage() != null && e.getMessage().contains("not found")) {
    // committed file vanished: check who deleted it, review task rename logs,
    // then rerun the job to rebuild a consistent output
  }
}

Prevention

Freeze the output tree (no deletes, no readers' cleanup) between task commit and job commit.
Verify tasks' rename success in logs when validating output on eventually-consistent stores.
Use the store-operations implementation built for your filesystem.

When it happens

Trigger: validate.output enabled and the destination of a task-committed file no longer exists when the job commit validates it: someone deleted output between task and job commit, a task's rename silently failed (non-atomic or eventually-consistent store), or manifests reference destinations outside the real output tree.

Common situations: Downstream consumers or cleanup scripts deleting files before the job finishes; tasks reporting success while the store lost the rename; misconfigured dest paths in custom store-operations implementations; retries creating manifests referencing paths that a later attempt removed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cc0d2c76ebdf087a. Report an issue: GitHub.