{"record":{"id":"5d45f33e80dd0a15","repo":"apache/hadoop","slug":"expected-a-file-renamed-from-sourcepath-found","errorCode":null,"errorMessage":"Expected a file renamed from {sourcePath}; found {destStatus}","messagePattern":"Expected a file renamed from (.+?); found (.+?)","errorType":"exception","errorClass":"OutputValidationException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java","lineNumber":133,"sourceCode":"   * @throws IOException IO problem.\n   * @throws OutputValidationException if the entry is not valid\n   */\n  private void validateOneFile(FileEntry entry) throws IOException {\n    updateAuditContext(OP_STAGE_JOB_VALIDATE_OUTPUT);\n\n    // report progress back\n    progress();\n    // look validate the file.\n    // raising an FNFE if the file isn't there.\n    FileStatus destStatus;\n    final Path sourcePath = entry.getSourcePath();\n    Path destPath = entry.getDestPath();\n    try {\n      destStatus = getFileStatus(destPath);\n\n      // it must be a file\n      if (!destStatus.isFile()) {\n        throw new OutputValidationException(destPath,\n            \"Expected a file renamed from \" + sourcePath\n                + \"; found \" + destStatus);\n      }\n      final long sourceSize = entry.getSize();\n      final long destSize = destStatus.getLen();\n\n      // etags, if the source had one.\n      final String sourceEtag = entry.getEtag();\n      if (getOperations().storePreservesEtagsThroughRenames(destStatus.getPath())\n          && isNotBlank(sourceEtag)) {\n        final String destEtag = ManifestCommitterSupport.getEtag(destStatus);\n        if (!sourceEtag.equals(destEtag)) {\n          LOG.warn(\"Etag of dest file {}: {} does not match that of manifest entry {}\",\n              destPath, destStatus, entry);\n          throw new OutputValidationException(destPath,\n              String.format(\"Expected the file\"\n                      + \" renamed from %s\"\n                      + \" with etag %s and length %s\"","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java#L115-L151","documentation":"ValidateRenamedFilesStage is the optional post-rename validation phase of the manifest committer job commit (enabled by mapreduce.manifest.committer.validate.output, default false). For each manifest entry it requires the destination to be a regular file; if getFileStatus() succeeds but destStatus.isFile() is false it throws OutputValidationException('Expected a file renamed from <sourcePath>; found <destStatus>') - typically a directory now occupies the destination.","triggerScenarios":"Validation stage runs, and for some entry the destination path resolves to a directory: another job committed a partition directory over the file destination, or the dest path structure collides with a directory created between task commit and job commit.","commonSituations":"Concurrent jobs writing the same output tree/partitions; partition layout conflicts where a file and a directory claim the same path; running with validate.output=true while sharing output with other writers.","solutions":["Identify the conflicting directory from the printed status and remove/relocate it, then re-commit or rerun.","Guarantee exclusive access to the output tree - one committing job at a time.","If validation is not required for this workflow, leave mapreduce.manifest.committer.validate.output at its default false.","Review the destPath pattern in the manifest entry against your partition layout to find the structural collision."],"exampleFix":"// before: validation enabled on a shared output tree\nconf.setBoolean(\"mapreduce.manifest.committer.validate.output\", true);\n\n// after: only validate when the output tree is exclusively yours\nconf.setBoolean(\"mapreduce.manifest.committer.validate.output\", false);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (OutputValidationException e) {\n  // validation-only failure with validate.output=true:\n  // inspect the entry's destPath, remove the conflicting directory, re-commit or rerun\n}","preventionTips":["Leave mapreduce.manifest.committer.validate.output=false unless you act on its failures.","Ensure partition layouts cannot make a file and a directory claim the same path.","Serialize committing jobs per output tree."],"tags":["hadoop","manifest-committer","output-validation","job-commit","path-conflict"],"backgroundTag":"output-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}