{"record":{"id":"cc0d2c76ebdf087a","repo":"apache/hadoop","slug":"expected-a-file-but-it-was-not-found","errorCode":null,"errorMessage":"Expected a file, but it was not found","messagePattern":"Expected a file, but it was not found","errorType":"exception","errorClass":"OutputValidationException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java","lineNumber":177,"sourceCode":"        }\n      }\n      // check the expected length after any etag validation\n      if (destSize != sourceSize) {\n        LOG.warn(\"Length of dest file {}: {} does not match that of manifest entry {}\",\n            destPath, destStatus, entry);\n        throw new OutputValidationException(destPath,\n            String.format(\"Expected the file\"\n                    + \" renamed from %s\"\n                    + \" with length %d\"\n                    + \" but found a file of length %d\",\n                sourcePath,\n                sourceSize,\n                destSize));\n      }\n\n    } catch (FileNotFoundException e) {\n      // file didn't exist\n      throw new OutputValidationException(destPath,\n          \"Expected a file, but it was not found\", e);\n    }\n    addFileCommitted(entry);\n  }\n\n}\n","sourceCodeStart":159,"sourceCodeEnd":184,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java#L159-L184","documentation":"During the manifest committer's optional validation stage, getFileStatus(destPath) for a manifest entry can throw FileNotFoundException; the stage converts it into OutputValidationException('Expected a file, but it was not found', e). The file the manifest says was renamed into the final output is simply absent at validation time - deleted after task commit, or the rename never actually landed.","triggerScenarios":"validate.output enabled and the destination of a task-committed file no longer exists when the job commit validates it: someone deleted output between task and job commit, a task's rename silently failed (non-atomic or eventually-consistent store), or manifests reference destinations outside the real output tree.","commonSituations":"Downstream consumers or cleanup scripts deleting files before the job finishes; tasks reporting success while the store lost the rename; misconfigured dest paths in custom store-operations implementations; retries creating manifests referencing paths that a later attempt removed.","solutions":["Check whether the dest path from the exception exists now; if a human/process deleted it, stop that and rerun the job.","Review task-side commit logs for rename failures for the affected entries - a missing file usually traces back to a task that 'succeeded' with a lost write.","Keep the output tree untouched until commitJob finishes.","If the store is eventually consistent, verify the store-operations implementation is one built for that filesystem."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (OutputValidationException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"not found\")) {\n    // committed file vanished: check who deleted it, review task rename logs,\n    // then rerun the job to rebuild a consistent output\n  }\n}","preventionTips":["Freeze the output tree (no deletes, no readers' cleanup) between task commit and job commit.","Verify tasks' rename success in logs when validating output on eventually-consistent stores.","Use the store-operations implementation built for your filesystem."],"tags":["hadoop","manifest-committer","file-not-found","output-validation","data-loss"],"backgroundTag":"committed-file-missing","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}