{"record":{"id":"078135ff49238ac7","repo":"apache/hadoop","slug":"expected-the-file-renamed-from-s-with-etag-s-and","errorCode":null,"errorMessage":"Expected the file renamed from %s with etag %s and length %s but found a file with etag %s and length %d","messagePattern":"Expected the file renamed from (.+?) with etag (.+?) and length (.+?) but found a file with etag (.+?) and length (.+?)","errorType":"exception","errorClass":"OutputValidationException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java","lineNumber":148,"sourceCode":"\n      // it must be a file\n      if (!destStatus.isFile()) {\n        throw new OutputValidationException(destPath,\n            \"Expected a file renamed from \" + sourcePath\n                + \"; found \" + destStatus);\n      }\n      final long sourceSize = entry.getSize();\n      final long destSize = destStatus.getLen();\n\n      // etags, if the source had one.\n      final String sourceEtag = entry.getEtag();\n      if (getOperations().storePreservesEtagsThroughRenames(destStatus.getPath())\n          && isNotBlank(sourceEtag)) {\n        final String destEtag = ManifestCommitterSupport.getEtag(destStatus);\n        if (!sourceEtag.equals(destEtag)) {\n          LOG.warn(\"Etag of dest file {}: {} does not match that of manifest entry {}\",\n              destPath, destStatus, entry);\n          throw new OutputValidationException(destPath,\n              String.format(\"Expected the file\"\n                      + \" renamed from %s\"\n                      + \" with etag %s and length %s\"\n                      + \" but found a file with etag %s and length %d\",\n                  sourcePath,\n                  sourceEtag,\n                  sourceSize,\n                  destEtag,\n                  destSize));\n\n        }\n      }\n      // check the expected length after any etag validation\n      if (destSize != sourceSize) {\n        LOG.warn(\"Length of dest file {}: {} does not match that of manifest entry {}\",\n            destPath, destStatus, entry);\n        throw new OutputValidationException(destPath,\n            String.format(\"Expected the file\"","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java#L130-L166","documentation":"In the manifest committer's optional validation stage, when the store preserves etags through renames (e.g. ABFS) and the manifest entry recorded a source etag, the destination file's etag is compared against it. A mismatch throws OutputValidationException('Expected the file renamed from %s with etag %s and length %s but found a file with etag %s and length %d'): the object now sitting at the destination is not the object that was renamed - it was overwritten or replaced after task commit.","triggerScenarios":"validate.output=true, storePreservesEtagsThroughRenames(destPath)==true, entry.getEtag() non-blank, and ManifestCommitterSupport.getEtag(destStatus) differs from the manifest's source etag - i.e. destination content changed between task commit and job-commit validation.","commonSituations":"A second job/worker overwrote the destination file between task commit and job commit; manual 'repairs' to output data; overlapping partitions between two pipelines committing to the same tree.","solutions":["Compare the two etags in the message to confirm the destination was replaced, then find the concurrent writer (job logs, access times) and stop the overlap.","Delete the conflicting destination and rerun the job to re-commit a consistent set of files.","Give each run an exclusive output path/partition set.","If this fires spuriously on your store, verify your store-operations etag handling (storePreservesEtagsThroughRenames must only be true where renames really preserve etags)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (OutputValidationException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"etag\")) {\n    // destination was replaced after task commit: find the concurrent writer,\n    // delete the foreign object, rerun the job into an exclusive output path\n  }\n}","preventionTips":["Quarantine output trees from other writers until commitJob completes.","Only rely on etag validation on stores whose renames truly preserve etags (ABFS), via a matching store-operations class.","Treat etag mismatch as data-lineage damage: investigate, never blindly delete."],"tags":["hadoop","manifest-committer","etag","data-integrity","output-validation"],"backgroundTag":"etag-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}