{"record":{"id":"c40b10b00171e2a3","repo":"apache/hadoop","slug":"expected-the-file-renamed-from-s-with-length-d-b","errorCode":null,"errorMessage":"Expected the file renamed from %s with length %d but found a file of length %d","messagePattern":"Expected the file renamed from (.+?) with length (.+?) but found a file of length (.+?)","errorType":"exception","errorClass":"OutputValidationException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java","lineNumber":165,"sourceCode":"              destPath, destStatus, entry);\n          throw new OutputValidationException(destPath,\n              String.format(\"Expected the file\"\n                      + \" renamed from %s\"\n                      + \" with etag %s and length %s\"\n                      + \" but found a file with etag %s and length %d\",\n                  sourcePath,\n                  sourceEtag,\n                  sourceSize,\n                  destEtag,\n                  destSize));\n\n        }\n      }\n      // check the expected length after any etag validation\n      if (destSize != sourceSize) {\n        LOG.warn(\"Length of dest file {}: {} does not match that of manifest entry {}\",\n            destPath, destStatus, entry);\n        throw new OutputValidationException(destPath,\n            String.format(\"Expected the file\"\n                    + \" renamed from %s\"\n                    + \" with length %d\"\n                    + \" but found a file of length %d\",\n                sourcePath,\n                sourceSize,\n                destSize));\n      }\n\n    } catch (FileNotFoundException e) {\n      // file didn't exist\n      throw new OutputValidationException(destPath,\n          \"Expected a file, but it was not found\", e);\n    }\n    addFileCommitted(entry);\n  }\n\n}","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java#L147-L183","documentation":"The manifest committer's optional validation stage compares each committed file's length on the destination against the size recorded in the task manifest (after any etag check). A difference throws OutputValidationException('Expected the file renamed from %s with length %d but found a file of length %d'): the destination is longer or shorter than the file that was task-committed, indicating a partial rename, truncation, or overwrite.","triggerScenarios":"validate.output enabled and destStatus.getLen() != entry.getSize(): the destination file was appended to, truncated, partially written by a failed copy/rename, or replaced by different-length content between task commit and validation.","commonSituations":"Another writer appending to output files (e.g. a downstream job started early); store-level rename/copy anomalies leaving a partial object; the same file committed twice via overlapping manifests; external tooling rewriting output files.","solutions":["Compare the two lengths in the message: longer usually means a concurrent appender, shorter usually means a partial write/copy.","Remove the inconsistent destination and rerun the job so manifests and files are committed atomically again.","Prevent readers/writers from touching the output tree until job commit completes.","If partial writes recur, investigate the store's rename durability (check store-operations implementation and FS health)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (OutputValidationException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"with length\")) {\n    // dest size != manifest size: check for appenders/partial renames,\n    // remove the inconsistent file, rerun the job\n  }\n}","preventionTips":["Start downstream consumers only after successful job commit, never during it.","Watch for duplicate task attempts committing the same entries (speculative execution settings).","If lengths mismatch repeatedly, audit the store's rename durability."],"tags":["hadoop","manifest-committer","file-size","data-integrity","output-validation"],"backgroundTag":"file-size-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}