apache/hadoop · error · OutputValidationException

Expected the file renamed from %s with length %d but found a

Error message

Expected the file renamed from %s with length %d but found a file of length %d

What it means

The manifest committer's optional validation stage compares each committed file's length on the destination against the size recorded in the task manifest (after any etag check). A difference throws OutputValidationException('Expected the file renamed from %s with length %d but found a file of length %d'): the destination is longer or shorter than the file that was task-committed, indicating a partial rename, truncation, or overwrite.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java:165

              destPath, destStatus, entry);
          throw new OutputValidationException(destPath,
              String.format("Expected the file"
                      + " renamed from %s"
                      + " with etag %s and length %s"
                      + " but found a file with etag %s and length %d",
                  sourcePath,
                  sourceEtag,
                  sourceSize,
                  destEtag,
                  destSize));

        }
      }
      // check the expected length after any etag validation
      if (destSize != sourceSize) {
        LOG.warn("Length of dest file {}: {} does not match that of manifest entry {}",
            destPath, destStatus, entry);
        throw new OutputValidationException(destPath,
            String.format("Expected the file"
                    + " renamed from %s"
                    + " with length %d"
                    + " but found a file of length %d",
                sourcePath,
                sourceSize,
                destSize));
      }

    } catch (FileNotFoundException e) {
      // file didn't exist
      throw new OutputValidationException(destPath,
          "Expected a file, but it was not found", e);
    }
    addFileCommitted(entry);
  }

}

View on GitHub (pinned to 2add963021)

Solutions

Compare the two lengths in the message: longer usually means a concurrent appender, shorter usually means a partial write/copy.
Remove the inconsistent destination and rerun the job so manifests and files are committed atomically again.
Prevent readers/writers from touching the output tree until job commit completes.
If partial writes recur, investigate the store's rename durability (check store-operations implementation and FS health).

Defensive patterns

Strategy: try-catch

Try / catch

try {
  job.waitForCompletion(true);
} catch (OutputValidationException e) {
  if (e.getMessage() != null && e.getMessage().contains("with length")) {
    // dest size != manifest size: check for appenders/partial renames,
    // remove the inconsistent file, rerun the job
  }
}

Prevention

Start downstream consumers only after successful job commit, never during it.
Watch for duplicate task attempts committing the same entries (speculative execution settings).
If lengths mismatch repeatedly, audit the store's rename durability.

When it happens

Trigger: validate.output enabled and destStatus.getLen() != entry.getSize(): the destination file was appended to, truncated, partially written by a failed copy/rename, or replaced by different-length content between task commit and validation.

Common situations: Another writer appending to output files (e.g. a downstream job started early); store-level rename/copy anomalies leaving a partial object; the same file committed twice via overlapping manifests; external tooling rewriting output files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c40b10b00171e2a3. Report an issue: GitHub.