apache/hadoop · error · OutputValidationException

Expected the file renamed from %s with etag %s and length %s

Error message

Expected the file renamed from %s with etag %s and length %s but found a file with etag %s and length %d

What it means

In the manifest committer's optional validation stage, when the store preserves etags through renames (e.g. ABFS) and the manifest entry recorded a source etag, the destination file's etag is compared against it. A mismatch throws OutputValidationException('Expected the file renamed from %s with etag %s and length %s but found a file with etag %s and length %d'): the object now sitting at the destination is not the object that was renamed - it was overwritten or replaced after task commit.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/ValidateRenamedFilesStage.java:148

      // it must be a file
      if (!destStatus.isFile()) {
        throw new OutputValidationException(destPath,
            "Expected a file renamed from " + sourcePath
                + "; found " + destStatus);
      }
      final long sourceSize = entry.getSize();
      final long destSize = destStatus.getLen();

      // etags, if the source had one.
      final String sourceEtag = entry.getEtag();
      if (getOperations().storePreservesEtagsThroughRenames(destStatus.getPath())
          && isNotBlank(sourceEtag)) {
        final String destEtag = ManifestCommitterSupport.getEtag(destStatus);
        if (!sourceEtag.equals(destEtag)) {
          LOG.warn("Etag of dest file {}: {} does not match that of manifest entry {}",
              destPath, destStatus, entry);
          throw new OutputValidationException(destPath,
              String.format("Expected the file"
                      + " renamed from %s"
                      + " with etag %s and length %s"
                      + " but found a file with etag %s and length %d",
                  sourcePath,
                  sourceEtag,
                  sourceSize,
                  destEtag,
                  destSize));

        }
      }
      // check the expected length after any etag validation
      if (destSize != sourceSize) {
        LOG.warn("Length of dest file {}: {} does not match that of manifest entry {}",
            destPath, destStatus, entry);
        throw new OutputValidationException(destPath,
            String.format("Expected the file"

View on GitHub (pinned to 2add963021)

Solutions

Compare the two etags in the message to confirm the destination was replaced, then find the concurrent writer (job logs, access times) and stop the overlap.
Delete the conflicting destination and rerun the job to re-commit a consistent set of files.
Give each run an exclusive output path/partition set.
If this fires spuriously on your store, verify your store-operations etag handling (storePreservesEtagsThroughRenames must only be true where renames really preserve etags).

Defensive patterns

Strategy: try-catch

Try / catch

try {
  job.waitForCompletion(true);
} catch (OutputValidationException e) {
  if (e.getMessage() != null && e.getMessage().contains("etag")) {
    // destination was replaced after task commit: find the concurrent writer,
    // delete the foreign object, rerun the job into an exclusive output path
  }
}

Prevention

Quarantine output trees from other writers until commitJob completes.
Only rely on etag validation on stores whose renames truly preserve etags (ABFS), via a matching store-operations class.
Treat etag mismatch as data-lineage damage: investigate, never blindly delete.

When it happens

Trigger: validate.output=true, storePreservesEtagsThroughRenames(destPath)==true, entry.getEtag() non-blank, and ManifestCommitterSupport.getEtag(destStatus) differs from the manifest's source etag - i.e. destination content changed between task commit and job-commit validation.

Common situations: A second job/worker overwrote the destination file between task commit and job commit; manual 'repairs' to output data; overlapping partitions between two pipelines committing to the same tree.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/078135ff49238ac7. Report an issue: GitHub.