apache/hadoop · error · PathIOException

Multipart part path mismatch: " + path

Error message

Multipart part path mismatch: " + path

What it means

Every S3A part handle embeds the destination key it was uploaded to; PartHandlePayload.validate() compares it against the path passed to complete() and throws PathIOException("Multipart part path mismatch: ...") on mismatch. The guard prevents completing an upload with parts that belong to a different destination object.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AMultipartUploader.java:463

        output.writeUTF(HEADER);
        output.writeUTF(path);
        output.writeUTF(uploadId);
        output.writeInt(partNumber);
        output.writeLong(len);
        output.writeUTF(etag);
        if (checksumAlgorithm != null && checksum != null) {
          output.writeUTF(checksumAlgorithm);
          output.writeUTF(checksum);
        }
      }
      return bytes.toByteArray();
    }

    public void validate(String uploadIdStr, Path filePath)
        throws PathIOException {
      String destUri = filePath.toUri().toString();
      if (!destUri.equals(path)) {
        throw new PathIOException(destUri,
            "Multipart part path mismatch: " + path);
      }
      if (!uploadIdStr.equals(uploadId)) {
        throw new PathIOException(destUri,
            "Multipart part ID mismatch: " + uploadId);
      }
    }
  }


}

View on GitHub (pinned to 2add963021)

Solutions

1. Complete the upload at exactly the same s3a:// destination path the parts were uploaded to; there is no rename - parts must match the final key.
2. If the destination must change, abort the upload, re-initiate at the new path, and re-upload the parts.
3. Audit custom committer code to make sure every PartHandle in the complete() list came from an upload() whose destination equals the complete() destination.
4. Catch PathIOException to detect the mismatch and trigger abort + retry of the whole upload.

Example fix

// before
try (MultipartUploader uploader = fs.createMultipartUploader()) {
  UploadHandle uh = uploader.initialize(finalPath, ...);
  PartHandle ph = uploader.upload(finalPath, uh, data, 1, false); // uploaded to finalPath
  uploader.complete(finalPath, uh, List.of(ph)); // path matches -> OK
}
// error case: uploader.complete(renamedPath, uh, List.of(ph)); // PathIOException

// enforce the invariant before completing
if (!partUploadedPath.equals(finalDestinationPath)) {
  throw new IllegalStateException(
      "Refusing to complete: part uploaded to " + partUploadedPath
      + " but completion target is " + finalDestinationPath);
}

try {
  uploader.complete(dest, uploadHandle, parts);
} catch (PathIOException e) {
  if (e.getMessage().contains("path mismatch")) {
    // parts belong elsewhere: complete at their original path or re-upload
  } else { throw e; }
}

When it happens

Trigger: Calling MultipartUploader.complete(Path A, ...) with a part handle produced while uploading to Path B; copying/renaming the intended destination between initiate() and complete(); aggregating part handles from parallel tasks that targeted different paths into one complete() call.

Common situations: Custom commit protocols or frameworks (Spark/MapReduce-style committers) that gather part handles from task attempts writing to per-attempt temporary directories and then complete at a different final path; reruns of failed tasks after the destination changed; unit tests reusing handles across paths.

Related errors

• Multipart part ID mismatch: " + uploadId
• Wrong header string: \"" + header + "\"
• Negative length
• Multipart IO request {sdkRequest} rejected {header}
• Cannot create a file whose name looks like a directory: '%s'

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a61f13b1fd866001. Report an issue: GitHub.