What causes "Multipart part ID mismatch: " + uploadId" in apache/hadoop?

Scope every PartHandle to the single UploadHandle returned by initiate(): on any retry, discard old parts, abort the old upload, and re-upload all parts under the new session.

apache/hadoop · error · PathIOException

Multipart part ID mismatch: " + uploadId

Error message

Multipart part ID mismatch: " + uploadId

What it means

Each S3A part handle stores the multipart uploadId it was uploaded under; PartHandlePayload.validate() throws PathIOException("Multipart part ID mismatch: ...") when that id differs from the UploadHandle passed to complete(). AWS would reject such parts anyway (NoSuchUpload), so S3A fails fast with a clear message.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AMultipartUploader.java:467

        output.writeLong(len);
        output.writeUTF(etag);
        if (checksumAlgorithm != null && checksum != null) {
          output.writeUTF(checksumAlgorithm);
          output.writeUTF(checksum);
        }
      }
      return bytes.toByteArray();
    }

    public void validate(String uploadIdStr, Path filePath)
        throws PathIOException {
      String destUri = filePath.toUri().toString();
      if (!destUri.equals(path)) {
        throw new PathIOException(destUri,
            "Multipart part path mismatch: " + path);
      }
      if (!uploadIdStr.equals(uploadId)) {
        throw new PathIOException(destUri,
            "Multipart part ID mismatch: " + uploadId);
      }
    }
  }


}

View on GitHub (pinned to 2add963021)

Solutions

Scope every PartHandle to the single UploadHandle returned by initiate(): on any retry, discard old parts, abort the old upload, and re-upload all parts under the new session.
In committer/framework code, key in-flight handles by uploadId so stale attempts cannot be merged into a new completion.
Catch PathIOException around complete(); on mismatch, abort and restart the multipart upload.
Verify with getS3AUploadId-style logging (or your own map) that part count per uploadId matches before completing.

Example fix

// before
UploadHandle uh1 = uploader.initialize(path, ...);
PartHandle ph1 = uploader.upload(path, uh1, data, 1, false);
// retry logic re-initializes:
UploadHandle uh2 = uploader.initialize(path, ...);
uploader.complete(path, uh2, List.of(ph1)); // boom: part belongs to uh1

// after: on retry, discard parts from the old session
Map<UploadHandle, List<PartHandle>> sessions = new HashMap<>();
sessions.computeIfAbsent(uh2, k -> new ArrayList<>()).add(
    uploader.upload(path, uh2, data, 1, false));
uploader.complete(path, uh2, sessions.get(uh2));

Defensive patterns

Strategy: validation

Validate before calling

// group parts strictly by their originating upload session
Map<UploadHandle, List<PartHandle>> bySession = new HashMap<>();
bySession.computeIfAbsent(uploadHandle, k -> new ArrayList<>())
    .add(partHandleFromThatSession);
// complete with only parts of THIS session
uploader.complete(path, uploadHandle, bySession.get(uploadHandle));

Try / catch

try {
  uploader.complete(path, uploadHandle, parts);
} catch (PathIOException e) {
  if (e.getMessage().contains("part ID mismatch")) {
    uploader.abort(uploadHandle);
    restartUploadWithFreshParts();
  } else { throw e; }
}

Prevention

On any attempt failure/retry, abort the old upload and discard its part handles.
Key in-flight state by uploadId so stale attempts cannot merge into a new completion.
Unit-test the retry path of your upload pipeline for handle/session consistency.

When it happens

Trigger: Mixing PartHandles from upload session X into complete() for UploadHandle Y (e.g. after a task retry re-initiated the upload); complete() called with an UploadHandle from initiate() but part handles collected from an earlier aborted attempt; passing handles between different uploader instances/sessions.

Common situations: Job/task retries where the driver re-initializes the multipart upload but old part handles from the previous attempt are still in the completion list; custom committers persisting handles across application attempts; concurrent writers sharing handle collections.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cf92af76c823144b. Report an issue: GitHub.