{"record":{"id":"cf92af76c823144b","repo":"apache/hadoop","slug":"multipart-part-id-mismatch-uploadid","errorCode":null,"errorMessage":"Multipart part ID mismatch: \" + uploadId","messagePattern":"Multipart part ID mismatch: \" \\+ uploadId","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AMultipartUploader.java","lineNumber":467,"sourceCode":"        output.writeLong(len);\n        output.writeUTF(etag);\n        if (checksumAlgorithm != null && checksum != null) {\n          output.writeUTF(checksumAlgorithm);\n          output.writeUTF(checksum);\n        }\n      }\n      return bytes.toByteArray();\n    }\n\n    public void validate(String uploadIdStr, Path filePath)\n        throws PathIOException {\n      String destUri = filePath.toUri().toString();\n      if (!destUri.equals(path)) {\n        throw new PathIOException(destUri,\n            \"Multipart part path mismatch: \" + path);\n      }\n      if (!uploadIdStr.equals(uploadId)) {\n        throw new PathIOException(destUri,\n            \"Multipart part ID mismatch: \" + uploadId);\n      }\n    }\n  }\n\n\n}\n","sourceCodeStart":449,"sourceCodeEnd":475,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AMultipartUploader.java#L449-L475","documentation":"Each S3A part handle stores the multipart uploadId it was uploaded under; PartHandlePayload.validate() throws PathIOException(\"Multipart part ID mismatch: ...\") when that id differs from the UploadHandle passed to complete(). AWS would reject such parts anyway (NoSuchUpload), so S3A fails fast with a clear message.","triggerScenarios":"Mixing PartHandles from upload session X into complete() for UploadHandle Y (e.g. after a task retry re-initiated the upload); complete() called with an UploadHandle from initiate() but part handles collected from an earlier aborted attempt; passing handles between different uploader instances/sessions.","commonSituations":"Job/task retries where the driver re-initializes the multipart upload but old part handles from the previous attempt are still in the completion list; custom committers persisting handles across application attempts; concurrent writers sharing handle collections.","solutions":["Scope every PartHandle to the single UploadHandle returned by initiate(): on any retry, discard old parts, abort the old upload, and re-upload all parts under the new session.","In committer/framework code, key in-flight handles by uploadId so stale attempts cannot be merged into a new completion.","Catch PathIOException around complete(); on mismatch, abort and restart the multipart upload.","Verify with getS3AUploadId-style logging (or your own map) that part count per uploadId matches before completing."],"exampleFix":"// before\nUploadHandle uh1 = uploader.initialize(path, ...);\nPartHandle ph1 = uploader.upload(path, uh1, data, 1, false);\n// retry logic re-initializes:\nUploadHandle uh2 = uploader.initialize(path, ...);\nuploader.complete(path, uh2, List.of(ph1)); // boom: part belongs to uh1\n\n// after: on retry, discard parts from the old session\nMap<UploadHandle, List<PartHandle>> sessions = new HashMap<>();\nsessions.computeIfAbsent(uh2, k -> new ArrayList<>()).add(\n    uploader.upload(path, uh2, data, 1, false));\nuploader.complete(path, uh2, sessions.get(uh2));","handlingStrategy":"validation","validationCode":"// group parts strictly by their originating upload session\nMap<UploadHandle, List<PartHandle>> bySession = new HashMap<>();\nbySession.computeIfAbsent(uploadHandle, k -> new ArrayList<>())\n    .add(partHandleFromThatSession);\n// complete with only parts of THIS session\nuploader.complete(path, uploadHandle, bySession.get(uploadHandle));","typeGuard":null,"tryCatchPattern":"try {\n  uploader.complete(path, uploadHandle, parts);\n} catch (PathIOException e) {\n  if (e.getMessage().contains(\"part ID mismatch\")) {\n    uploader.abort(uploadHandle);\n    restartUploadWithFreshParts();\n  } else { throw e; }\n}","preventionTips":["On any attempt failure/retry, abort the old upload and discard its part handles.","Key in-flight state by uploadId so stale attempts cannot merge into a new completion.","Unit-test the retry path of your upload pipeline for handle/session consistency."],"tags":["hadoop-aws","multipart-uploader","part-handle","upload-id","committer"],"backgroundTag":"handle-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}