{"record":{"id":"e95be3b33c78f0fd","repo":"apache/hadoop","slug":"part-size-mismatched-d-d","errorCode":null,"errorMessage":"part size mismatched: %d != %d","messagePattern":"part size mismatched: (.+?) != (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java","lineNumber":473,"sourceCode":"\n    return getFileChecksum(keyPath);\n  }\n\n  private byte[] getFileChecksum(Path keyPath) {\n    return getFileMD5(keyPath);\n  }\n\n  private static byte[] getFileMD5(Path keyPath) {\n    try {\n      return DigestUtils.md5(Files.readAllBytes(keyPath));\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  private static void checkPartFile(Part part, File partFile) throws IOException {\n    if (part.size() != partFile.length()) {\n      throw new RuntimeException(String.format(\"part size mismatched: %d != %d\",\n          part.size(), partFile.length()));\n    }\n\n    try (FileInputStream inputStream = new FileInputStream(partFile)) {\n      String md5Hex = DigestUtils.md5Hex(inputStream);\n      if (!Objects.equals(part.eTag(), md5Hex)) {\n        throw new RuntimeException(String.format(\"part etag mismatched: %s != %s\",\n            part.eTag(), md5Hex));\n      }\n    }\n  }\n\n  private List<Integer> listPartNums(File uploadDir) {\n    try (Stream<Path> stream = Files.list(uploadDir.toPath())) {\n      return stream\n          .map(f -> Integer.valueOf(f.toFile().getName()))\n          .collect(Collectors.toList());\n    } catch (IOException e) {","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java#L455-L491","documentation":"FileStore.checkPartFile() validates every staged part during completeUpload: if the size recorded in the caller's Part object differs from the actual on-disk part file length it throws 'part size mismatched'. The Part metadata you passed in and the bytes staged under __STAGING__ disagree, so the object would be assembled from unverified data.","triggerScenarios":"completeUpload where a Part.size() does not match its staged file: uploadPartCopy range arithmetic producing a declared size of (end - start + 1) while a different number of bytes was staged, a truncated/extended part file from a failed or retried upload, or hand-constructed Part objects with wrong sizes.","commonSituations":"Off-by-one errors in copySourceRangeEnd (inclusive end vs exclusive length); retrying uploadPart after a partial failure where the part file already contains some bytes (append path); mixing Part metadata from a different upload attempt with the current staging directory.","solutions":["Never construct Part objects by hand — use only the Part instances returned by uploadPart()/uploadPartCopy(), whose size/eTag reflect what was staged","For uploadPartCopy, verify range math: staged size is copySourceRangeEnd - copySourceRangeStart + 1 (inclusive end), clamped to the source object length via head(srcKey)","If any part upload was retried after a partial write, abort the upload and redo all parts under a new uploadId","Ensure nothing truncates or appends to files inside __STAGING__ between upload and complete"],"exampleFix":"// before: off-by-one copy range (exclusive end assumed)\nlong end = start + len; // declared size ends up len+1\nPart p = storage.uploadPartCopy(srcKey, dstKey, uploadId, n, start, end); // part size mismatched\n// after: derive the inclusive end from the real source length\nObjectInfo src = storage.head(srcKey);\nlong last = Math.min(start + len - 1, src.size() - 1);\nPart p = storage.uploadPartCopy(srcKey, dstKey, uploadId, n, start, last);","handlingStrategy":"validation","validationCode":"ObjectInfo src = storage.head(srcKey);\nlong last = Math.min(copySourceRangeEnd, src.size() - 1);\nif (last < copySourceRangeStart) {\n  throw new IOException(\"empty or inverted copy range: [\" + copySourceRangeStart + \",\" + last + \"]\");\n}\nlong expected = last - copySourceRangeStart + 1;\nPart p = storage.uploadPartCopy(srcKey, dstKey, uploadId, partNum, copySourceRangeStart, last);\nif (p.size() != expected) {\n  throw new IOException(\"part \" + partNum + \" size \" + p.size() + \" != expected \" + expected);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat copySourceRangeEnd as INCLUSIVE and clamp it to head(srcKey).size() - 1 before calling uploadPartCopy","Use only Part objects returned by uploadPart/uploadPartCopy — never rebuild them with your own size math","Never retry a part upload against the same part number after a partial failure; start a fresh upload instead","Add boundary tests for 0-length and end-of-object ranges around uploadPartCopy"],"tags":["hadoop","tos","filestore","multipart-upload","data-integrity"],"backgroundTag":"multipart-parts-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}