{"record":{"id":"8aa5fb6cfa9fd230","repo":"apache/hadoop","slug":"part-etag-mismatched-s-s","errorCode":null,"errorMessage":"part etag mismatched: %s != %s","messagePattern":"part etag mismatched: (.+?) != (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java","lineNumber":480,"sourceCode":"\n  private static byte[] getFileMD5(Path keyPath) {\n    try {\n      return DigestUtils.md5(Files.readAllBytes(keyPath));\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  private static void checkPartFile(Part part, File partFile) throws IOException {\n    if (part.size() != partFile.length()) {\n      throw new RuntimeException(String.format(\"part size mismatched: %d != %d\",\n          part.size(), partFile.length()));\n    }\n\n    try (FileInputStream inputStream = new FileInputStream(partFile)) {\n      String md5Hex = DigestUtils.md5Hex(inputStream);\n      if (!Objects.equals(part.eTag(), md5Hex)) {\n        throw new RuntimeException(String.format(\"part etag mismatched: %s != %s\",\n            part.eTag(), md5Hex));\n      }\n    }\n  }\n\n  private List<Integer> listPartNums(File uploadDir) {\n    try (Stream<Path> stream = Files.list(uploadDir.toPath())) {\n      return stream\n          .map(f -> Integer.valueOf(f.toFile().getName()))\n          .collect(Collectors.toList());\n    } catch (IOException e) {\n      LOG.error(\"failed to list part files.\");\n      throw new RuntimeException(e);\n    }\n  }\n\n  @Override\n  public void abortMultipartUpload(String key, String uploadId) {","sourceCodeStart":462,"sourceCodeEnd":498,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java#L462-L498","documentation":"FileStore.checkPartFile() recomputes the MD5 of every staged part file during completeUpload and compares it to the eTag recorded in the caller's Part; a mismatch throws 'part etag mismatched'. The bytes on disk no longer hash to what was reported at upload time, meaning the staged part was corrupted, overwritten, or its metadata comes from a different attempt. The filestore only supports the MD5 checksum type (enforced at initialize).","triggerScenarios":"completeUpload where a part file inside __STAGING__ changed after uploadPart() computed its MD5 — disk corruption, another process/writer touching the staging dir, a retried part upload appending to an existing part file, or Part objects reused from an earlier upload attempt.","commonSituations":"Test cleanup or parallel tests mutating the shared staging tree; storage-bit-rot on flaky tmp mounts; mixing Part results across retry attempts; accidentally configuring a non-MD5 checksum expectation elsewhere and reconciling tags manually.","solutions":["Abort the upload (abortMultipartUpload) and redo all parts under a fresh uploadId, using only the newly returned Part objects","Verify nothing external modifies <root>/__STAGING__ during uploads: run flaky-test staging on local disk, isolate roots per test","Check filesystem health if parts change with no concurrent writer (disk errors on the volume backing the store)","Keep fs.filestore checksum type at the default MD5 — other types are rejected at initialize() anyway"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// verify your own view of each part before completing\nbyte[] buf = payloadFor(partNum);\nPart p = storage.uploadPart(key, uploadId, partNum,\n    () -> new java.io.ByteArrayInputStream(buf), buf.length);\nif (!org.apache.commons.codec.digest.DigestUtils.md5Hex(buf).equals(p.eTag())) {\n  throw new IOException(\"part \" + partNum + \" eTag drift; staging data is unreliable\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use the eTag returned by uploadPart as the single source of truth; never mix Part metadata across retry attempts","Isolate the __STAGING__ tree per test/process so nothing external mutates staged parts","Keep the filestore root on healthy local disk; investigate flaky mounts when checksums drift with no concurrent writer","Abort and fully redo an upload (new uploadId) rather than patching a mismatched part"],"tags":["hadoop","tos","filestore","multipart-upload","checksum","data-integrity"],"backgroundTag":"checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}