{"record":{"id":"4eac66a86a442d2a","repo":"apache/hadoop","slug":"md5-file-at-references-file-named-but-we-exp","errorCode":null,"errorMessage":"MD5 file at {} references file named {} but we expected it to reference {}","messagePattern":"MD5 file at (.+?) references file named (.+?) but we expected it to reference (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java","lineNumber":116,"sourceCode":"  /**\n   * Read the md5 checksum stored alongside the given data file.\n   * @param dataFile the file containing data\n   * @return the checksum stored in dataFile.md5\n   */\n  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {\n    final File md5File = getDigestFileForFile(dataFile);\n    if (!md5File.exists()) {\n      return null;\n    }\n\n    final Matcher matcher = readStoredMd5(md5File);\n    String storedHash = matcher.group(1);\n    File referencedFile = new File(matcher.group(2));\n\n    // Sanity check: Make sure that the file referenced in the .md5 file at\n    // least has the same name as the file we expect\n    if (!referencedFile.getName().equals(dataFile.getName())) {\n      throw new IOException(\n          \"MD5 file at \" + md5File + \" references file named \" +\n          referencedFile.getName() + \" but we expected it to reference \" +\n          dataFile);\n    }\n    return new MD5Hash(storedHash);\n  }\n  \n  /**\n   * Read dataFile and compute its MD5 checksum.\n   */\n  public static MD5Hash computeMd5ForFile(File dataFile) throws IOException {\n    InputStream in = Files.newInputStream(dataFile.toPath());\n    try {\n      MessageDigest digester = MD5Hash.getDigester();\n      DigestInputStream dis = new DigestInputStream(in, digester);\n      IOUtils.copyBytes(dis, new IOUtils.NullOutputStream(), 128*1024);\n      \n      return new MD5Hash(digester.digest());","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java#L98-L134","documentation":"After parsing a sidecar, MD5FileUtils sanity-checks that the filename written inside the .md5 matches the data file's own name; a mismatch throws this IOException showing 'references file named X but we expected it to reference Y'. The guard prevents a sidecar copied next to the wrong artifact from silently validating it.","triggerScenarios":"readStoredMd5ForFile (and thus verifySavedMD5) where X.md5 contains 'hash  Y' with Y != X's basename: data file renamed without updating its sidecar, sidecar copied from another checkpoint generation (e.g., fsimage_...019.md5 next to fsimage_...020), or a sidecar generated with a full path of a differently named file.","commonSituations":"Operators archiving fsimages under new names; checkpoint promotion/restores into differently named files; FSImage.renameMD5File misuse where the new name was not propagated to sidecar contents.","solutions":["Make the reference match: rewrite the sidecar's first line as '<same hash>  <correct data file name>'.","Or regenerate the sidecar entirely from the current data file (md5sum or MD5FileUtils.saveMD5File).","When renaming data files programmatically, use MD5FileUtils.renameMD5File(old, new) - it rewrites the referenced filename correctly (see 3259 for its precondition).","Audit mixed-generation storage directories and re-pair each data file with its own sidecar."],"exampleFix":"# before: fsimage_0000000000000000020.md5 contains\n#   8a5f0f1e...c3  fsimage_0000000000000000019\n# -> references file named fsimage_...019 but we expected ...020\n\n# after: repoint the sidecar at the correct file (hash unchanged)\necho \"8a5f0f1e...c3  fsimage_0000000000000000020\" > fsimage_0000000000000000020.md5","handlingStrategy":"validation","validationCode":"// confirm the sidecar's referenced name before triggering verification\nMatcher m = readSidecar(md5File); // hash + referenced filename\nif (!m.group(2).endsWith(dataFile.getName())) {\n  LOG.warn(\"Sidecar {} references {} - repoint or regenerate before verify\",\n      md5File, m.group(2));\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Rename data files only through MD5FileUtils.renameMD5File, which rewrites the referenced name.","Never mix fsimage/checkpoint generations across directories - copy data+sidecar pairs intact.","When archiving under new names, regenerate sidecars rather than editing hashes."],"tags":["md5","checksum","file-naming","fsimage","sidecar"],"backgroundTag":"checksum-file-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}