apache/hadoop · error · IOException

MD5 file at {} references file named {} but we expected it t

Error message

MD5 file at {} references file named {} but we expected it to reference {}

What it means

After parsing a sidecar, MD5FileUtils sanity-checks that the filename written inside the .md5 matches the data file's own name; a mismatch throws this IOException showing 'references file named X but we expected it to reference Y'. The guard prevents a sidecar copied next to the wrong artifact from silently validating it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java:116

  /**
   * Read the md5 checksum stored alongside the given data file.
   * @param dataFile the file containing data
   * @return the checksum stored in dataFile.md5
   */
  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {
    final File md5File = getDigestFileForFile(dataFile);
    if (!md5File.exists()) {
      return null;
    }

    final Matcher matcher = readStoredMd5(md5File);
    String storedHash = matcher.group(1);
    File referencedFile = new File(matcher.group(2));

    // Sanity check: Make sure that the file referenced in the .md5 file at
    // least has the same name as the file we expect
    if (!referencedFile.getName().equals(dataFile.getName())) {
      throw new IOException(
          "MD5 file at " + md5File + " references file named " +
          referencedFile.getName() + " but we expected it to reference " +
          dataFile);
    }
    return new MD5Hash(storedHash);
  }
  
  /**
   * Read dataFile and compute its MD5 checksum.
   */
  public static MD5Hash computeMd5ForFile(File dataFile) throws IOException {
    InputStream in = Files.newInputStream(dataFile.toPath());
    try {
      MessageDigest digester = MD5Hash.getDigester();
      DigestInputStream dis = new DigestInputStream(in, digester);
      IOUtils.copyBytes(dis, new IOUtils.NullOutputStream(), 128*1024);
      
      return new MD5Hash(digester.digest());

View on GitHub (pinned to 2add963021)

Solutions

1. Make the reference match: rewrite the sidecar's first line as '<same hash> <correct data file name>'.
2. Or regenerate the sidecar entirely from the current data file (md5sum or MD5FileUtils.saveMD5File).
3. When renaming data files programmatically, use MD5FileUtils.renameMD5File(old, new) - it rewrites the referenced filename correctly (see 3259 for its precondition).
4. Audit mixed-generation storage directories and re-pair each data file with its own sidecar.

Example fix

# before: fsimage_0000000000000000020.md5 contains
#   8a5f0f1e...c3  fsimage_0000000000000000019
# -> references file named fsimage_...019 but we expected ...020

# after: repoint the sidecar at the correct file (hash unchanged)
echo "8a5f0f1e...c3  fsimage_0000000000000000020" > fsimage_0000000000000000020.md5

// confirm the sidecar's referenced name before triggering verification
Matcher m = readSidecar(md5File); // hash + referenced filename
if (!m.group(2).endsWith(dataFile.getName())) {
  LOG.warn("Sidecar {} references {} - repoint or regenerate before verify",
      md5File, m.group(2));
}

When it happens

Trigger: readStoredMd5ForFile (and thus verifySavedMD5) where X.md5 contains 'hash Y' with Y != X's basename: data file renamed without updating its sidecar, sidecar copied from another checkpoint generation (e.g., fsimage_...019.md5 next to fsimage_...020), or a sidecar generated with a full path of a differently named file.

Common situations: Operators archiving fsimages under new names; checkpoint promotion/restores into differently named files; FSImage.renameMD5File misuse where the new name was not propagated to sidecar contents.

Related errors

• File " + url + " computed digest " + computedDigest + " does
• File {} did not match stored MD5 checksum (stored: {}, comp
• Wrong length: {digest.length}
• Wrong length: {hex.length}
• Directory {dir} is in an inconsistent state: Message digest

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4eac66a86a442d2a. Report an issue: GitHub.