apache/hadoop · critical · IOException

File {} did not match stored MD5 checksum (stored: {}, comp

Error message

File {} did not match stored MD5 checksum  (stored: {}, computed: {}

What it means

MD5FileUtils.verifySavedMD5 compares the hash stored in the '.md5' sidecar file against the expected (typically freshly computed) hash of the data file. A mismatch means the file's current bytes are not what they were when the checksum was saved - the canonical signal of corruption, truncation, or a mixed-generation file+sidecar pair. This is how the NameNode detects damaged fsimage/edit files before trusting them.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java:62

public abstract class MD5FileUtils {
  private static final Logger LOG = LoggerFactory.getLogger(
      MD5FileUtils.class);

  public static final String MD5_SUFFIX = ".md5";
  private static final Pattern LINE_REGEX =
    Pattern.compile("([0-9a-f]{32}) [ \\*](.+)");
  
  /**
   * Verify that the previously saved md5 for the given file matches
   * expectedMd5.
   * @throws IOException 
   */
  public static void verifySavedMD5(File dataFile, MD5Hash expectedMD5)
      throws IOException {
    MD5Hash storedHash = readStoredMd5ForFile(dataFile);
    // Check the hash itself
    if (!expectedMD5.equals(storedHash)) {
      throw new IOException(
          "File " + dataFile + " did not match stored MD5 checksum " +
          " (stored: " + storedHash + ", computed: " + expectedMD5);
    }
  }
  
  /**
   * Read the md5 file stored alongside the given data file
   * and match the md5 file content.
   * @param md5File the file containing md5 data
   * @return a matcher with two matched groups
   *   where group(1) is the md5 string and group(2) is the data file path.
   */
  private static Matcher readStoredMd5(File md5File) throws IOException {
    BufferedReader reader =
        new BufferedReader(new InputStreamReader(
            Files.newInputStream(md5File.toPath()), StandardCharsets.UTF_8));
    String md5Line;
    try {

View on GitHub (pinned to 2add963021)

Solutions

1. Triage both hashes: 'md5sum <datafile>' versus the 32 hex chars inside '<datafile>.md5' - decides whether the data or the sidecar is the stale half.
2. If the data file is corrupt, restore the file AND its .md5 together from a consistent source: another storage directory, secondary/standby NameNode checkpoint, or backup.
3. If only the sidecar is stale (file verified good by other means, e.g., it loads with 'hdfs oiv'), regenerate it with MD5FileUtils.saveMD5File / md5sum.
4. Investigate the root cause: disk SMART errors, truncated transfers (re-fetch with size verification), and re-run hdfs fsck.

Example fix

// before
MD5FileUtils.verifySavedMD5(imgFile, MD5FileUtils.computeMd5ForFile(imgFile));
// throws 'did not match stored MD5 checksum' with both hashes buried in the message

// after - explicit triage before failing
MD5Hash stored = MD5FileUtils.readStoredMd5ForFile(imgFile);
if (stored != null) {
  MD5Hash computed = MD5FileUtils.computeMd5ForFile(imgFile);
  if (!stored.equals(computed)) {
    throw new IOException(String.format(
        "fsimage %s inconsistent (stored=%s, computed=%s): "
        + "restore file and .md5 as a pair from a healthy checkpoint",
        imgFile, stored, computed));
  }
}

// pre-verify the pair yourself and decide policy before the library throws
File md5 = MD5FileUtils.getDigestFileForFile(dataFile);
if (md5.exists()) {
  String line = new String(Files.readAllBytes(md5.toPath()), StandardCharsets.UTF_8).split("\\s+")[0];
  String computed = MD5FileUtils.computeMd5ForFile(dataFile).toString();
  if (!line.equals(computed)) {
    // decide: restore file+md5 pair from backup, or regenerate sidecar
    LOG.warn("Checksum drift on {}: stored={} computed={}", dataFile, line, computed);
  }
}

try {
  MD5FileUtils.verifySavedMD5(dataFile, expected);
} catch (IOException e) { // 'did not match stored MD5 checksum'
  // message carries stored and computed hashes: triage which half is stale,
  // restore file+sidecar as a consistent pair, never mix generations
  throw new IOException("Integrity failure on " + dataFile
      + " - restore data file and .md5 together: " + e.getMessage(), e);
}

When it happens

Trigger: verifySavedMD5(dataFile, computeMd5ForFile(dataFile)) after the data file changed since saveMD5File ran: truncated or partially downloaded fsimage (get/image transfer interrupted), bit rot on failing disks, interrupted save leaving new bytes with an old sidecar, or a sidecar copied from a different checkpoint generation.

Common situations: NameNode failing to load a checkpoint with 'Image file is corrupt'; fetching fsimage via HTTP tools that truncated the transfer; mixed storage directories after a restore; backup scripts copying data file and sidecar from different points in time.

Related errors

• File " + url + " computed digest " + computedDigest + " does
• MD5 file at {} references file named {} but we expected it t
• not able to find the highest writable parent dir
• Wrong length: {digest.length}
• Wrong length: {hex.length}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bdd2cc65c9f13b22. Report an issue: GitHub.