apache/hadoop · error · IOException

Invalid MD5 file {}: the content "{}" does not match the exp

Error message

Invalid MD5 file {}: the content "{}" does not match the expected pattern.

What it means

The first line of a .md5 sidecar must match the regex ([0-9a-f]{32})[ *](.+): exactly 32 lowercase hexadecimal characters, a single space or asterisk separator, then a non-empty filename. Any other content throws 'Invalid MD5 file' - this is a format/parsing failure, distinct from an I/O read failure (3256) or a hash mismatch (3255).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java:92

   */
  private static Matcher readStoredMd5(File md5File) throws IOException {
    BufferedReader reader =
        new BufferedReader(new InputStreamReader(
            Files.newInputStream(md5File.toPath()), StandardCharsets.UTF_8));
    String md5Line;
    try {
      md5Line = reader.readLine();
      if (md5Line == null) { md5Line = ""; }
      md5Line = md5Line.trim();
    } catch (IOException ioe) {
      throw new IOException("Error reading md5 file at " + md5File, ioe);
    } finally {
      IOUtils.cleanupWithLogger(LOG, reader);
    }
    
    Matcher matcher = LINE_REGEX.matcher(md5Line);
    if (!matcher.matches()) {
      throw new IOException("Invalid MD5 file " + md5File + ": the content \""
          + md5Line + "\" does not match the expected pattern.");
    }
    return matcher;
  }

  /**
   * Read the md5 checksum stored alongside the given data file.
   * @param dataFile the file containing data
   * @return the checksum stored in dataFile.md5
   */
  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {
    final File md5File = getDigestFileForFile(dataFile);
    if (!md5File.exists()) {
      return null;
    }

    final Matcher matcher = readStoredMd5(md5File);
    String storedHash = matcher.group(1);

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the first line: cat <file>.md5 - it must look like 'a1b2...f3e4 fsimage_0000000000000000050'.
2. Regenerate in the exact format: md5sum <datafile> > <datafile>.md5 (md5sum's 'hash name' output matches the pattern), or MD5FileUtils.saveMD5File from Java.
3. Fix deviations: lowercase the hex (tr 'A-F' 'a-f'), remove BOM/extra text, ensure the filename is present.
4. If checksum verification is optional at that call site, delete the malformed sidecar - readStoredMd5ForFile treats a missing file as 'no stored checksum'.

Example fix

# before: fsimage_0000000000000000050.md5 contains (uppercase hash)
#   9A0369B2EB...
# -> Invalid MD5 file ... does not match the expected pattern

# after: regenerate lowercase, 'hash  filename' format
tr 'A-F' 'a-f' < fsimage_0000000000000000050.md5 > tmp.md5 && mv tmp.md5 fsimage_0000000000000000050.md5
# or simply: md5sum fsimage_0000000000000000050 > fsimage_0000000000000000050.md5

// pre-validate the sidecar line against the exact expected pattern
private static final Pattern MD5_LINE = Pattern.compile("([0-9a-f]{32})[ \\*](.+)");

static boolean sidecarWellFormed(File md5) throws IOException {
  if (!md5.exists()) return true; // absent is fine
  String line = new BufferedReader(new InputStreamReader(
      Files.newInputStream(md5.toPath()), StandardCharsets.UTF_8)).readLine();
  return line != null && MD5_LINE.matcher(line.trim()).matches();
}

static boolean isWellFormedMd5Line(String s) {
  return s != null && s.matches("([0-9a-f]{32})[ \\*](.+)");
}

When it happens

Trigger: readStoredMd5ForFile/verifySavedMD5/renameMD5File on a sidecar whose first line is empty (zero-byte file from an interrupted creation), contains uppercase hex (some Windows/tools emit uppercase), has 31/33 hex chars, adds a BOM or leading text, or omits the filename after the separator.

Common situations: Hand-recreated sidecars; checksum tools with different output formats; editors inserting BOM/CRLF or trailing junk; zero-length .md5 left after a crash mid-write (saveMD5File itself uses AtomicFileOutputStream, so externally created files are the usual culprits).

Related errors

• Wrong length: {digest.length}
• Wrong length: {hex.length}
• blockSize should be a multiple of checksumsize
• Bytes per checksum must be greater than 0
• Not a hex character: {c}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1711b1764a716e32. Report an issue: GitHub.