{"record":{"id":"1711b1764a716e32","repo":"apache/hadoop","slug":"invalid-md5-file-the-content-does-not-mat","errorCode":null,"errorMessage":"Invalid MD5 file {}: the content \"{}\" does not match the expected pattern.","messagePattern":"Invalid MD5 file (.+?): the content \"(.+?)\" does not match the expected pattern\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java","lineNumber":92,"sourceCode":"   */\n  private static Matcher readStoredMd5(File md5File) throws IOException {\n    BufferedReader reader =\n        new BufferedReader(new InputStreamReader(\n            Files.newInputStream(md5File.toPath()), StandardCharsets.UTF_8));\n    String md5Line;\n    try {\n      md5Line = reader.readLine();\n      if (md5Line == null) { md5Line = \"\"; }\n      md5Line = md5Line.trim();\n    } catch (IOException ioe) {\n      throw new IOException(\"Error reading md5 file at \" + md5File, ioe);\n    } finally {\n      IOUtils.cleanupWithLogger(LOG, reader);\n    }\n    \n    Matcher matcher = LINE_REGEX.matcher(md5Line);\n    if (!matcher.matches()) {\n      throw new IOException(\"Invalid MD5 file \" + md5File + \": the content \\\"\"\n          + md5Line + \"\\\" does not match the expected pattern.\");\n    }\n    return matcher;\n  }\n\n  /**\n   * Read the md5 checksum stored alongside the given data file.\n   * @param dataFile the file containing data\n   * @return the checksum stored in dataFile.md5\n   */\n  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {\n    final File md5File = getDigestFileForFile(dataFile);\n    if (!md5File.exists()) {\n      return null;\n    }\n\n    final Matcher matcher = readStoredMd5(md5File);\n    String storedHash = matcher.group(1);","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java#L74-L110","documentation":"The first line of a .md5 sidecar must match the regex ([0-9a-f]{32})[ *](.+): exactly 32 lowercase hexadecimal characters, a single space or asterisk separator, then a non-empty filename. Any other content throws 'Invalid MD5 file' - this is a format/parsing failure, distinct from an I/O read failure (3256) or a hash mismatch (3255).","triggerScenarios":"readStoredMd5ForFile/verifySavedMD5/renameMD5File on a sidecar whose first line is empty (zero-byte file from an interrupted creation), contains uppercase hex (some Windows/tools emit uppercase), has 31/33 hex chars, adds a BOM or leading text, or omits the filename after the separator.","commonSituations":"Hand-recreated sidecars; checksum tools with different output formats; editors inserting BOM/CRLF or trailing junk; zero-length .md5 left after a crash mid-write (saveMD5File itself uses AtomicFileOutputStream, so externally created files are the usual culprits).","solutions":["Inspect the first line: cat <file>.md5 - it must look like 'a1b2...f3e4  fsimage_0000000000000000050'.","Regenerate in the exact format: md5sum <datafile> > <datafile>.md5 (md5sum's 'hash  name' output matches the pattern), or MD5FileUtils.saveMD5File from Java.","Fix deviations: lowercase the hex (tr 'A-F' 'a-f'), remove BOM/extra text, ensure the filename is present.","If checksum verification is optional at that call site, delete the malformed sidecar - readStoredMd5ForFile treats a missing file as 'no stored checksum'."],"exampleFix":"# before: fsimage_0000000000000000050.md5 contains (uppercase hash)\n#   9A0369B2EB...\n# -> Invalid MD5 file ... does not match the expected pattern\n\n# after: regenerate lowercase, 'hash  filename' format\ntr 'A-F' 'a-f' < fsimage_0000000000000000050.md5 > tmp.md5 && mv tmp.md5 fsimage_0000000000000000050.md5\n# or simply: md5sum fsimage_0000000000000000050 > fsimage_0000000000000000050.md5","handlingStrategy":"validation","validationCode":"// pre-validate the sidecar line against the exact expected pattern\nprivate static final Pattern MD5_LINE = Pattern.compile(\"([0-9a-f]{32})[ \\\\*](.+)\");\n\nstatic boolean sidecarWellFormed(File md5) throws IOException {\n  if (!md5.exists()) return true; // absent is fine\n  String line = new BufferedReader(new InputStreamReader(\n      Files.newInputStream(md5.toPath()), StandardCharsets.UTF_8)).readLine();\n  return line != null && MD5_LINE.matcher(line.trim()).matches();\n}","typeGuard":"static boolean isWellFormedMd5Line(String s) {\n  return s != null && s.matches(\"([0-9a-f]{32})[ \\\\*](.+)\");\n}","tryCatchPattern":null,"preventionTips":["Generate sidecars only with md5sum or MD5FileUtils.saveMD5File - never by hand or scripts with custom formatting.","Keep hashes lowercase hex; beware Windows tools emitting uppercase.","Watch for zero-byte sidecars after crashes and BOMs from editors."],"tags":["md5","regex","file-format","validation","checksum"],"backgroundTag":"checksum-file-format-invalid","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}