{"record":{"id":"d16c29601741b9f3","repo":"apache/hadoop","slug":"error-reading-md5-file-at","errorCode":null,"errorMessage":"Error reading md5 file at {}","messagePattern":"Error reading md5 file at (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java","lineNumber":85,"sourceCode":"  \n  /**\n   * Read the md5 file stored alongside the given data file\n   * and match the md5 file content.\n   * @param md5File the file containing md5 data\n   * @return a matcher with two matched groups\n   *   where group(1) is the md5 string and group(2) is the data file path.\n   */\n  private static Matcher readStoredMd5(File md5File) throws IOException {\n    BufferedReader reader =\n        new BufferedReader(new InputStreamReader(\n            Files.newInputStream(md5File.toPath()), StandardCharsets.UTF_8));\n    String md5Line;\n    try {\n      md5Line = reader.readLine();\n      if (md5Line == null) { md5Line = \"\"; }\n      md5Line = md5Line.trim();\n    } catch (IOException ioe) {\n      throw new IOException(\"Error reading md5 file at \" + md5File, ioe);\n    } finally {\n      IOUtils.cleanupWithLogger(LOG, reader);\n    }\n    \n    Matcher matcher = LINE_REGEX.matcher(md5Line);\n    if (!matcher.matches()) {\n      throw new IOException(\"Invalid MD5 file \" + md5File + \": the content \\\"\"\n          + md5Line + \"\\\" does not match the expected pattern.\");\n    }\n    return matcher;\n  }\n\n  /**\n   * Read the md5 checksum stored alongside the given data file.\n   * @param dataFile the file containing data\n   * @return the checksum stored in dataFile.md5\n   */\n  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java#L67-L103","documentation":"readStoredMd5 wraps only IOExceptions from reading the first line of the sidecar ('Error reading md5 file at ...'). The stream is opened outside the guarded block, so a missing file surfaces as a raw FileNotFoundException instead; this message specifically means the .md5 exists but readLine() failed - an I/O-level fault, not a format problem (that is error 3257).","triggerScenarios":"verifySavedMD5 / readStoredMd5ForFile / renameMD5File while the .md5 exists but reading fails: EIO from failing storage, permissions revoked between the existence check and the read, or the file being replaced/truncated concurrently by another writer.","commonSituations":"Storage directories on disks with SMART errors; admins or security tooling tightening permissions after startup; races with a concurrent checkpoint writer replacing the sidecar.","solutions":["Check readability as the Hadoop user: ls -l <file>.md5 and head -c64 <file>.md5; fix chmod/chown if denied.","Look for hardware/filesystem faults: dmesg for I/O errors, fsck the volume, review SMART data.","Remove or restore the sidecar once the data file's integrity is confirmed by other means (readStoredMd5ForFile tolerates a missing sidecar by returning null).","Serialize checkpoint writes so two processes never replace the same sidecar concurrently."],"exampleFix":"// before\nMD5FileUtils.verifySavedMD5(dataFile, expected); // wraps read failure, cause chain lost at call site\n\n// after - probe readability first, keep the cause for triage\nFile md5 = MD5FileUtils.getDigestFileForFile(dataFile);\nif (md5.exists() && !Files.isReadable(md5.toPath())) {\n  throw new IOException(\"md5 sidecar not readable, fix perms: \" + md5);\n}\ntry {\n  MD5FileUtils.verifySavedMD5(dataFile, expected);\n} catch (IOException e) {\n  LOG.error(\"Failed reading {} (disk/perms?) - cause: {}\", md5, e.getMessage());\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":"File md5 = MD5FileUtils.getDigestFileForFile(dataFile);\nif (md5.exists() && !Files.isReadable(md5.toPath())) {\n  LOG.warn(\"Fix permissions on {} before checksum read\", md5);\n}","typeGuard":null,"tryCatchPattern":"try {\n  MD5FileUtils.verifySavedMD5(dataFile, expected);\n} catch (IOException e) { // 'Error reading md5 file at'\n  // distinguish from format error (3257) and mismatch (3255): this wraps a low-level read failure;\n  // check perms/disk, then retry or drop the sidecar (missing sidecar is tolerated)\n  LOG.error(\"I/O failure reading {} - cause: {}\", sidecarPath, e.getMessage());\n  throw e;\n}","preventionTips":["Lock down storage-directory permissions at deployment so they cannot drift under the daemon.","Schedule disk health checks on NameNode/JournalNode volumes.","Avoid concurrent writers to the same sidecar; serialize checkpoint finalization."],"tags":["filesystem","md5","io-error","permissions","checksum"],"backgroundTag":"file-read-io-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}