What causes "Error reading md5 file at {}" in apache/hadoop?

Check readability as the Hadoop user: ls -l .md5 and head -c64 .md5; fix chmod/chown if denied.

apache/hadoop · error · IOException

Error reading md5 file at {}

Error message

Error reading md5 file at {}

What it means

readStoredMd5 wraps only IOExceptions from reading the first line of the sidecar ('Error reading md5 file at ...'). The stream is opened outside the guarded block, so a missing file surfaces as a raw FileNotFoundException instead; this message specifically means the .md5 exists but readLine() failed - an I/O-level fault, not a format problem (that is error 3257).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java:85

  
  /**
   * Read the md5 file stored alongside the given data file
   * and match the md5 file content.
   * @param md5File the file containing md5 data
   * @return a matcher with two matched groups
   *   where group(1) is the md5 string and group(2) is the data file path.
   */
  private static Matcher readStoredMd5(File md5File) throws IOException {
    BufferedReader reader =
        new BufferedReader(new InputStreamReader(
            Files.newInputStream(md5File.toPath()), StandardCharsets.UTF_8));
    String md5Line;
    try {
      md5Line = reader.readLine();
      if (md5Line == null) { md5Line = ""; }
      md5Line = md5Line.trim();
    } catch (IOException ioe) {
      throw new IOException("Error reading md5 file at " + md5File, ioe);
    } finally {
      IOUtils.cleanupWithLogger(LOG, reader);
    }
    
    Matcher matcher = LINE_REGEX.matcher(md5Line);
    if (!matcher.matches()) {
      throw new IOException("Invalid MD5 file " + md5File + ": the content \""
          + md5Line + "\" does not match the expected pattern.");
    }
    return matcher;
  }

  /**
   * Read the md5 checksum stored alongside the given data file.
   * @param dataFile the file containing data
   * @return the checksum stored in dataFile.md5
   */
  public static MD5Hash readStoredMd5ForFile(File dataFile) throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

Check readability as the Hadoop user: ls -l <file>.md5 and head -c64 <file>.md5; fix chmod/chown if denied.
Look for hardware/filesystem faults: dmesg for I/O errors, fsck the volume, review SMART data.
Remove or restore the sidecar once the data file's integrity is confirmed by other means (readStoredMd5ForFile tolerates a missing sidecar by returning null).
Serialize checkpoint writes so two processes never replace the same sidecar concurrently.

Example fix

// before
MD5FileUtils.verifySavedMD5(dataFile, expected); // wraps read failure, cause chain lost at call site

// after - probe readability first, keep the cause for triage
File md5 = MD5FileUtils.getDigestFileForFile(dataFile);
if (md5.exists() && !Files.isReadable(md5.toPath())) {
  throw new IOException("md5 sidecar not readable, fix perms: " + md5);
}
try {
  MD5FileUtils.verifySavedMD5(dataFile, expected);
} catch (IOException e) {
  LOG.error("Failed reading {} (disk/perms?) - cause: {}", md5, e.getMessage());
  throw e;
}

Defensive patterns

Strategy: try-catch

Validate before calling

File md5 = MD5FileUtils.getDigestFileForFile(dataFile);
if (md5.exists() && !Files.isReadable(md5.toPath())) {
  LOG.warn("Fix permissions on {} before checksum read", md5);
}

Try / catch

try {
  MD5FileUtils.verifySavedMD5(dataFile, expected);
} catch (IOException e) { // 'Error reading md5 file at'
  // distinguish from format error (3257) and mismatch (3255): this wraps a low-level read failure;
  // check perms/disk, then retry or drop the sidecar (missing sidecar is tolerated)
  LOG.error("I/O failure reading {} - cause: {}", sidecarPath, e.getMessage());
  throw e;
}

Prevention

Lock down storage-directory permissions at deployment so they cannot drift under the daemon.
Schedule disk health checks on NameNode/JournalNode volumes.
Avoid concurrent writers to the same sidecar; serialize checkpoint finalization.

When it happens

Trigger: verifySavedMD5 / readStoredMd5ForFile / renameMD5File while the .md5 exists but reading fails: EIO from failing storage, permissions revoked between the existence check and the read, or the file being replaced/truncated concurrently by another writer.

Common situations: Storage directories on disks with SMART errors; admins or security tooling tightening permissions after startup; races with a concurrent checkpoint writer replacing the sidecar.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d16c29601741b9f3. Report an issue: GitHub.