{"record":{"id":"4d7c7495b9f249a4","repo":"apache/hadoop","slug":"fail-to-get-block-md5-for","errorCode":null,"errorMessage":"Fail to get block MD5 for {}","messagePattern":"Fail to get block MD5 for (.+?)","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java","lineNumber":496,"sourceCode":"                                   ChecksumCombineMode combineMode)\n        throws IOException {\n      super(src, length, blockLocations, namenode, client, combineMode);\n    }\n\n    @Override\n    void checksumBlocks() throws IOException {\n      // get block checksum for each block\n      for (blockIdx = 0;\n           blockIdx < getLocatedBlocks().size() && getRemaining() >= 0;\n           blockIdx++) {\n        if (isRefetchBlocks()) {  // refetch to get fresh tokens\n          refetchBlocks();\n        }\n\n        LocatedBlock locatedBlock = getLocatedBlocks().get(blockIdx);\n\n        if (!checksumBlock(locatedBlock)) {\n          throw new PathIOException(\n              getSrc(), \"Fail to get block MD5 for \" + locatedBlock);\n        }\n      }\n    }\n\n    /**\n     * Return true when sounds good to continue or retry, false when severe\n     * condition or totally failed.\n     */\n    private boolean checksumBlock(LocatedBlock locatedBlock) {\n      ExtendedBlock block = locatedBlock.getBlock();\n      if (getRemaining() < block.getNumBytes()) {\n        block.setNumBytes(getRemaining());\n      }\n      setRemaining(getRemaining() - block.getNumBytes());\n\n      DatanodeInfo[] datanodes = locatedBlock.getLocations();\n","sourceCodeStart":478,"sourceCodeEnd":514,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java#L478-L514","documentation":"ReplicatedFileChecksumComputer tries every located replica of a block; checksumBlock() returns false only when no replica could be read (all DataNodes dead, corrupt, or erroring), and the caller wraps that as a PathIOException naming the source file and the LocatedBlock. It means the file checksum could not be computed because one block is effectively unavailable, distinct from checksum-value mismatches.","triggerScenarios":"getFileChecksum() on a replicated file where a block's every located replica fails: DataNodes down or decommissioning, replicas corrupt (checksum errors recorded during the attempt), block under reconstruction, or block tokens expired and refetch exhausted.","commonSituations":"Clusters with dead/decommissioned DataNodes that still own located replicas; hardware failures on all replicas' hosts; aggressive DataNode restarts during a checksum sweep; files with missing blocks reported by fsck.","solutions":["Run hdfs fsck /path -files -blocks -locations to identify the unavailable block and its replicas","Restore/restart the DataNodes hosting the missing replicas, then retry the checksum","If replicas are truly lost, recover the file from source or accept data loss (fsck -move/-delete) — checksum verification is impossible without a readable replica","Retry after NameNode block reports refresh locations; transient decommissions self-heal"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-check block health before checksum sweeps:\n// hdfs fsck /path -files -blocks -locations  (expect 'HEALTHY')\n// Programmatically: read the file fully first to confirm every block is servable.","typeGuard":null,"tryCatchPattern":"int attempts = 0;\nwhile (true) {\n  try {\n    return dfsClient.getFileChecksum(path);\n  } catch (PathIOException e) {\n    if (++attempts >= 3 || !String.valueOf(e.getMessage()).contains(\"Fail to get block MD5\")) throw e;\n    sleep(backoff(attempts)); // DataNodes may be restarting; locations refresh on retry\n  }\n}","preventionTips":["Run fsck before large checksum verification jobs to surface unavailable blocks early","Monitor DataNode liveness; checksum jobs amplify the impact of dead replicas","Retry with backoff — located replicas refresh from the NameNode between attempts"],"tags":["hdfs","checksum","replica-unavailable","datanode","block-health"],"backgroundTag":"all-replicas-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}