{"record":{"id":"2b874c322cbde93c","repo":"apache/hadoop","slug":"checksum-failed-at-failedpos-for-replica-repli","errorCode":null,"errorMessage":"Checksum failed at {failedPos} for replica: {replica}","messagePattern":"Checksum failed at (.+?) for replica: (.+?)","errorType":"exception","errorClass":"ChecksumException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":739,"sourceCode":"   */\n  public void verifyChecksum(final byte[] buf, final int dataOffset,\n      final int datalen, final int numChunks, final int checksumOffset)\n      throws ChecksumException {\n    int dOff = dataOffset;\n    int cOff = checksumOffset;\n    int dLeft = datalen;\n\n    for (int i = 0; i < numChunks; i++) {\n      checksum.reset();\n      int dLen = Math.min(dLeft, chunkSize);\n      checksum.update(buf, dOff, dLen);\n      if (!checksum.compare(buf, cOff)) {\n        long failedPos = offset + datalen - dLeft;\n        StringBuilder replicaInfoString = new StringBuilder();\n        if (replica != null) {\n          replicaInfoString.append(\" for replica: \" + replica.toString());\n        }\n        throw new ChecksumException(\"Checksum failed at \" + failedPos\n            + replicaInfoString, failedPos);\n      }\n      dLeft -= dLen;\n      dOff += dLen;\n      cOff += checksumSize;\n    }\n  }\n  \n  /**\n   * sendBlock() is used to read block and its metadata and stream the data to\n   * either a client or to another datanode. \n   * \n   * @param out  stream to which the block is written to\n   * @param baseStream optional. if non-null, <code>out</code> is assumed to \n   *        be a wrapper over this stream. This enables optimizations for\n   *        sending the data, e.g. \n   *        {@link SocketOutputStream#transferToFully(FileChannel, \n   *        long, int)}.","sourceCodeStart":721,"sourceCodeEnd":757,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L721-L757","documentation":"While serving chunks with verifyChecksum enabled, BlockSender recomputes each chunk's checksum and compares it against the stored checksum in the meta file (checksum.compare(buf, cOff)). A mismatch means the bytes on disk no longer match their recorded checksum, so ChecksumException is thrown carrying failedPos (offset + datalen - dLeft, the exact byte position of the first bad chunk) and the replica details — silent data corruption has been detected on this replica.","triggerScenarios":"readBlock with checksum verification on data whose on-disk bytes diverge from the .meta checksums: bit rot on the disk, corruption introduced by a bad disk controller/RAM, or a block file modified out-of-band. Any subsequent read of that chunk throws at failedPos.","commonSituations":"Bit rot on consumer drives without T10-DIF/end-to-end checksums; faulty cables/controllers writing garbage; memory errors corrupting data before write; the datanode block scanner finding the same corruption in the background.","solutions":["Let the read fail over: DFSClient transparently tries another replica, so client impact is limited unless all replicas are corrupt.","Report/remove the corrupt replica: the block scanner or an fsck-triggered report marks it corrupt and the NN schedules re-replication from a good copy.","If multiple replicas fail checksums, restore the file from backup/snapshot — the data is genuinely lost.","Harden the stack: enable dfs.datanode.scan.period.hours, use ECC RAM, and prefer drives/paths with data-integrity protection."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"static boolean isChecksumFailure(IOException e) {\n  return e instanceof ChecksumException;\n}","tryCatchPattern":"try {\n  in.read(buf, off, len);\n} catch (ChecksumException e) {\n  reportCorruptBlockToNameNode(e); // includes failedPos from the exception\n  in = reopenFromNextReplica();    // DFSInputStream-style failover\n  in.read(buf, off, len);\n}","preventionTips":["Keep checksum verification on (never set dfs.client.read.shortcircuit.skipChecksum=true in production).","Enable the datanode block scanner so background scans surface corruption before reads do.","If ChecksumException repeats across all replicas of one block, stop retrying and restore from backup — data is lost."],"tags":["hdfs","datanode","read-path","checksum","data-corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}