apache/hadoop · critical · org.apache.hadoop.fs.ChecksumException

Transaction is corrupt. Calculated checksum is {calculatedCh

Error message

Transaction is corrupt. Calculated checksum is {calculatedChecksum} but read checksum {expectedChecksum}

What it means

The definitive corruption signal for modern edits segments: LengthPrefixedReader CRC32s the opLength-minus-4 record bytes and compares against the stored 4-byte checksum int; a mismatch throws ChecksumException whose position field carries the record's txid. Data corruption here means the bytes on disk no longer match what was written.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:5269

      } else  if (opLength < MIN_OP_LENGTH) {
        throw new IOException("Op " + (int)opCodeByte + " has size " +
            opLength + ", but the minimum op size is " + MIN_OP_LENGTH);
      }
      long txid = in.readLong();
      // Verify checksum
      in.reset();
      in.mark(maxOpSize);
      checksum.reset();
      for (int rem = opLength - CHECKSUM_LENGTH; rem > 0;) {
        int toRead = Math.min(temp.length, rem);
        IOUtils.readFully(in, temp, 0, toRead);
        checksum.update(temp, 0, toRead);
        rem -= toRead;
      }
      int expectedChecksum = in.readInt();
      int calculatedChecksum = (int)checksum.getValue();
      if (expectedChecksum != calculatedChecksum) {
        throw new ChecksumException(
            "Transaction is corrupt. Calculated checksum is " +
            calculatedChecksum + " but read checksum " +
            expectedChecksum, txid);
      }
      return txid;
    }
  }

  /**
   * Read edit logs which have a checksum and a transaction ID, but not a
   * length.
   */
  private static class ChecksummedReader extends Reader {
    private final Checksum checksum;

    ChecksummedReader(Checksum checksum, DataInputStream in,
                      StreamLimiter limiter, int logVersion) {
      super(new DataInputStream(

View on GitHub (pinned to 2add963021)

Solutions

1. Note the txid from the exception, then run 'hdfs namenode -recover' to skip forward from that transaction and checkpoint immediately
2. Pull an uncorrupted copy of the same segment from the QJM majority / SecondaryNameNode and replace the bad file instead of skipping
3. Roll back to the previous fsimage checkpoint if skipping would lose needed transactions
4. Scrub the storage (smartctl, memory test) before trusting the node with metadata again

Example fix

# before
hdfs namenode   # ChecksumException: Transaction is corrupt. Calculated checksum is ... but read checksum ...
# after
hdfs namenode -recover   # skip from the reported txid, then: hdfs fsck / && hdfs dfsadmin -saveNamespace

# cheap full-CRC sweep before NameNode start / standby bootstrap
for f in /dfs/name/current/edits_*; do hdfs offlineEditsViewer -i "$f" -o /dev/null || echo "BAD: $f"; done

try {
  FSEditLogOp op = reader.readOp();
} catch (org.apache.hadoop.fs.ChecksumException ce) {
  long corruptTxid = ce.getPos(); // ChecksumException carries the txid here
  LOG.error("CRC mismatch at txid " + corruptTxid + " in " + segment, ce);
  // prefer restoring a clean QJM copy; else 'hdfs namenode -recover' to skip from corruptTxid
}

When it happens

Trigger: Bit rot on the journal/name disk; torn writes from a crash before fsync; a truncated tail whose length field still parsed; segments damaged in transfer to a standby or between JournalNodes.

Common situations: Failing disks, journal dirs on unreliable storage, crashes mid-transaction, network corruption of segment transfers, tampered segment files.

Related errors

• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum from the byte array of length
• The value %d does not map to a valid checksum Type
• The log file {} seems to contain valid transactions ; journa
• Same delegation token being added twice; invalid entry in fs

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3acf1ad744ea239d. Report an issue: GitHub.