apache/hadoop · critical · InvalidChecksumSizeException

Could not create DataChecksum of type %d with bytesPerChecks

Error message

Could not create DataChecksum of type %d with bytesPerChecksum %d

What it means

DataChecksum.newDataChecksum(DataInputStream) reads the 1-byte checksum type and 4-byte bytesPerChecksum directly from the wire (block data transfer and checksum-reading paths). If no checksummer exists for that (type, bytesPerChecksum) pair, it throws InvalidChecksumSizeException echoing both integers — the stream did not contain a valid checksum header where one was expected.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java:174

    return csum;
  }
  
  /**
   * This constructs a DataChecksum by reading HEADER_LEN bytes from input
   * stream <i>in</i>.
   *
   * @param in data input stream.
   * @throws IOException raised on errors performing I/O.
   * @return DataChecksum by reading HEADER_LEN
   *         bytes from input stream.
   */
  public static DataChecksum newDataChecksum( DataInputStream in )
                                 throws IOException {
    int type = in.readByte();
    int bpc = in.readInt();
    DataChecksum summer = newDataChecksum(mapByteToChecksumType(type), bpc);
    if ( summer == null ) {
      throw new InvalidChecksumSizeException("Could not create DataChecksum "
          + "of type " + type + " with bytesPerChecksum " + bpc);
    }
    return summer;
  }

  private static Type mapByteToChecksumType(int type)
      throws InvalidChecksumSizeException{
    try {
      return Type.valueOf(type);
    } catch (IllegalArgumentException e) {
      throw new InvalidChecksumSizeException("The value "+type+" does not map"+
        " to a valid checksum Type");
    }
  }
  
  /**
   * Writes the checksum header to the output stream <i>out</i>.
   *

View on GitHub (pinned to 2add963021)

Solutions

Verify the stream is positioned at a checksum header before calling (read the 5 header bytes deliberately)
Check block metadata version compatibility between writer and reader (BlockMetadataHeader layout)
Run `hdfs fsck` and re-replicate the corrupted block
On version skew, finish the rolling upgrade before reading affected data with older components

Defensive patterns

Strategy: try-catch

Try / catch

try {
  DataChecksum csum = DataChecksum.newDataChecksum(in);
} catch (InvalidChecksumSizeException e) {
  // stream not at a valid checksum header: realign, re-open, or fail over to another replica
  reopenStreamAtLastBoundary();
}

Prevention

Read block metadata through BlockMetadataHeader helpers instead of raw stream reads
Complete rolling upgrades before old components read new-format checksums
Unit-test stream positioning whenever you touch block-reading code

When it happens

Trigger: The stream is positioned at the wrong offset (misaligned read of block checksum data); a peer writing a different data layout; corrupted checksum header bytes in block metadata being read back.

Common situations: Reading block metadata written by an incompatible Hadoop version; stream positioning bugs after partial reads; storage corruption in the checksum header region.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/71f2e734e3197510. Report an issue: GitHub.