apache/hadoop · critical · InvalidChecksumSizeException

Could not create DataChecksum from the byte array of length

Error message

Could not create DataChecksum  from the byte array of length %d and bytesPerCheckSum of %d

What it means

The 5 header bytes were present, but the decoded bytesPerChecksum does not form a valid DataChecksum — the factory newDataChecksum(type, bytesPerChecksum) returned null (e.g. bytesPerChecksum <= 0 or inconsistent with the checksum type). The exception reports the byte-array length and the decoded bytesPerCheckSum, pointing at a corrupted header rather than a short buffer.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java:152

   * @throws InvalidChecksumSizeException when the stored checksum is invalid.
   */
  public static DataChecksum newDataChecksum(byte[] bytes, int offset)
      throws InvalidChecksumSizeException {
    if (offset < 0 || bytes.length < offset + getChecksumHeaderSize()) {
      throw new InvalidChecksumSizeException("Could not create DataChecksum "
          + " from the byte array of length " + bytes.length
          + " and offset "+ offset);
    }
    
    // like readInt():
    int bytesPerChecksum = ( (bytes[offset+1] & 0xff) << 24 ) | 
                           ( (bytes[offset+2] & 0xff) << 16 ) |
                           ( (bytes[offset+3] & 0xff) << 8 )  |
                           ( (bytes[offset+4] & 0xff) );
    DataChecksum csum = newDataChecksum(mapByteToChecksumType(bytes[offset]),
        bytesPerChecksum);
    if (csum == null) {
      throw new InvalidChecksumSizeException(("Could not create DataChecksum "
          + " from the byte array of length " + bytes.length
          + " and bytesPerCheckSum of "+ bytesPerChecksum));
    }
    return csum;
  }
  
  /**
   * This constructs a DataChecksum by reading HEADER_LEN bytes from input
   * stream <i>in</i>.
   *
   * @param in data input stream.
   * @throws IOException raised on errors performing I/O.
   * @return DataChecksum by reading HEADER_LEN
   *         bytes from input stream.
   */
  public static DataChecksum newDataChecksum( DataInputStream in )
                                 throws IOException {
    int type = in.readByte();

View on GitHub (pinned to 2add963021)

Solutions

  1. Run `hdfs fsck` on the file/block and re-replicate the corrupted block
  2. Verify io.bytes.per.checksum (default 512) is consistent across writers and readers
  3. Confirm the buffer offset actually sits at a checksum header boundary
  4. Catch InvalidChecksumSizeException and fail over to another replica or re-create the data
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-decode the header you are about to hand over
int bpc = ((buf[offset+1] & 0xff) << 24) | ((buf[offset+2] & 0xff) << 16)
         | ((buf[offset+3] & 0xff) << 8) | (buf[offset+4] & 0xff);
if (bpc <= 0) {
  throw new IOException("implausible bytesPerChecksum " + bpc + " — buffer likely misaligned");
}

Try / catch

try {
  DataChecksum csum = DataChecksum.newDataChecksum(buf, offset);
} catch (InvalidChecksumSizeException e) {
  // decoded bytesPerChecksum invalid: corruption, not a code bug
  logCorruptionAndFailOver(e);
}

Prevention

When it happens

Trigger: Corrupted checksum header bytes decoding to bytesPerChecksum <= 0; reading non-checksum payload as a header because the buffer/stream is misaligned; a writer using checksum settings (e.g. io.bytes.per.checksum) that the reader's checksum factory rejects.

Common situations: HDFS block corruption; reading blocks written by a misconfigured or incompatible writer; buffer positioning bugs in custom readers that treat data bytes as a header.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/825620f56a0f5348. Report an issue: GitHub.