apache/hadoop · critical · InvalidChecksumSizeException

Could not create DataChecksum from the byte array of length

Error message

Could not create DataChecksum  from the byte array of length %d and offset %d

What it means

DataChecksum.newDataChecksum(byte[] bytes, int offset) reconstructs a checksum header (1 type byte + 4 bytes of bytesPerChecksum) from a buffer position. If offset is negative or fewer than checksum-header-size bytes remain after offset, it throws InvalidChecksumSizeException (an IOException) reporting the array length and offset. In practice the caller handed in a truncated or misaligned buffer — typically a corrupted or partially-read checksum header.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java:139

    case CRC32C:
      return new DataChecksum(type, newCrc32C(), bytesPerChecksum);
    default:
      return null;  
    }
  }
  
  /**
   * Creates a DataChecksum from HEADER_LEN bytes from arr[offset].
   *
   * @param bytes bytes.
   * @param offset offset.
   * @return DataChecksum of the type in the array or null in case of an error.
   * @throws InvalidChecksumSizeException when the stored checksum is invalid.
   */
  public static DataChecksum newDataChecksum(byte[] bytes, int offset)
      throws InvalidChecksumSizeException {
    if (offset < 0 || bytes.length < offset + getChecksumHeaderSize()) {
      throw new InvalidChecksumSizeException("Could not create DataChecksum "
          + " from the byte array of length " + bytes.length
          + " and offset "+ offset);
    }
    
    // like readInt():
    int bytesPerChecksum = ( (bytes[offset+1] & 0xff) << 24 ) | 
                           ( (bytes[offset+2] & 0xff) << 16 ) |
                           ( (bytes[offset+3] & 0xff) << 8 )  |
                           ( (bytes[offset+4] & 0xff) );
    DataChecksum csum = newDataChecksum(mapByteToChecksumType(bytes[offset]),
        bytesPerChecksum);
    if (csum == null) {
      throw new InvalidChecksumSizeException(("Could not create DataChecksum "
          + " from the byte array of length " + bytes.length
          + " and bytesPerCheckSum of "+ bytesPerChecksum));
    }
    return csum;
  }

View on GitHub (pinned to 2add963021)

Solutions

Run `hdfs fsck <path>` on the affected files to confirm corruption and repair via re-replication
Audit the calling read loop for short reads — guarantee the buffer holds at least offset + getChecksumHeaderSize() bytes before calling
Catch InvalidChecksumSizeException and retry the read against a different replica
If the data is reproducible, rewrite the affected file/block

Example fix

// before: called on a possibly-short buffer
DataChecksum csum = DataChecksum.newDataChecksum(buf, off);
// after: guard the header length first
if (off < 0 || buf.length < off + DataChecksum.getChecksumHeaderSize()) {
  throw new IOException("truncated checksum header at offset " + off);
}
DataChecksum csum = DataChecksum.newDataChecksum(buf, off);

Defensive patterns

Strategy: try-catch

Validate before calling

int hdr = DataChecksum.getChecksumHeaderSize();
if (offset < 0 || buf.length < offset + hdr) {
  // refill the buffer before attempting header decode
  fill(buf, offset, hdr);
}

Try / catch

try {
  DataChecksum csum = DataChecksum.newDataChecksum(buf, offset);
} catch (InvalidChecksumSizeException e) {
  // header truncated or corrupt: treat the replica as bad, fail over to another one
  markReplicaBad(blockId);
  retryReadFromReplica(blockId, otherReplica);
}

Prevention

Guarantee the read loop fills offset + getChecksumHeaderSize() bytes before decoding
Schedule periodic hdfs fsck to catch corrupt blocks before readers do
Keep writer and reader Hadoop versions compatible during rolling upgrades

When it happens

Trigger: A short read delivering fewer than 5 header bytes at the given offset when reading block checksum headers; block/checksum data corrupted so the header region is cut off; off-by-one offset arithmetic in custom block-reading code calling this factory method.

Common situations: HDFS block corruption (verify with hdfs fsck); interrupted or uncommitted writes leaving truncated packets; reading data written by an incompatible writer; custom FileIO adapters that misalign buffers.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f4050e0a9a49798c. Report an issue: GitHub.