{"record":{"id":"f4050e0a9a49798c","repo":"apache/hadoop","slug":"could-not-create-datachecksum-from-the-byte-array","errorCode":null,"errorMessage":"Could not create DataChecksum  from the byte array of length %d and offset %d","messagePattern":"Could not create DataChecksum  from the byte array of length (.+?) and offset (.+?)","errorType":"exception","errorClass":"InvalidChecksumSizeException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java","lineNumber":139,"sourceCode":"    case CRC32C:\n      return new DataChecksum(type, newCrc32C(), bytesPerChecksum);\n    default:\n      return null;  \n    }\n  }\n  \n  /**\n   * Creates a DataChecksum from HEADER_LEN bytes from arr[offset].\n   *\n   * @param bytes bytes.\n   * @param offset offset.\n   * @return DataChecksum of the type in the array or null in case of an error.\n   * @throws InvalidChecksumSizeException when the stored checksum is invalid.\n   */\n  public static DataChecksum newDataChecksum(byte[] bytes, int offset)\n      throws InvalidChecksumSizeException {\n    if (offset < 0 || bytes.length < offset + getChecksumHeaderSize()) {\n      throw new InvalidChecksumSizeException(\"Could not create DataChecksum \"\n          + \" from the byte array of length \" + bytes.length\n          + \" and offset \"+ offset);\n    }\n    \n    // like readInt():\n    int bytesPerChecksum = ( (bytes[offset+1] & 0xff) << 24 ) | \n                           ( (bytes[offset+2] & 0xff) << 16 ) |\n                           ( (bytes[offset+3] & 0xff) << 8 )  |\n                           ( (bytes[offset+4] & 0xff) );\n    DataChecksum csum = newDataChecksum(mapByteToChecksumType(bytes[offset]),\n        bytesPerChecksum);\n    if (csum == null) {\n      throw new InvalidChecksumSizeException((\"Could not create DataChecksum \"\n          + \" from the byte array of length \" + bytes.length\n          + \" and bytesPerCheckSum of \"+ bytesPerChecksum));\n    }\n    return csum;\n  }","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java#L121-L157","documentation":"DataChecksum.newDataChecksum(byte[] bytes, int offset) reconstructs a checksum header (1 type byte + 4 bytes of bytesPerChecksum) from a buffer position. If offset is negative or fewer than checksum-header-size bytes remain after offset, it throws InvalidChecksumSizeException (an IOException) reporting the array length and offset. In practice the caller handed in a truncated or misaligned buffer — typically a corrupted or partially-read checksum header.","triggerScenarios":"A short read delivering fewer than 5 header bytes at the given offset when reading block checksum headers; block/checksum data corrupted so the header region is cut off; off-by-one offset arithmetic in custom block-reading code calling this factory method.","commonSituations":"HDFS block corruption (verify with hdfs fsck); interrupted or uncommitted writes leaving truncated packets; reading data written by an incompatible writer; custom FileIO adapters that misalign buffers.","solutions":["Run `hdfs fsck <path>` on the affected files to confirm corruption and repair via re-replication","Audit the calling read loop for short reads — guarantee the buffer holds at least offset + getChecksumHeaderSize() bytes before calling","Catch InvalidChecksumSizeException and retry the read against a different replica","If the data is reproducible, rewrite the affected file/block"],"exampleFix":"// before: called on a possibly-short buffer\nDataChecksum csum = DataChecksum.newDataChecksum(buf, off);\n// after: guard the header length first\nif (off < 0 || buf.length < off + DataChecksum.getChecksumHeaderSize()) {\n  throw new IOException(\"truncated checksum header at offset \" + off);\n}\nDataChecksum csum = DataChecksum.newDataChecksum(buf, off);","handlingStrategy":"try-catch","validationCode":"int hdr = DataChecksum.getChecksumHeaderSize();\nif (offset < 0 || buf.length < offset + hdr) {\n  // refill the buffer before attempting header decode\n  fill(buf, offset, hdr);\n}","typeGuard":null,"tryCatchPattern":"try {\n  DataChecksum csum = DataChecksum.newDataChecksum(buf, offset);\n} catch (InvalidChecksumSizeException e) {\n  // header truncated or corrupt: treat the replica as bad, fail over to another one\n  markReplicaBad(blockId);\n  retryReadFromReplica(blockId, otherReplica);\n}","preventionTips":["Guarantee the read loop fills offset + getChecksumHeaderSize() bytes before decoding","Schedule periodic hdfs fsck to catch corrupt blocks before readers do","Keep writer and reader Hadoop versions compatible during rolling upgrades"],"tags":["hdfs","checksum","data-integrity","corruption","datachecksum"],"backgroundTag":"data-corruption-detected","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}