{"record":{"id":"825620f56a0f5348","repo":"apache/hadoop","slug":"could-not-create-datachecksum-from-the-byte-array-825620","errorCode":null,"errorMessage":"Could not create DataChecksum  from the byte array of length %d and bytesPerCheckSum of %d","messagePattern":"Could not create DataChecksum  from the byte array of length (.+?) and bytesPerCheckSum of (.+?)","errorType":"exception","errorClass":"InvalidChecksumSizeException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java","lineNumber":152,"sourceCode":"   * @throws InvalidChecksumSizeException when the stored checksum is invalid.\n   */\n  public static DataChecksum newDataChecksum(byte[] bytes, int offset)\n      throws InvalidChecksumSizeException {\n    if (offset < 0 || bytes.length < offset + getChecksumHeaderSize()) {\n      throw new InvalidChecksumSizeException(\"Could not create DataChecksum \"\n          + \" from the byte array of length \" + bytes.length\n          + \" and offset \"+ offset);\n    }\n    \n    // like readInt():\n    int bytesPerChecksum = ( (bytes[offset+1] & 0xff) << 24 ) | \n                           ( (bytes[offset+2] & 0xff) << 16 ) |\n                           ( (bytes[offset+3] & 0xff) << 8 )  |\n                           ( (bytes[offset+4] & 0xff) );\n    DataChecksum csum = newDataChecksum(mapByteToChecksumType(bytes[offset]),\n        bytesPerChecksum);\n    if (csum == null) {\n      throw new InvalidChecksumSizeException((\"Could not create DataChecksum \"\n          + \" from the byte array of length \" + bytes.length\n          + \" and bytesPerCheckSum of \"+ bytesPerChecksum));\n    }\n    return csum;\n  }\n  \n  /**\n   * This constructs a DataChecksum by reading HEADER_LEN bytes from input\n   * stream <i>in</i>.\n   *\n   * @param in data input stream.\n   * @throws IOException raised on errors performing I/O.\n   * @return DataChecksum by reading HEADER_LEN\n   *         bytes from input stream.\n   */\n  public static DataChecksum newDataChecksum( DataInputStream in )\n                                 throws IOException {\n    int type = in.readByte();","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java#L134-L170","documentation":"The 5 header bytes were present, but the decoded bytesPerChecksum does not form a valid DataChecksum — the factory newDataChecksum(type, bytesPerChecksum) returned null (e.g. bytesPerChecksum <= 0 or inconsistent with the checksum type). The exception reports the byte-array length and the decoded bytesPerCheckSum, pointing at a corrupted header rather than a short buffer.","triggerScenarios":"Corrupted checksum header bytes decoding to bytesPerChecksum <= 0; reading non-checksum payload as a header because the buffer/stream is misaligned; a writer using checksum settings (e.g. io.bytes.per.checksum) that the reader's checksum factory rejects.","commonSituations":"HDFS block corruption; reading blocks written by a misconfigured or incompatible writer; buffer positioning bugs in custom readers that treat data bytes as a header.","solutions":["Run `hdfs fsck` on the file/block and re-replicate the corrupted block","Verify io.bytes.per.checksum (default 512) is consistent across writers and readers","Confirm the buffer offset actually sits at a checksum header boundary","Catch InvalidChecksumSizeException and fail over to another replica or re-create the data"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-decode the header you are about to hand over\nint bpc = ((buf[offset+1] & 0xff) << 24) | ((buf[offset+2] & 0xff) << 16)\n         | ((buf[offset+3] & 0xff) << 8) | (buf[offset+4] & 0xff);\nif (bpc <= 0) {\n  throw new IOException(\"implausible bytesPerChecksum \" + bpc + \" — buffer likely misaligned\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  DataChecksum csum = DataChecksum.newDataChecksum(buf, offset);\n} catch (InvalidChecksumSizeException e) {\n  // decoded bytesPerChecksum invalid: corruption, not a code bug\n  logCorruptionAndFailOver(e);\n}","preventionTips":["Pin io.bytes.per.checksum (default 512) consistently across the cluster","Verify stream/buffer alignment before each header decode","Monitor hdfs fsck reports for blocks with bad checksum headers"],"tags":["hdfs","checksum","data-integrity","corruption","bytes-per-checksum"],"backgroundTag":"data-corruption-detected","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}