{"record":{"id":"a3c9ae765ae8be2c","repo":"weaviate/weaviate","slug":"block-checksum-mismatch","errorCode":null,"errorMessage":"block checksum mismatch","messagePattern":"block checksum mismatch","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"adapters/repos/db/vector/hnsw/compact/snapshot_reader.go","lineNumber":760,"sourceCode":"\t\treturn err\n\t}\n\n\treturn validateSnapshotBlockRanges(ranges, int(nodeCount), r.logger)\n}\n\n// readBlockConcurrent parses a single block and populates nodes in the result.\n// Uses mutex protection for concurrent access to the result.\nfunc (r *SnapshotReader) readBlockConcurrent(buf []byte, res *ent.DeserializationResult, mu *sync.Mutex) (snapshotBlockRange, error) {\n\tif len(buf) < 8 {\n\t\treturn snapshotBlockRange{}, fmt.Errorf(\"block too small: %d bytes\", len(buf))\n\t}\n\n\t// Verify checksum\n\tblockChecksum := binary.LittleEndian.Uint32(buf[:4])\n\thasher := crc32.NewIEEE()\n\t_, _ = hasher.Write(buf[4:])\n\tif hasher.Sum32() != blockChecksum {\n\t\treturn snapshotBlockRange{}, fmt.Errorf(\"block checksum mismatch\")\n\t}\n\n\t// Read block length from end\n\tblockLen := binary.LittleEndian.Uint32(buf[len(buf)-4:])\n\tif blockLen < 8 {\n\t\treturn snapshotBlockRange{}, fmt.Errorf(\"block length too small: %d bytes\", blockLen)\n\t}\n\tif blockLen > uint32(len(buf)-8) {\n\t\treturn snapshotBlockRange{}, fmt.Errorf(\"block length %d exceeds buffer payload %d\", blockLen, len(buf)-8)\n\t}\n\tblock := buf[4 : 4+blockLen]\n\n\t// Use byteops.ReadWriter instead of bytes.Reader + binary.Read\n\trw := byteops.NewReadWriter(block)\n\n\t// Read start node ID\n\tstartNodeID := rw.ReadUint64()\n\tcurrNodeID := startNodeID","sourceCodeStart":742,"sourceCodeEnd":778,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/vector/hnsw/compact/snapshot_reader.go#L742-L778","documentation":"Every snapshot body block stores a CRC32 (IEEE) checksum over its payload. readBlockConcurrent recomputes it over buf[4:] and compares with the stored value; a mismatch means the block was corrupted or altered on disk, and the block (and therefore the whole read) fails.","triggerScenarios":"readBlockConcurrent (invoked concurrently from readBodyConcurrent's worker goroutines) on any body block where crc32(buf[4:]) != the leading uint32 checksum — bit rot, torn write, or a modified block.","commonSituations":"Failing disk / silent bit corruption between write and read, interrupted snapshot write, or corruption introduced during backup copy/restore.","solutions":["Delete the corrupted .snapshot file and rebuild the HNSW index from the commitlog / stored objects.","Restore the snapshot file (or shard) from a verified backup and retry.","Investigate storage health (dmesg, SMART, filesystem logs) — checksum failures often indicate hardware-level corruption, not a software bug."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-flight integrity check of the whole snapshot file against a stored/backup checksum:\n// sum := crc32 of file at write time; compare after restore before starting the shard.","typeGuard":null,"tryCatchPattern":"if _, err := reader.Read(rsa); err != nil {\n\tif strings.Contains(err.Error(), \"block checksum mismatch\") {\n\t\tlogger.Errorf(\"snapshot block corrupt (possible disk failure); rebuilding index: %v\", err)\n\t\tos.Remove(snapshotPath)\n\t\treturn rebuildFromCommitlog()\n\t}\n\treturn err\n}","preventionTips":["Store an end-to-end checksum of snapshot files at creation and verify after any copy/restore.","Monitor disk SMART status and correct bit-rot (e.g. with ZFS/ECC).","Keep at least one verified backup of each shard before maintenance operations.","Alert on any checksum mismatch — treat as a hardware/health incident, not just a load failure."],"tags":["hnsw","snapshot","checksum","corruption","blocks"],"backgroundTag":"checksum-mismatch","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}