weaviate/weaviate · critical

block checksum mismatch

Error message

block checksum mismatch

What it means

Every snapshot body block stores a CRC32 (IEEE) checksum over its payload. readBlockConcurrent recomputes it over buf[4:] and compares with the stored value; a mismatch means the block was corrupted or altered on disk, and the block (and therefore the whole read) fails.

Source

Thrown at adapters/repos/db/vector/hnsw/compact/snapshot_reader.go:760

		return err
	}

	return validateSnapshotBlockRanges(ranges, int(nodeCount), r.logger)
}

// readBlockConcurrent parses a single block and populates nodes in the result.
// Uses mutex protection for concurrent access to the result.
func (r *SnapshotReader) readBlockConcurrent(buf []byte, res *ent.DeserializationResult, mu *sync.Mutex) (snapshotBlockRange, error) {
	if len(buf) < 8 {
		return snapshotBlockRange{}, fmt.Errorf("block too small: %d bytes", len(buf))
	}

	// Verify checksum
	blockChecksum := binary.LittleEndian.Uint32(buf[:4])
	hasher := crc32.NewIEEE()
	_, _ = hasher.Write(buf[4:])
	if hasher.Sum32() != blockChecksum {
		return snapshotBlockRange{}, fmt.Errorf("block checksum mismatch")
	}

	// Read block length from end
	blockLen := binary.LittleEndian.Uint32(buf[len(buf)-4:])
	if blockLen < 8 {
		return snapshotBlockRange{}, fmt.Errorf("block length too small: %d bytes", blockLen)
	}
	if blockLen > uint32(len(buf)-8) {
		return snapshotBlockRange{}, fmt.Errorf("block length %d exceeds buffer payload %d", blockLen, len(buf)-8)
	}
	block := buf[4 : 4+blockLen]

	// Use byteops.ReadWriter instead of bytes.Reader + binary.Read
	rw := byteops.NewReadWriter(block)

	// Read start node ID
	startNodeID := rw.ReadUint64()
	currNodeID := startNodeID

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Delete the corrupted .snapshot file and rebuild the HNSW index from the commitlog / stored objects.
2. Restore the snapshot file (or shard) from a verified backup and retry.
3. Investigate storage health (dmesg, SMART, filesystem logs) — checksum failures often indicate hardware-level corruption, not a software bug.

// Pre-flight integrity check of the whole snapshot file against a stored/backup checksum:
// sum := crc32 of file at write time; compare after restore before starting the shard.

if _, err := reader.Read(rsa); err != nil {
	if strings.Contains(err.Error(), "block checksum mismatch") {
		logger.Errorf("snapshot block corrupt (possible disk failure); rebuilding index: %v", err)
		os.Remove(snapshotPath)
		return rebuildFromCommitlog()
	}
	return err
}

When it happens

Trigger: readBlockConcurrent (invoked concurrently from readBodyConcurrent's worker goroutines) on any body block where crc32(buf[4:]) != the leading uint32 checksum — bit rot, torn write, or a modified block.

Common situations: Failing disk / silent bit corruption between write and read, interrupted snapshot write, or corruption introduced during backup copy/restore.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• metadata checksum mismatch
• block too small: %d bytes
• read body
• snapshot body missing for %d nodes
• block length too small: %d bytes

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/a3c9ae765ae8be2c. Report an issue: GitHub.