thanos-io/thanos · critical

mismatched checksum (got , expected )

Error message

mismatched checksum (got %v, expected %v)

What it means

After s2.Decode decompressed a chunkTypeCompressedData chunk, the CRC32 checksum of the decoded bytes did not match the checksum stored in the chunk header. This detects data corruption introduced either before compression (writer bug/bit rot) or after decompression (memory corruption), so the iterator aborts instead of returning wrong postings.

Solutions

1. Restore the block from a healthy replica or backup; this indicates real data corruption.
2. Run `promtool tsdb analyze` / `promtool check blocks` on the block directory to confirm and locate corruption.
3. Re-download the block (e.g. via Thanos/Cortex bucket re-sync) and verify checksums in transit.
4. Check storage hardware (SMART status) and disable any intermediary that rewrites bytes (compression proxies, FTP ASCII mode).

// verify block-level checksums before decoding
func blockChecksumOk(meta BlockMeta, sum uint32) bool { return meta.CRC == sum }

it := NewPostingsIterator(blob)
if !it.Next() {
    if err := it.Err(); err != nil {
        var expected uint32
        if errors.Is(err, ErrMismatchedChecksum) || strings.Contains(err.Error(), "mismatched checksum") {
            // treat block as corrupt: fail over to replica
        }
    }
}

When it happens

Trigger: Next() reads a compressed chunk whose stored checksum disagrees with crc32 of the s2-decompressed output — typically corrupted bytes on disk, a bad network transfer, or mismatched checksum function/endianness between writer and reader versions.

Common situations: Failing disk/SSD sectors in a TSDB block, interrupted block copy or rsync, downloading blocks over an unreliable link without verification, or tampered/incorrectly merged block files.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• got bad identifier
• snappy decode
• unrecognize postings format
• header not found
• missing magic snappy marker

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6218adaedc56a26f. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/postings_codec.go:306

			}
		}

		encodedBuf := it.input[:chunkLen]

		// NOTE(GiedriusS): we can probably optimize this better but this should be rare enough
		// and not cause any problems.
		if len(remainder) > 0 {
			remainderCopy := make([]byte, 0, len(remainder))
			remainderCopy = append(remainderCopy, remainder...)
			remainder = remainderCopy
		}
		decoded, err := s2.Decode(it.buf, encodedBuf[checksumSize:])
		if err != nil {
			it.err = err
			return false
		}
		if crc(decoded) != checksum {
			it.err = fmt.Errorf("mismatched checksum (got %v, expected %v)", crc(decoded), checksum)
			return false
		}
		if len(remainder) > 0 {
			it.db.B = append(remainder, decoded...)
		} else {
			it.db.B = decoded
		}
	case chunkTypeUncompressedData:
		if !it.readSnappyIdentifier {
			it.err = fmt.Errorf("missing magic snappy marker")
			return false
		}
		if len(it.input) < 4 {
			it.err = io.ErrUnexpectedEOF
			return false
		}
		checksum := uint32(it.input[0]) | uint32(it.input[1])<<8 | uint32(it.input[2])<<16 | uint32(it.input[3])<<24
		if len(it.input) < chunkLen {

View on GitHub (pinned to 35b8b99117)