nats-io/nats-server · error

sanity check failed

Error message

sanity check failed

What it means

Thrown while scanning recovered message block data when a parsed record fails the quick sanity checks: negative data length, computed lengths inconsistent (shlen vs dlen-recordHashSize), record length over the buffer end, or record length beyond rlBadThresh. It guards against interpreting garbage as valid messages after truncation/corruption.

Source

Thrown at server/filestore.go:6496

	}

	for index, lbuf := uint32(0), uint32(len(buf)); index < lbuf; {
		if index+msgHdrSize > lbuf {
			return fmt.Errorf("message overrun")
		}
		hdr := buf[index : index+msgHdrSize]
		rl, slen := le.Uint32(hdr[0:]), int(le.Uint16(hdr[20:]))
		hasHeaders := rl&hbit != 0
		// Clear any headers bit that could be set.
		rl &^= hbit
		shlen := slen
		if hasHeaders {
			shlen += 4
		}
		dlen := int(rl) - msgHdrSize
		// Do some quick sanity checks here.
		if dlen < 0 || shlen > (dlen-recordHashSize) || dlen > int(rl) || index+rl > lbuf || rl > rlBadThresh {
			return fmt.Errorf("sanity check failed")
		}
		// Only need to process non-deleted messages.
		seq := le.Uint64(hdr[4:])
		ts := int64(le.Uint64(hdr[12:]))

		if !isDeleted(seq) {
			// Check for tombstones.
			if seq&tbit != 0 {
				seq = seq &^ tbit
				// If this entry is for a lower seq than ours then keep around.
				// We also check that it is greater than our floor. Floor is zero on normal
				// calls to compact.
				// If the global delete map is set, check if a tombstone is still
				// referencing a message in another block. If not, it can be removed.
				if seq < fseq && seq >= floor && (fsDmap == nil || fsDmap.Exists(seq)) {
					nbuf = append(nbuf, buf[index:index+rl]...)
				}
			} else {

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Reset/recover the affected stream (remove its files and let JetStream recreate) after confirming no healthy copy exists
2. Restore the store directory from a consistent offline backup
3. Pin server version across cluster nodes and upgrades so block formats match
4. Run disk/filesystem health checks to rule out hardware corruption

// Go: detect sanity failures during recovery and fall back to stream reset
if err := recoverStream(storeDir, streamName); err != nil {
    if strings.Contains(err.Error(), "sanity check failed") {
        // corrupt block: restore from consistent backup or reset stream
    }
    return err
}

When it happens

Trigger: Recovery scanning a block file with corrupted or misaligned record bytes — e.g. bit rot, partial writes, or data written by a mismatched store version — where rl/dlen values are implausible.

Common situations: Restore from inconsistent backups, running an older server against newer-format block files (or vice versa), failing disks, manual edits of store files.

Related errors

• message overrun
• failed to load block from disk: %w
• failed to decompress block: %w
• rebuildState for block %d failed: %w
• bad index file

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/17ef03a481914339. Report an issue: GitHub.