nats-io/nats-server · critical

snapshot check detected %d bad messages

Error message

snapshot check detected %d bad messages

What it means

Raised when a snapshot consistency check completes but finds messages: the verification read (ld) contains bad messages, so the snapshot is rejected rather than producing a corrupt backup. The count of bad messages is included in the message. This indicates data problems in the stream's file storage rather than an API misuse.

Source

Thrown at server/filestore.go:12919

	}
	// Mark us as snapshotting
	fs.sips += 1
	fs.mu.Unlock()

	if checkMsgs {
		ld, err := fs.checkMsgs()
		clearSips := func() {
			fs.mu.Lock()
			fs.sips--
			fs.mu.Unlock()
		}
		if err != nil {
			clearSips()
			return nil, fmt.Errorf("snapshot check failed: %w", err)
		}
		if ld != nil && len(ld.Msgs) > 0 {
			clearSips()
			return nil, fmt.Errorf("snapshot check detected %d bad messages", len(ld.Msgs))
		}
	}

	pr, pw := net.Pipe()

	// Set a write deadline here to protect ourselves.
	if deadline > 0 {
		pw.SetWriteDeadline(time.Now().Add(deadline))
	}

	// We can add to our stream while snapshotting but not "user" delete anything.
	var state StreamState
	fs.FastState(&state)

	// Stream in separate Go routine.
	errCh := make(chan error, 1)
	go fs.streamSnapshot(pw, includeConsumers, errCh)

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Treat the stream as suspect: validate/repair by restoring the stream from a known-good backup.
2. Check server logs and the wrapped errors around this check for the specific bad sequences.
3. Avoid copying live filestore directories; use proper snapshot/backup APIs.
4. If corruption is hardware-driven, replace/repair storage and rebuild the stream.

// Go: on bad-message detection, fall back to a known-good backup
if strings.Contains(err.Error(), "snapshot check detected") {
    log.Printf("stream data integrity compromised: %v", err)
    // restore stream from last good snapshot
}

When it happens

Trigger: Stream snapshot verification where the check read returns non-empty ld.Msgs after verifying snapshot consistency, meaning messages read back differ from what the snapshot state expects.

Common situations: Streams affected by prior disk corruption, crashed writes, or files tampered with/synced inconsistently (e.g. rsync of a live stream directory); running an older server on data written by a newer version.

Related errors

• snapshot check failed: %w
• no interest
• no flow response
• unknown compression algorithm
• name required

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/c5ba65162f77a757. Report an issue: GitHub.