nats-io/nats-server · error

message sequence %d out of order

Error message

message sequence %d out of order

What it means

RestoreStreamV2 enforces that message sequences in the backup are strictly increasing and contiguous. This error is returned when a message's sequence (seq) is less than or equal to the last stored sequence (lseq), meaning the backup contains out-of-order or duplicate sequence data that would break stream state invariants.

Source

Thrown at server/stream_backup.go:516

			if err := checkUsageLimits(); err != nil {
				return nil, err
			}
		}
		buf, err := io.ReadAll(tr)
		if err != nil {
			return nil, fmt.Errorf("failed to read message sequence %d: %w", seq, err)
		}
		if hdr.HeaderSize > int64(len(buf)) {
			return nil, fmt.Errorf("failed to parse message sequence %d: invalid header length", seq)
		}
		if int64(len(buf)) != declaredSize {
			return nil, fmt.Errorf("failed to read message sequence %d: unexpected payload size", seq)
		}
		subj := hdr.Name
		mhdr := buf[:hdr.HeaderSize]
		msg := buf[hdr.HeaderSize : hdr.HeaderSize+hdr.PayloadSize]
		if seq <= lseq {
			return nil, fmt.Errorf("message sequence %d out of order", seq)
		}
		// We could have deleted messages since the last message we stored, if so
		// we should work out what the gap is and skip those sequences.
		if gap := seq - lseq - 1; gap > 0 {
			if err := store.SkipMsgs(lseq+1, gap); err != nil {
				return nil, fmt.Errorf("failed to process gap: %w", err)
			}
		}
		lseq = seq
		ttl, err := getMessageTTL(mhdr)
		if err != nil {
			return nil, fmt.Errorf("failed to parse message TTL: %w", err)
		}
		hdrTime := time.Unix(0, hdr.Timestamp)
		if ttl > 0 && time.Now().After(hdrTime.Add(time.Duration(ttl)*time.Second)) {
			// If the TTL has exceeded then there isn't much point in storing the message,
			// but we still need to preserve the sequence.
			if err := store.SkipMsgs(seq, 1); err != nil {

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Restore from a single, complete backup file produced in one session; do not concatenate backups
2. Verify the archive integrity (checksum) and re-create the backup if corrupt
3. Check that the backup was not appended/merged across multiple snapshots
4. If resuming incremental backups, use the server-supported mechanisms rather than manual merging

// Never merge or append separate backup archives; verify a single archive
if isMergedArchive(path) { return fmt.Errorf("refusing to restore merged/concatenated backup") }

if err := restore(...); err != nil && strings.Contains(err.Error(), "out of order") {
    // mark backup invalid; obtain a new full backup
}

When it happens

Trigger: A backup block carries a seq <= lseq, i.e. duplicate or decreasing sequences; caused by corrupt backup data, wrongly concatenated backups, or replaying blocks from an older snapshot into a newer one.

Common situations: Merging two backup files, resuming an interrupted backup incorrectly, or restoring a corrupted archive where block boundaries were lost and records misparsed.

Related errors

• failed to parse message sequence %d: invalid header length
• failed to read message sequence %d: unexpected payload size
• failed to process trailing gap: %w
• rebuildState for block %d failed: %w
• bad index file

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/d69ea5cbe90958fa. Report an issue: GitHub.