nats-io/nats-server · error

failed to process gap

Error message

failed to process gap: %w

What it means

When RestoreStreamV2 detects a gap between the previous sequence and the current one (deleted messages in the source), it calls store.SkipMsgs(lseq+1, gap) to reserve those sequences. This error wraps a failure of that skip operation on the target stream's message store, meaning the destination store could not advance past the missing sequences.

Solutions

1. Check disk space and filesystem health on the JetStream storage directory
2. Remove/clean the partially restored stream and retry the restore into a fresh stream
3. Inspect server logs for the underlying SkipMsgs/storage error
4. Verify the storage backend (file store) is not corrupted by running server restart and letting JetStream recover

// before restore: check storage health
if free, err := diskFree(jsStoreDir); err != nil || free < neededBytes { return fmt.Errorf("insufficient JetStream storage") }

if err != nil && strings.Contains(err.Error(), "failed to process gap") {
    // inspect server logs for storage error; free space, then restore into a fresh stream
}

When it happens

Trigger: store.SkipMsgs returns an error while padding a gap; typically due to underlying storage failure (disk full, I/O error), message block corruption in the destination stream, or the store being in an inconsistent state mid-restore.

Common situations: Restoring onto a server whose JetStream storage is failing (disk full, bad block), or restoring a backup containing deleted messages onto a stream with conflicting existing state.

Related errors

• failed to process expired message sequence
• failed to process trailing gap
• failed to store message sequence
• backup was truncated
• could not create storage directory

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/694bb6bb7a97964d. Report an issue: GitHub.

Appendix: source

Thrown at server/stream_backup.go:522

			return nil, fmt.Errorf("failed to read message sequence %d: %w", seq, err)
		}
		if hdr.HeaderSize > int64(len(buf)) {
			return nil, fmt.Errorf("failed to parse message sequence %d: invalid header length", seq)
		}
		if int64(len(buf)) != declaredSize {
			return nil, fmt.Errorf("failed to read message sequence %d: unexpected payload size", seq)
		}
		subj := hdr.Name
		mhdr := buf[:hdr.HeaderSize]
		msg := buf[hdr.HeaderSize : hdr.HeaderSize+hdr.PayloadSize]
		if seq <= lseq {
			return nil, fmt.Errorf("message sequence %d out of order", seq)
		}
		// We could have deleted messages since the last message we stored, if so
		// we should work out what the gap is and skip those sequences.
		if gap := seq - lseq - 1; gap > 0 {
			if err := store.SkipMsgs(lseq+1, gap); err != nil {
				return nil, fmt.Errorf("failed to process gap: %w", err)
			}
		}
		lseq = seq
		ttl, err := getMessageTTL(mhdr)
		if err != nil {
			return nil, fmt.Errorf("failed to parse message TTL: %w", err)
		}
		hdrTime := time.Unix(0, hdr.Timestamp)
		if ttl > 0 && time.Now().After(hdrTime.Add(time.Duration(ttl)*time.Second)) {
			// If the TTL has exceeded then there isn't much point in storing the message,
			// but we still need to preserve the sequence.
			if err := store.SkipMsgs(seq, 1); err != nil {
				return nil, fmt.Errorf("failed to process expired message sequence %d: %w", seq, err)
			}
			releaseRestoreBytes(storedSize)
			continue
		}
		if err = store.StoreRawMsg(subj, mhdr, msg, seq, hdr.Timestamp, ttl, false); err != nil {

View on GitHub (pinned to 3a66a489d2)