nats-io/nats-server · error

failed to process gap: %w

Error message

failed to process gap: %w

What it means

When RestoreStreamV2 detects a gap between the previous sequence and the current one (deleted messages in the source), it calls store.SkipMsgs(lseq+1, gap) to reserve those sequences. This error wraps a failure of that skip operation on the target stream's message store, meaning the destination store could not advance past the missing sequences.

Source

Thrown at server/stream_backup.go:522

			return nil, fmt.Errorf("failed to read message sequence %d: %w", seq, err)
		}
		if hdr.HeaderSize > int64(len(buf)) {
			return nil, fmt.Errorf("failed to parse message sequence %d: invalid header length", seq)
		}
		if int64(len(buf)) != declaredSize {
			return nil, fmt.Errorf("failed to read message sequence %d: unexpected payload size", seq)
		}
		subj := hdr.Name
		mhdr := buf[:hdr.HeaderSize]
		msg := buf[hdr.HeaderSize : hdr.HeaderSize+hdr.PayloadSize]
		if seq <= lseq {
			return nil, fmt.Errorf("message sequence %d out of order", seq)
		}
		// We could have deleted messages since the last message we stored, if so
		// we should work out what the gap is and skip those sequences.
		if gap := seq - lseq - 1; gap > 0 {
			if err := store.SkipMsgs(lseq+1, gap); err != nil {
				return nil, fmt.Errorf("failed to process gap: %w", err)
			}
		}
		lseq = seq
		ttl, err := getMessageTTL(mhdr)
		if err != nil {
			return nil, fmt.Errorf("failed to parse message TTL: %w", err)
		}
		hdrTime := time.Unix(0, hdr.Timestamp)
		if ttl > 0 && time.Now().After(hdrTime.Add(time.Duration(ttl)*time.Second)) {
			// If the TTL has exceeded then there isn't much point in storing the message,
			// but we still need to preserve the sequence.
			if err := store.SkipMsgs(seq, 1); err != nil {
				return nil, fmt.Errorf("failed to process expired message sequence %d: %w", seq, err)
			}
			releaseRestoreBytes(storedSize)
			continue
		}
		if err = store.StoreRawMsg(subj, mhdr, msg, seq, hdr.Timestamp, ttl, false); err != nil {

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Check disk space and filesystem health on the JetStream storage directory
2. Remove/clean the partially restored stream and retry the restore into a fresh stream
3. Inspect server logs for the underlying SkipMsgs/storage error
4. Verify the storage backend (file store) is not corrupted by running server restart and letting JetStream recover

// before restore: check storage health
if free, err := diskFree(jsStoreDir); err != nil || free < neededBytes { return fmt.Errorf("insufficient JetStream storage") }

if err != nil && strings.Contains(err.Error(), "failed to process gap") {
    // inspect server logs for storage error; free space, then restore into a fresh stream
}

When it happens

Trigger: store.SkipMsgs returns an error while padding a gap; typically due to underlying storage failure (disk full, I/O error), message block corruption in the destination stream, or the store being in an inconsistent state mid-restore.

Common situations: Restoring onto a server whose JetStream storage is failing (disk full, bad block), or restoring a backup containing deleted messages onto a stream with conflicting existing state.

Related errors

• failed to process expired message sequence %d: %w
• failed to store message sequence %d: %w
• failed to process trailing gap: %w
• storage type can not be updated
• error creating store for stream

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/694bb6bb7a97964d. Report an issue: GitHub.