{"record":{"id":"694bb6bb7a97964d","repo":"nats-io/nats-server","slug":"failed-to-process-gap-w","errorCode":null,"errorMessage":"failed to process gap: %w","messagePattern":"failed to process gap: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/stream_backup.go","lineNumber":522,"sourceCode":"\t\t\treturn nil, fmt.Errorf(\"failed to read message sequence %d: %w\", seq, err)\n\t\t}\n\t\tif hdr.HeaderSize > int64(len(buf)) {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse message sequence %d: invalid header length\", seq)\n\t\t}\n\t\tif int64(len(buf)) != declaredSize {\n\t\t\treturn nil, fmt.Errorf(\"failed to read message sequence %d: unexpected payload size\", seq)\n\t\t}\n\t\tsubj := hdr.Name\n\t\tmhdr := buf[:hdr.HeaderSize]\n\t\tmsg := buf[hdr.HeaderSize : hdr.HeaderSize+hdr.PayloadSize]\n\t\tif seq <= lseq {\n\t\t\treturn nil, fmt.Errorf(\"message sequence %d out of order\", seq)\n\t\t}\n\t\t// We could have deleted messages since the last message we stored, if so\n\t\t// we should work out what the gap is and skip those sequences.\n\t\tif gap := seq - lseq - 1; gap > 0 {\n\t\t\tif err := store.SkipMsgs(lseq+1, gap); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"failed to process gap: %w\", err)\n\t\t\t}\n\t\t}\n\t\tlseq = seq\n\t\tttl, err := getMessageTTL(mhdr)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse message TTL: %w\", err)\n\t\t}\n\t\thdrTime := time.Unix(0, hdr.Timestamp)\n\t\tif ttl > 0 && time.Now().After(hdrTime.Add(time.Duration(ttl)*time.Second)) {\n\t\t\t// If the TTL has exceeded then there isn't much point in storing the message,\n\t\t\t// but we still need to preserve the sequence.\n\t\t\tif err := store.SkipMsgs(seq, 1); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"failed to process expired message sequence %d: %w\", seq, err)\n\t\t\t}\n\t\t\treleaseRestoreBytes(storedSize)\n\t\t\tcontinue\n\t\t}\n\t\tif err = store.StoreRawMsg(subj, mhdr, msg, seq, hdr.Timestamp, ttl, false); err != nil {","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/stream_backup.go#L504-L540","documentation":"When RestoreStreamV2 detects a gap between the previous sequence and the current one (deleted messages in the source), it calls store.SkipMsgs(lseq+1, gap) to reserve those sequences. This error wraps a failure of that skip operation on the target stream's message store, meaning the destination store could not advance past the missing sequences.","triggerScenarios":"store.SkipMsgs returns an error while padding a gap; typically due to underlying storage failure (disk full, I/O error), message block corruption in the destination stream, or the store being in an inconsistent state mid-restore.","commonSituations":"Restoring onto a server whose JetStream storage is failing (disk full, bad block), or restoring a backup containing deleted messages onto a stream with conflicting existing state.","solutions":["Check disk space and filesystem health on the JetStream storage directory","Remove/clean the partially restored stream and retry the restore into a fresh stream","Inspect server logs for the underlying SkipMsgs/storage error","Verify the storage backend (file store) is not corrupted by running server restart and letting JetStream recover"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before restore: check storage health\nif free, err := diskFree(jsStoreDir); err != nil || free < neededBytes { return fmt.Errorf(\"insufficient JetStream storage\") }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"failed to process gap\") {\n    // inspect server logs for storage error; free space, then restore into a fresh stream\n}","preventionTips":["Monitor disk space on JetStream store directories","Restore into new/empty streams to avoid state conflicts","Fix filesystem errors before large restores","Review server logs for SkipMsgs/storage failures after any failed restore"],"tags":["jetstream","restore","storage"],"backgroundTag":"jetstream-storage-failure","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}