{"record":{"id":"8b1dceed9cc285e4","repo":"nats-io/nats-server","slug":"failed-to-process-expired-message-sequence-d-w","errorCode":null,"errorMessage":"failed to process expired message sequence %d: %w","messagePattern":"failed to process expired message sequence (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/stream_backup.go","lineNumber":535,"sourceCode":"\t\t}\n\t\t// We could have deleted messages since the last message we stored, if so\n\t\t// we should work out what the gap is and skip those sequences.\n\t\tif gap := seq - lseq - 1; gap > 0 {\n\t\t\tif err := store.SkipMsgs(lseq+1, gap); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"failed to process gap: %w\", err)\n\t\t\t}\n\t\t}\n\t\tlseq = seq\n\t\tttl, err := getMessageTTL(mhdr)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse message TTL: %w\", err)\n\t\t}\n\t\thdrTime := time.Unix(0, hdr.Timestamp)\n\t\tif ttl > 0 && time.Now().After(hdrTime.Add(time.Duration(ttl)*time.Second)) {\n\t\t\t// If the TTL has exceeded then there isn't much point in storing the message,\n\t\t\t// but we still need to preserve the sequence.\n\t\t\tif err := store.SkipMsgs(seq, 1); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"failed to process expired message sequence %d: %w\", seq, err)\n\t\t\t}\n\t\t\treleaseRestoreBytes(storedSize)\n\t\t\tcontinue\n\t\t}\n\t\tif err = store.StoreRawMsg(subj, mhdr, msg, seq, hdr.Timestamp, ttl, false); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to store message sequence %d: %w\", seq, err)\n\t\t}\n\t\treleaseRestoreBytes(storedSize)\n\t}\n\n\tif !eob {\n\t\treturn mset, fmt.Errorf(\"backup was truncated\")\n\t}\n\n\t// Need to make sure that we pad out with skip msgs to preserve the last\n\t// sequence, otherwise trailing deleted messages could reuse sequence numbers.\n\tif lseq < nstate.LastSeq {\n\t\tgap := nstate.LastSeq - lseq","sourceCodeStart":517,"sourceCodeEnd":553,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/stream_backup.go#L517-L553","documentation":"When a backed-up message has a positive TTL and its timestamp plus TTL is already in the past, RestoreStreamV2 skips storing it but must preserve the sequence by calling store.SkipMsgs(seq, 1). This error wraps a failure of that skip, meaning the destination store could not reserve the expired message's sequence.","triggerScenarios":"store.SkipMsgs(seq, 1) errors while skipping an already-expired message during restore; driven by storage-layer failures (disk full, I/O error, corrupted message block in the destination stream).","commonSituations":"Restoring backups containing expired per-message-TTL messages onto a server with failing storage or a destination stream in a bad state from a previous failed restore.","solutions":["Check JetStream storage health: free disk space, filesystem errors, and server logs","Delete the partially restored stream and restore into a fresh stream","Retry after fixing the underlying storage issue reported in the server logs","Validate the destination stream's message blocks are consistent before re-attempting"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if free, err := diskFree(jsStoreDir); err != nil || free < neededBytes { return fmt.Errorf(\"insufficient storage for restore\") }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"expired message sequence\") {\n    // storage failure while skipping expired msgs; check server logs, fix storage, retry\n}","preventionTips":["Ensure ample free disk space before restoring TTL-enabled streams","Delete partial restore artifacts before retrying","Monitor storage I/O errors in server logs","Restore into a fresh stream for clean sequence reservation"],"tags":["jetstream","restore","ttl","storage"],"backgroundTag":"jetstream-storage-failure","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}