{"record":{"id":"4851bf6f06d4827f","repo":"nats-io/nats-server","slug":"rebuildstate-for-block-d-failed-w","errorCode":null,"errorMessage":"rebuildState for block %d failed: %w","messagePattern":"rebuildState for block (.+?) failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/filestore.go","lineNumber":7542,"sourceCode":"\t}\n\n\tif err := fs.checkAndFlushLastBlock(); err != nil {\n\t\treturn nil, storeErr(fmt.Errorf(\"flush of last block failed: %w\", err))\n\t}\n\n\t// Clear any global subject state.\n\tfs.psim, fs.tsl = fs.psim.Empty(), 0\n\n\tfor _, mb := range fs.blks {\n\t\t// Make sure encryption loaded if needed for the block.\n\t\tif err := fs.loadEncryptionForMsgBlock(mb); err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"loading encryption for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\t// FIXME(dlc) - check tombstones here too?\n\t\tld, _, err := mb.rebuildState()\n\t\tif err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"rebuildState for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\tif ld != nil {\n\t\t\t// Rebuild fs state too.\n\t\t\tfs.rebuildStateLocked(ld)\n\t\t}\n\t\tif err = fs.populateGlobalPerSubjectInfo(mb); err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"populating per-subject info for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t}\n\n\treturn fs.ld, firstErr\n}\n\n// Lock should be held.\nfunc (mb *msgBlock) enableForWriting(fip bool) error {\n\tif mb == nil {","sourceCodeStart":7524,"sourceCodeEnd":7560,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/filestore.go#L7524-L7560","documentation":"Emitted during stream recovery when a message block's rebuildState fails — the store could not reconstruct the block's in-memory index (message offsets, sequence mappings) from the file. The block is skipped and the error recorded via storeErr, so the stream recovers with potentially missing messages from that block.","triggerScenarios":"Recover() iterating fs.blks where mb.rebuildState() returns an error: truncated or partially written .blk file, corrupt per-message metadata, mismatch between file size and recorded write offsets after a crash.","commonSituations":"Power loss / kill -9 without clean shutdown leaving a partially flushed block; disk corruption; manually editing or copying incomplete stream files; storage layer (NFS) losing writes.","solutions":["Read the wrapped cause in server logs — it pinpoints the block index and failure","Restart the server; JetStream may recover other blocks and mark this one bad","If block data is expendable, remove that block's .blk/.fst files so recovery skips it","Check storage health (fsck, SMART) and re-restore the stream from a backup","Enable/disable filestore sync options and upgrade NATS — rebuildState has been hardened over releases"],"exampleFix":"// skip a corrupt block so the stream recovers with earlier data\nrm /path/jetstream/$ACC/streams/ORDERS/msgblk/7.blk\nrm /path/jetstream/$ACC/streams/ORDERS/msgblk/7.fst\n# then restart nats-server","handlingStrategy":"validation","validationCode":"// Go: confirm stream state matches expectations after any recovery error\ninfo, err := js.StreamInfo(ctx, \"ORDERS\")\nif err == nil && info.State.FirstSeq > expectedFirstSeq {\n    log.Printf(\"blocks were skipped during recovery: first seq now %d\", info.State.FirstSeq)\n}","typeGuard":null,"tryCatchPattern":"if err != nil {\n    if strings.Contains(err.Error(), \"rebuildState for block\") {\n        // partial recovery: a block was skipped\n        // 1. capture server logs  2. restore from backup or accept data loss\n        return reconcileAfterBlockLoss(js, \"ORDERS\")\n    }\n    return err\n}","preventionTips":["Use UPS/journalling or battery-backed disks where data loss is unacceptable","Always shut down nats-server gracefully (SIGTERM) before host restarts","Take regular `nats stream backup` snapshots","Don't hand-edit or truncate .blk/.fst files while the server runs"],"tags":["jetstream","filestore","corruption","rebuild-state","stream-recovery"],"backgroundTag":"stream-block-corruption","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}