{"record":{"id":"29c3206ac393974b","repo":"nats-io/nats-server","slug":"populating-per-subject-info-for-block-d-failed","errorCode":null,"errorMessage":"populating per-subject info for block %d failed: %w","messagePattern":"populating per-subject info for block (.+?) failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/filestore.go","lineNumber":7550,"sourceCode":"\n\tfor _, mb := range fs.blks {\n\t\t// Make sure encryption loaded if needed for the block.\n\t\tif err := fs.loadEncryptionForMsgBlock(mb); err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"loading encryption for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\t// FIXME(dlc) - check tombstones here too?\n\t\tld, _, err := mb.rebuildState()\n\t\tif err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"rebuildState for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\tif ld != nil {\n\t\t\t// Rebuild fs state too.\n\t\t\tfs.rebuildStateLocked(ld)\n\t\t}\n\t\tif err = fs.populateGlobalPerSubjectInfo(mb); err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"populating per-subject info for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t}\n\n\treturn fs.ld, firstErr\n}\n\n// Lock should be held.\nfunc (mb *msgBlock) enableForWriting(fip bool) error {\n\tif mb == nil {\n\t\treturn errNoMsgBlk\n\t}\n\tif mb.mfd != nil {\n\t\treturn nil\n\t}\n\tmb.fs.dios.acquire()\n\tmfd, err := os.OpenFile(mb.mfn, os.O_CREATE|os.O_RDWR, defaultFilePerms)\n\tmb.fs.dios.release()","sourceCodeStart":7532,"sourceCodeEnd":7568,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/filestore.go#L7532-L7568","documentation":"Emitted during stream recovery when populateGlobalPerSubjectInfo fails for a block. After rebuildState succeeds, the store builds per-subject index info for subject-filtered consumers; failing here means the block's subject metadata could not be processed. The error is recorded via storeErr and that block is skipped in the per-subject pass.","triggerScenarios":"Recover() on a stream with multi-subject (wildcard/multi-filter) data where the block's subject index entries are corrupt or inconsistent with the rebuilt state.","commonSituations":"Corruption from unclean shutdown on a stream that used many distinct subjects; OOM/interrupted writes while subject maps were being flushed; version mismatches in on-disk subject index format after an upgrade.","solutions":["Check the wrapped cause in server logs for the failing block and underlying error","Restart the server to retry recovery after any transient storage issue","If one block is consistently bad, remove that block's files and re-create/resync the stream","Verify no NATS version downgrade occurred (older binary reading newer on-disk format); align versions","Restore the stream from backup if per-subject data is required"],"exampleFix":"// align server version before restarting recovery\nnats-server -v\n# upgrade/downgrade to the version that wrote the stream, then restart\nsystemctl restart nats-server","handlingStrategy":"validation","validationCode":"// Go: after recovery, verify per-subject counts for filtered consumers\ninfo, err := js.StreamInfo(ctx, \"ORDERS\")\nif err == nil {\n    for subj, n := range info.State.Subjects {\n        if n == 0 && expectMsgs(subj) {\n            log.Printf(\"subject %s lost messages — per-subject info skipped for a block\", subj)\n        }\n    }\n}","typeGuard":null,"tryCatchPattern":"if err != nil {\n    if strings.Contains(err.Error(), \"populating per-subject info for block\") {\n        // filtered consumers may miss msgs in the skipped block\n        return repopulateOrResync(js, \"ORDERS\")\n    }\n    return err\n}","preventionTips":["Avoid NATS version downgrades on existing datadirs; check versions before upgrades","Gracefully stop the server before storage maintenance","Monitor `nats stream info --subjects` counts against expected traffic","Back up streams that use multi-filter consumers"],"tags":["jetstream","filestore","per-subject-index","stream-recovery"],"backgroundTag":"stream-block-corruption","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}