{"record":{"id":"6dca2a8f4284284a","repo":"nats-io/nats-server","slug":"error-opening-msg-block-file-q-v","errorCode":null,"errorMessage":"error opening msg block file [%q]: %v","messagePattern":"error opening msg block file \\[%q\\]: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/filestore.go","lineNumber":7570,"sourceCode":"\t\t}\n\t}\n\n\treturn fs.ld, firstErr\n}\n\n// Lock should be held.\nfunc (mb *msgBlock) enableForWriting(fip bool) error {\n\tif mb == nil {\n\t\treturn errNoMsgBlk\n\t}\n\tif mb.mfd != nil {\n\t\treturn nil\n\t}\n\tmb.fs.dios.acquire()\n\tmfd, err := os.OpenFile(mb.mfn, os.O_CREATE|os.O_RDWR, defaultFilePerms)\n\tmb.fs.dios.release()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error opening msg block file [%q]: %v\", mb.mfn, err)\n\t}\n\tmb.mfd = mfd\n\n\t// Spin up our flusher loop if needed.\n\tif !fip {\n\t\tmb.spinUpFlushLoopLocked()\n\t}\n\n\treturn nil\n}\n\n// Helper function to place a delete tombstone.\nfunc (mb *msgBlock) writeTombstone(seq uint64, ts int64) error {\n\treturn mb.writeMsgRecord(emptyRecordLen, seq|tbit, _EMPTY_, nil, nil, ts, true)\n}\n\n// Helper function to place a delete tombstone without flush.\n// Lock should not be held.","sourceCodeStart":7552,"sourceCodeEnd":7588,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/filestore.go#L7552-L7588","documentation":"Returned when opening a message block's backing file (os.OpenFile with O_CREATE|O_RDWR) fails while recovering a stream. The block cannot be brought online, so the stream recovery for it fails. The wrapped %v carries the OS-level cause (permission denied, no such file, too many open files, etc.).","triggerScenarios":"Recover/openStream where mb.mfn cannot be opened: wrong permissions on the .blk file, datadir on a dead mount, or fd exhaustion (EMFILE) when a stream has thousands of blocks.","commonSituations":"Running nats-server as a different user than the datadir owner; NFS mount hung; ulimit -n too low for large streams with many blocks; SELinux/AppArmor blocking file access.","solutions":["Read the wrapped OS error: fix permissions (chown/chmod) or remount the storage if needed","Raise the file descriptor limit (ulimit -n / LimitNOFILE= high value in systemd) for large streams","Verify the datadir path is correct and the server user owns the jetstream directory","If the file is missing entirely, restore it from backup or remove the block record"],"exampleFix":"# before: EPERM/EMFILE on block files\nchown -R nats:nats /var/lib/nats/jetstream\n# systemd unit\n[Service]\nLimitNOFILE=1000000","handlingStrategy":"validation","validationCode":"// Go: pre-flight checks before starting/recovering a server\n// 1. user owns datadir, 2. fd limit high enough for block count\nif err := checkDatadirWritable(storeDir, runAsUser); err != nil {\n    return fmt.Errorf(\"datadir not writable by %s: %w\", runAsUser, err)\n}\nblocks, _ := filepath.Glob(filepath.Join(storeDir, \"streams/*/msgblk/*.blk\"))\nif len(blocks) > fdSoftLimit/4 {\n    log.Printf(\"raise LimitNOFILE: %d blocks may exhaust fds\", len(blocks))\n}","typeGuard":null,"tryCatchPattern":"if err != nil {\n    if strings.Contains(err.Error(), \"error opening msg block file\") && strings.Contains(err.Error(), \"permission denied\") {\n        return fixOwnership(storeDir) // chown -R nats:nats, then re-open\n    }\n    if strings.Contains(err.Error(), \"too many open files\") {\n        return raiseFdLimitAndRetry()\n    }\n    return err\n}","preventionTips":["Run nats-server as the same user that owns the jetstream datadir","Set LimitNOFILE=1000000 in the systemd unit for large streams","Avoid NFS for jetstream storage; use local disks","Add startup health checks that verify datadir readability before accepting traffic"],"tags":["jetstream","filestore","permissions","too-many-open-files","stream-recovery"],"backgroundTag":"file-open-permission-denied","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}