{"record":{"id":"8b8631a7866aa8d0","repo":"nats-io/nats-server","slug":"flush-of-last-block-failed-w","errorCode":null,"errorMessage":"flush of last block failed: %w","messagePattern":"flush of last block failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/filestore.go","lineNumber":7527,"sourceCode":"\treturn nil\n}\n\n// This will check all the checksums on messages and report back any sequence numbers with errors.\nfunc (fs *fileStore) checkMsgs() (*LostStreamData, error) {\n\tfs.mu.Lock()\n\tdefer fs.mu.Unlock()\n\n\tvar firstErr error\n\tstoreErr := func(err error) error {\n\t\tfs.warn(\"checkMsgs: %v\", err)\n\t\tif firstErr == nil {\n\t\t\tfirstErr = err\n\t\t}\n\t\treturn err\n\t}\n\n\tif err := fs.checkAndFlushLastBlock(); err != nil {\n\t\treturn nil, storeErr(fmt.Errorf(\"flush of last block failed: %w\", err))\n\t}\n\n\t// Clear any global subject state.\n\tfs.psim, fs.tsl = fs.psim.Empty(), 0\n\n\tfor _, mb := range fs.blks {\n\t\t// Make sure encryption loaded if needed for the block.\n\t\tif err := fs.loadEncryptionForMsgBlock(mb); err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"loading encryption for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\t// FIXME(dlc) - check tombstones here too?\n\t\tld, _, err := mb.rebuildState()\n\t\tif err != nil {\n\t\t\t_ = storeErr(fmt.Errorf(\"rebuildState for block %d failed: %w\", mb.index, err))\n\t\t\tcontinue\n\t\t}\n\t\tif ld != nil {","sourceCodeStart":7509,"sourceCodeEnd":7545,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/filestore.go#L7509-L7545","documentation":"Wrapped by StreamRestore (or Recover/Load) when checkAndFlushLastBlock fails — the in-memory buffered last message block could not be flushed to disk. The underlying cause (I/O error, closed file, encryption error) is wrapped with %w, and the whole thing is passed through storeErr.","triggerScenarios":"Restoring a stream from a snapshot, or completing a stream load, where the final block flush to the msg file fails — disk full, I/O error, or the block's file descriptor is invalid.","commonSituations":"Stream restore finishing on a disk that is full or has become read-only; NFS/EBS timeouts during restore; permissions changed on the stream's datadir mid-operation.","solutions":["Inspect the wrapped cause: check free disk space (df -h) and datadir permissions","Retry the stream restore after clearing space or fixing mount health","Verify no external process holds/locks files in the stream directory","Check server logs for the matching storeErr output identifying the failing block"],"exampleFix":"// ensure headroom before a large restore\ndu -sh $NATS_STORE_DIR\n# free space / enlarge volume, then re-run:\nnats stream restore ORDERS orders.bak","handlingStrategy":"retry","validationCode":"// Go: check headroom before restoring a large stream\nif err := checkFreeSpace(storeDir, requiredBytes); err != nil {\n    return fmt.Errorf(\"insufficient space for restore: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"var serr *nats.APIError\nif errors.As(err, &serr) && strings.Contains(serr.Description, \"flush of last block\") {\n    if isTransient(serr.Description) { // disk pressure, I/O timeout\n        time.Sleep(retryBackoff)\n        return restoreStream() // idempotent retry\n    }\n}","preventionTips":["Keep free disk space above the size of the stream being restored","Avoid restores onto NFS/unreliable volumes; use local or healthy EBS disks","Set filesystem alerts (e.g. 80% full) before maintenance windows","Verify datadir permissions are stable for the nats-server user"],"tags":["jetstream","filestore","disk-full","io-error","stream-restore"],"backgroundTag":"disk-full-flush-failed","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}