{"record":{"id":"d4af8215b8dc7371","repo":"nats-io/nats-server","slug":"node-write-error-v","errorCode":null,"errorMessage":"node write error: %v","messagePattern":"node write error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/jetstream_cluster.go","lineNumber":1057,"sourceCode":"\t}\n\tstreamWerr := mset.getWriteErr()\n\tswitch {\n\tcase replicas <= 1:\n\t\treturn nil // No further checks for R=1 streams\n\n\tcase node == nil:\n\t\treturn errors.New(\"group node missing\")\n\n\tcase msetNode == nil:\n\t\t// Can happen when the stream's node is not yet initialized.\n\t\treturn errors.New(\"stream node missing\")\n\n\tcase node != msetNode:\n\t\ts.Warnf(\"Detected stream cluster node skew '%s > %s'\", acc.GetName(), streamName)\n\t\treturn errors.New(\"cluster node skew detected\")\n\n\tcase nrgWerr != nil:\n\t\treturn fmt.Errorf(\"node write error: %v\", nrgWerr)\n\n\tcase streamWerr != nil:\n\t\treturn fmt.Errorf(\"stream write error: %v\", streamWerr)\n\n\tcase !mset.isMonitorRunning():\n\t\treturn errors.New(\"monitor goroutine not running\")\n\n\tcase mset.isCatchingUp():\n\t\treturn errors.New(\"stream catching up\")\n\n\tcase !node.Healthy():\n\t\treturn errors.New(\"group node unhealthy\")\n\n\tdefault:\n\t\treturn nil\n\t}\n}\n","sourceCodeStart":1039,"sourceCodeEnd":1075,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/jetstream_cluster.go#L1039-L1075","documentation":"During the stream portion of a JetStream health check, the node/raft group write returned an error (nrgWerr) when checking or updating the stream's raft node state. The health check reports 'node write error' with the underlying error, indicating the stream's meta/node raft group cannot be written to — typically an unavailable quorum or a shutdown raft group.","triggerScenarios":"Raft group for the stream has no quorum (majority of peers down or partitioned); raft group already shut down (server stopping or stream being deleted); disk I/O failure on the raft log preventing writes; calling health checks during leader transitions.","commonSituations":"Cluster degraded after node loss so raft writes fail; snapshot/compaction issues or full disk on peers; health endpoints probed by load balancers while the cluster is recovering.","solutions":["Restore raft quorum: restart or repair the missing peers so the stream's raft group has a majority","Check server logs and disk space/free space on raft log directories on all peers","If the stream is being deleted/replaced, wait for the operation to finish and re-run health checks","As a last resort, use the stream reset/recovery tooling (or recreate the stream) if the raft state is irrecoverably corrupt"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before probing health, verify quorum peers are reachable\nfor _, peer := range clusterPeers { if !reachable(peer) { skipHealthCheck(peer) } }\n","typeGuard":"func isNodeWriteErr(err error) bool { return strings.HasPrefix(err.Error(), \"node write error\") }\n","tryCatchPattern":"if err := healthz(); err != nil && strings.HasPrefix(err.Error(), \"node write error\") {\n    // quorum/raft issue: back off and retry after peers recover\n}\n","preventionTips":["Ensure raft log directories have free disk space and correct permissions","Monitor cluster member health and alert before quorum is lost","Avoid running health probes during shutdown/leader transitions"],"tags":["jetstream","raft","health-check","quorum"],"backgroundTag":"jetstream-raft-write-failed","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}