{"record":{"id":"45177dba5f1c488e","repo":"nats-io/nats-server","slug":"cluster-node-skew-detected","errorCode":null,"errorMessage":"cluster node skew detected","messagePattern":"cluster node skew detected","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/jetstream_cluster.go","lineNumber":1054,"sourceCode":"\tvar nrgWerr error\n\tif node != nil {\n\t\tnrgWerr = node.GetWriteErr()\n\t}\n\tstreamWerr := mset.getWriteErr()\n\tswitch {\n\tcase replicas <= 1:\n\t\treturn nil // No further checks for R=1 streams\n\n\tcase node == nil:\n\t\treturn errors.New(\"group node missing\")\n\n\tcase msetNode == nil:\n\t\t// Can happen when the stream's node is not yet initialized.\n\t\treturn errors.New(\"stream node missing\")\n\n\tcase node != msetNode:\n\t\ts.Warnf(\"Detected stream cluster node skew '%s > %s'\", acc.GetName(), streamName)\n\t\treturn errors.New(\"cluster node skew detected\")\n\n\tcase nrgWerr != nil:\n\t\treturn fmt.Errorf(\"node write error: %v\", nrgWerr)\n\n\tcase streamWerr != nil:\n\t\treturn fmt.Errorf(\"stream write error: %v\", streamWerr)\n\n\tcase !mset.isMonitorRunning():\n\t\treturn errors.New(\"monitor goroutine not running\")\n\n\tcase mset.isCatchingUp():\n\t\treturn errors.New(\"stream catching up\")\n\n\tcase !node.Healthy():\n\t\treturn errors.New(\"group node unhealthy\")\n\n\tdefault:\n\t\treturn nil","sourceCodeStart":1036,"sourceCodeEnd":1072,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/jetstream_cluster.go#L1036-L1072","documentation":"Returned by jetStream.isStreamHealthy when the assigned Raft group node and the stream's actual Raft node are different objects (node != msetNode). This is node skew: the meta assignment points at one peer while the local stream is running another, indicating stale or divergent cluster state. The server logs a warning ('Detected stream cluster node skew') alongside the error.","triggerScenarios":"HEALTHCHECK on a peer where a stream was moved/reassigned and either the old or new assignment hasn't fully propagated, or after peer removal the meta group reassigned the stream to a different node than the one the local stream uses.","commonSituations":"Changing stream replica placement (Nats-Config-Operation or peer operations) while health checks run; meta Raft lag; partially applied peer remove operations; split-brain recovery.","solutions":["Let the meta group converge and re-run the health check; skew is usually transient during reassignment","Verify peer state with $JS.API.SERVERS / jsz and remove stale peers ($JS.API.META server removal) if assignments are stuck","Restart the affected server to force re-sync of assignments and stream nodes"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"cluster node skew detected\") {\n    // check for reassignment in progress; retry after backoff, escalate if it persists\n    time.Sleep(backoff)\n    return healthcheck()\n}","preventionTips":["Avoid health probes while moving/removing stream peers","Keep meta Raft healthy; stale peers cause assignment/node mismatch","After peer removal operations, verify convergence before resuming health checks"],"tags":["jetstream","raft","clustering","skew","healthcheck"],"backgroundTag":"jetstream-cluster-node-skew","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}