What causes "cluster node skew detected" in nats-io/nats-server?

Let the meta group converge and re-run the health check; skew is usually transient during reassignment

cluster node skew detected

Error message

cluster node skew detected

What it means

Returned by jetStream.isStreamHealthy when the assigned Raft group node and the stream's actual Raft node are different objects (node != msetNode). This is node skew: the meta assignment points at one peer while the local stream is running another, indicating stale or divergent cluster state. The server logs a warning ('Detected stream cluster node skew') alongside the error.

Source

Thrown at server/jetstream_cluster.go:1054

	var nrgWerr error
	if node != nil {
		nrgWerr = node.GetWriteErr()
	}
	streamWerr := mset.getWriteErr()
	switch {
	case replicas <= 1:
		return nil // No further checks for R=1 streams

	case node == nil:
		return errors.New("group node missing")

	case msetNode == nil:
		// Can happen when the stream's node is not yet initialized.
		return errors.New("stream node missing")

	case node != msetNode:
		s.Warnf("Detected stream cluster node skew '%s > %s'", acc.GetName(), streamName)
		return errors.New("cluster node skew detected")

	case nrgWerr != nil:
		return fmt.Errorf("node write error: %v", nrgWerr)

	case streamWerr != nil:
		return fmt.Errorf("stream write error: %v", streamWerr)

	case !mset.isMonitorRunning():
		return errors.New("monitor goroutine not running")

	case mset.isCatchingUp():
		return errors.New("stream catching up")

	case !node.Healthy():
		return errors.New("group node unhealthy")

	default:
		return nil

View on GitHub (pinned to 3a66a489d2)

Solutions

Let the meta group converge and re-run the health check; skew is usually transient during reassignment
Verify peer state with $JS.API.SERVERS / jsz and remove stale peers ($JS.API.META server removal) if assignments are stuck
Restart the affected server to force re-sync of assignments and stream nodes

Defensive patterns

Strategy: retry

Try / catch

if err != nil && strings.Contains(err.Error(), "cluster node skew detected") {
    // check for reassignment in progress; retry after backoff, escalate if it persists
    time.Sleep(backoff)
    return healthcheck()
}

Prevention

Avoid health probes while moving/removing stream peers
Keep meta Raft healthy; stale peers cause assignment/node mismatch
After peer removal operations, verify convergence before resuming health checks

When it happens

Trigger: HEALTHCHECK on a peer where a stream was moved/reassigned and either the old or new assignment hasn't fully propagated, or after peer removal the meta group reassigned the stream to a different node than the one the local stream uses.

Common situations: Changing stream replica placement (Nats-Config-Operation or peer operations) while health checks run; meta Raft lag; partially applied peer remove operations; split-brain recovery.

Related errors

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/45177dba5f1c488e. Report an issue: GitHub.