nats-io/nats-server · critical

consumer write error: %v

Error message

consumer write error: %v

What it means

In NATS Server's JetStream cluster code, checkClusterHealth (or similar meta-node health check) inspects several failure modes and returns the first problem found. This specific error means the in-memory Raft log entry for a consumer assignment could not be written to the consumer's Raft group, indicating the consumer's raft node failed to append the entry. It is a cluster-consistency guard: if the consumer group's write path is broken, the server refuses to report healthy so the failure can be corrected via leader election or peer restart.

Source

Thrown at server/jetstream_cluster.go:1150

	case node == nil:
		return errors.New("group node missing")

	case oNode == nil:
		// Can happen when the consumer's node is not yet initialized.
		return errors.New("consumer node missing")

	case node != oNode:
		mset.mu.RLock()
		accName, streamName := mset.acc.GetName(), mset.cfg.Name
		mset.mu.RUnlock()
		s.Warnf("Detected consumer cluster node skew '%s > %s > %s'", accName, streamName, consumer)
		return errors.New("cluster node skew detected")

	case nrgWerr != nil:
		return fmt.Errorf("node write error: %v", nrgWerr)

	case consumerWerr != nil:
		return fmt.Errorf("consumer write error: %v", consumerWerr)

	case !o.isMonitorRunning():
		return errors.New("monitor goroutine not running")

	case !node.Healthy():
		return errors.New("group node unhealthy")

	default:
		return nil
	}
}

// subjectsOverlap checks all existing stream assignments for the account cross-cluster for subject overlap
// Use only for clustered JetStream
// Read lock should be held.
func (js *jetStream) subjectsOverlap(acc string, subjects []string, osa *streamAssignment) bool {
	for sa := range js.streamAssignmentsOrInflightSeq(acc) {
		// can't overlap yourself, assume osa pre-checked for deep equal if passed

View on GitHub (pinned to 3a66a489d2)

Solutions

  1. Check the affected node's logs for underlying raft/disk errors and free disk space or fix I/O issues.
  2. Verify cluster quorum: run `nats server raft list-consumers` / `nats str info <stream>` and confirm all peers are up; restart the unhealthy peer.
  3. Delete and recreate the affected consumer if its raft state is corrupted (back up data dir first).
  4. Upgrade nats-server to the latest patch release, as raft write error handling has had fixes.
Defensive patterns

Strategy: retry

Validate before calling

// before relying on the node, check cluster health
info, _ := nc.Request("$SYS.REQ.SERVER.PING.JZSTREAM", nil, time.Second*5)
// verify consumer raft groups have leaders before writes
if !allConsumerGroupsHaveLeaders() { scheduleRetry() }

Try / catch

err := checkClusterHealth(ctx)
if errors.Is(err, errConsumerWrite) || strings.Contains(err.Error(), "consumer write error") {
    // backoff and re-verify cluster health before retrying
    time.Sleep(backoff)
    return checkClusterHealth(ctx)
}

Prevention

When it happens

Trigger: A JetStream clustered server (nats-server running with a store cluster / Raft peers) has a consumer whose Raft group write (consumerWerr from the nrg/consumer node's Write call) returned an error during a health check, e.g. disk I/O failure, raft group shutdown, or peer loss while a consumer assignment update was being proposed.

Common situations: Operators see this in server health/status output or logs when a cluster node's disk is full or slow, when a consumer raft group lost quorum, or during rolling restarts where the consumer leader steps down while writes are in flight.

Related errors

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/26fbc50d165b76ef. Report an issue: GitHub.