nats-io/nats-server · critical

consumer write error

Error message

consumer write error: %v

What it means

In NATS Server's JetStream cluster code, checkClusterHealth (or similar meta-node health check) inspects several failure modes and returns the first problem found. This specific error means the in-memory Raft log entry for a consumer assignment could not be written to the consumer's Raft group, indicating the consumer's raft node failed to append the entry. It is a cluster-consistency guard: if the consumer group's write path is broken, the server refuses to report healthy so the failure can be corrected via leader election or peer restart.

Solutions

1. Check the affected node's logs for underlying raft/disk errors and free disk space or fix I/O issues.
2. Verify cluster quorum: run `nats server raft list-consumers` / `nats str info <stream>` and confirm all peers are up; restart the unhealthy peer.
3. Delete and recreate the affected consumer if its raft state is corrupted (back up data dir first).
4. Upgrade nats-server to the latest patch release, as raft write error handling has had fixes.

// before relying on the node, check cluster health
info, _ := nc.Request("$SYS.REQ.SERVER.PING.JZSTREAM", nil, time.Second*5)
// verify consumer raft groups have leaders before writes
if !allConsumerGroupsHaveLeaders() { scheduleRetry() }

err := checkClusterHealth(ctx)
if errors.Is(err, errConsumerWrite) || strings.Contains(err.Error(), "consumer write error") {
    // backoff and re-verify cluster health before retrying
    time.Sleep(backoff)
    return checkClusterHealth(ctx)
}

When it happens

Trigger: A JetStream clustered server (nats-server running with a store cluster / Raft peers) has a consumer whose Raft group write (consumerWerr from the nrg/consumer node's Write call) returned an error during a health check, e.g. disk I/O failure, raft group shutdown, or peer loss while a consumer assignment update was being proposed.

Common situations: Operators see this in server health/status output or logs when a cluster node's disk is full or slow, when a consumer raft group lost quorum, or during rolling restarts where the consumer leader steps down while writes are in flight.

Related errors

• consumer closed
• JS_CONSUMER_OFFLINE
• raft: illegal peer
• shutting down
• unsupported consumer

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/26fbc50d165b76ef. Report an issue: GitHub.

Appendix: source

Thrown at server/jetstream_cluster.go:1150

	case node == nil:
		return errors.New("group node missing")

	case oNode == nil:
		// Can happen when the consumer's node is not yet initialized.
		return errors.New("consumer node missing")

	case node != oNode:
		mset.mu.RLock()
		accName, streamName := mset.acc.GetName(), mset.cfg.Name
		mset.mu.RUnlock()
		s.Warnf("Detected consumer cluster node skew '%s > %s > %s'", accName, streamName, consumer)
		return errors.New("cluster node skew detected")

	case nrgWerr != nil:
		return fmt.Errorf("node write error: %v", nrgWerr)

	case consumerWerr != nil:
		return fmt.Errorf("consumer write error: %v", consumerWerr)

	case !o.isMonitorRunning():
		return errors.New("monitor goroutine not running")

	case !node.Healthy():
		return errors.New("group node unhealthy")

	default:
		return nil
	}
}

// subjectsOverlap checks all existing stream assignments for the account cross-cluster for subject overlap
// Use only for clustered JetStream
// Read lock should be held.
func (js *jetStream) subjectsOverlap(acc string, subjects []string, osa *streamAssignment) bool {
	for sa := range js.streamAssignmentsOrInflightSeq(acc) {
		// can't overlap yourself, assume osa pre-checked for deep equal if passed

View on GitHub (pinned to 3a66a489d2)