{"record":{"id":"26fbc50d165b76ef","repo":"nats-io/nats-server","slug":"consumer-write-error-v","errorCode":null,"errorMessage":"consumer write error: %v","messagePattern":"consumer write error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/jetstream_cluster.go","lineNumber":1150,"sourceCode":"\tcase node == nil:\n\t\treturn errors.New(\"group node missing\")\n\n\tcase oNode == nil:\n\t\t// Can happen when the consumer's node is not yet initialized.\n\t\treturn errors.New(\"consumer node missing\")\n\n\tcase node != oNode:\n\t\tmset.mu.RLock()\n\t\taccName, streamName := mset.acc.GetName(), mset.cfg.Name\n\t\tmset.mu.RUnlock()\n\t\ts.Warnf(\"Detected consumer cluster node skew '%s > %s > %s'\", accName, streamName, consumer)\n\t\treturn errors.New(\"cluster node skew detected\")\n\n\tcase nrgWerr != nil:\n\t\treturn fmt.Errorf(\"node write error: %v\", nrgWerr)\n\n\tcase consumerWerr != nil:\n\t\treturn fmt.Errorf(\"consumer write error: %v\", consumerWerr)\n\n\tcase !o.isMonitorRunning():\n\t\treturn errors.New(\"monitor goroutine not running\")\n\n\tcase !node.Healthy():\n\t\treturn errors.New(\"group node unhealthy\")\n\n\tdefault:\n\t\treturn nil\n\t}\n}\n\n// subjectsOverlap checks all existing stream assignments for the account cross-cluster for subject overlap\n// Use only for clustered JetStream\n// Read lock should be held.\nfunc (js *jetStream) subjectsOverlap(acc string, subjects []string, osa *streamAssignment) bool {\n\tfor sa := range js.streamAssignmentsOrInflightSeq(acc) {\n\t\t// can't overlap yourself, assume osa pre-checked for deep equal if passed","sourceCodeStart":1132,"sourceCodeEnd":1168,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/jetstream_cluster.go#L1132-L1168","documentation":"In NATS Server's JetStream cluster code, checkClusterHealth (or similar meta-node health check) inspects several failure modes and returns the first problem found. This specific error means the in-memory Raft log entry for a consumer assignment could not be written to the consumer's Raft group, indicating the consumer's raft node failed to append the entry. It is a cluster-consistency guard: if the consumer group's write path is broken, the server refuses to report healthy so the failure can be corrected via leader election or peer restart.","triggerScenarios":"A JetStream clustered server (nats-server running with a store cluster / Raft peers) has a consumer whose Raft group write (consumerWerr from the nrg/consumer node's Write call) returned an error during a health check, e.g. disk I/O failure, raft group shutdown, or peer loss while a consumer assignment update was being proposed.","commonSituations":"Operators see this in server health/status output or logs when a cluster node's disk is full or slow, when a consumer raft group lost quorum, or during rolling restarts where the consumer leader steps down while writes are in flight.","solutions":["Check the affected node's logs for underlying raft/disk errors and free disk space or fix I/O issues.","Verify cluster quorum: run `nats server raft list-consumers` / `nats str info <stream>` and confirm all peers are up; restart the unhealthy peer.","Delete and recreate the affected consumer if its raft state is corrupted (back up data dir first).","Upgrade nats-server to the latest patch release, as raft write error handling has had fixes."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before relying on the node, check cluster health\ninfo, _ := nc.Request(\"$SYS.REQ.SERVER.PING.JZSTREAM\", nil, time.Second*5)\n// verify consumer raft groups have leaders before writes\nif !allConsumerGroupsHaveLeaders() { scheduleRetry() }","typeGuard":null,"tryCatchPattern":"err := checkClusterHealth(ctx)\nif errors.Is(err, errConsumerWrite) || strings.Contains(err.Error(), \"consumer write error\") {\n    // backoff and re-verify cluster health before retrying\n    time.Sleep(backoff)\n    return checkClusterHealth(ctx)\n}","preventionTips":["Monitor disk usage and raft group health on every JetStream node.","Alert on leaderless raft groups via monitoring endpoints.","Avoid writing consumer assignments during rolling restarts.","Keep all nodes on the same nats-server version."],"tags":["jetstream","raft","cluster","consumer"],"backgroundTag":"raft-group-unhealthy","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}