{"record":{"id":"28d61140aad16091","repo":"nats-io/nats-server","slug":"shutting-down","errorCode":null,"errorMessage":"shutting down","messagePattern":"shutting down","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"server/jetstream_cluster.go","lineNumber":3488,"sourceCode":"\tif node := s.lookupRaftNode(rg.Name); node != nil {\n\t\tif node.State() == Closed {\n\t\t\t// We're waiting for this node to finish shutting down before we replace it.\n\t\t\tjs.mu.Unlock()\n\t\t\tnode.WaitForStop()\n\t\t\tjs.mu.Lock()\n\t\t\tgoto retry\n\t\t}\n\t\ts.Debugf(\"JetStream cluster already has raft group %q assigned\", rg.Name)\n\t\trg.node = node\n\t\treturn node, nil\n\t}\n\n\ts.Debugf(\"JetStream cluster creating raft group:%+v\", rg)\n\n\tsysAcc := s.SystemAccount()\n\tif sysAcc == nil {\n\t\ts.Debugf(\"JetStream cluster detected shutdown processing raft group: %+v\", rg)\n\t\treturn nil, errors.New(\"shutting down\")\n\t}\n\n\t// Check here to see if we have a max HA Assets limit set.\n\tif maxHaAssets := s.getOpts().JetStreamLimits.MaxHAAssets; maxHaAssets > 0 {\n\t\tif s.numRaftNodes()+len(cc.creatingRaftGroups) > maxHaAssets {\n\t\t\ts.Warnf(\"Maximum HA Assets limit reached: %d\", maxHaAssets)\n\t\t\t// Since the meta leader assigned this, send a statsz update to them to get them up to date.\n\t\t\tgo s.sendStatszUpdate()\n\t\t\treturn nil, errors.New(\"system limit reached\")\n\t\t}\n\t}\n\n\t// Register an in-flight sentinel so concurrent callers for the same group\n\t// will wait for us. Then drop js.mu around all the blocking work below\n\t// (file store creation, peer state read, snapshot replay, fsyncs) so we\n\t// don't serialize every stream/consumer assignment behind one disk fsync.\n\tif cc.creatingRaftGroups == nil {\n\t\tcc.creatingRaftGroups = make(map[string]chan struct{})","sourceCodeStart":3470,"sourceCodeEnd":3506,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/jetstream_cluster.go#L3470-L3506","documentation":"This error is returned by the JetStream cluster code when it is asked to create a raft group (an HA asset such as a stream or consumer mirror/replica) while the server is in the middle of shutting down. During shutdown the system account is torn down first; if `s.SystemAccount()` returns nil the code cannot proceed and aborts raft group creation with this sentinel error. It is an expected, benign race between cluster meta-leader assignments and server termination, not a bug in itself.","triggerScenarios":"A JetStream cluster meta leader assigns a new raft group (stream or consumer creation in clustered mode) to this server at the same moment the server is shutting down; `createRaftGroup` (server/jetstream_cluster.go:3488) finds the system account already gone and returns the error.","commonSituations":"Rolling upgrades of a NATS cluster where streams/consumers are being created concurrently; a server being gracefully removed while automation keeps provisioning assets; fast restart loops where shutdown overlaps in-flight JetStream API requests.","solutions":["Verify the server was intentionally shut down; if so no action is needed — retry the JetStream asset creation once the server is back up","Check server logs immediately before this message for the real shutdown cause (signal, config reload, supervisor restart)","If unexpected, inspect systemd/supervisor/orchestrator logs to see why the process was restarted","Ensure provisioning automation retries asset creation against remaining cluster members","Upgrade NATS server if the error appears without a corresponding shutdown event"],"exampleFix":"// before: blind retry loop against a shutting-down server\ns.requestCreateStream(cfg)\n// after: detect shutdown and re-provision against a live peer\nif err := s.requestCreateStream(cfg); err != nil && strings.Contains(err.Error(), \"shutting down\") {\n    time.Sleep(restartGrace)\n    s.requestCreateStream(cfg) // retry on restarted or peer server\n}","handlingStrategy":"retry","validationCode":"// Before issuing JetStream clustered APIs, confirm the target server is healthy\nresp, _ := http.Get(\"http://server:8222/healthz\")\nif resp == nil || resp.StatusCode != 200 { /* pick another cluster member */ }","typeGuard":null,"tryCatchPattern":"// Retry with backoff on transient shutdown race\nfor i := 0; i < 3; i++ {\n  err := createStream(cfg)\n  if err == nil || !strings.Contains(err.Error(), \"shutting down\") { break }\n  time.Sleep(backoff(i))\n}","preventionTips":["Check /healthz before issuing provisioning requests during rolling restarts","Drain workloads from a server before shutting it down (nats-server -sl quit after removing it from routing)","Make provisioning automation idempotent and retry-aware"],"tags":["jetstream","cluster","shutdown","raft"],"backgroundTag":"server-shutdown-in-progress","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}