nats-io/nats-server · info

shutting down

Error message

shutting down

What it means

This error is returned by the JetStream cluster code when it is asked to create a raft group (an HA asset such as a stream or consumer mirror/replica) while the server is in the middle of shutting down. During shutdown the system account is torn down first; if `s.SystemAccount()` returns nil the code cannot proceed and aborts raft group creation with this sentinel error. It is an expected, benign race between cluster meta-leader assignments and server termination, not a bug in itself.

Source

Thrown at server/jetstream_cluster.go:3488

	if node := s.lookupRaftNode(rg.Name); node != nil {
		if node.State() == Closed {
			// We're waiting for this node to finish shutting down before we replace it.
			js.mu.Unlock()
			node.WaitForStop()
			js.mu.Lock()
			goto retry
		}
		s.Debugf("JetStream cluster already has raft group %q assigned", rg.Name)
		rg.node = node
		return node, nil
	}

	s.Debugf("JetStream cluster creating raft group:%+v", rg)

	sysAcc := s.SystemAccount()
	if sysAcc == nil {
		s.Debugf("JetStream cluster detected shutdown processing raft group: %+v", rg)
		return nil, errors.New("shutting down")
	}

	// Check here to see if we have a max HA Assets limit set.
	if maxHaAssets := s.getOpts().JetStreamLimits.MaxHAAssets; maxHaAssets > 0 {
		if s.numRaftNodes()+len(cc.creatingRaftGroups) > maxHaAssets {
			s.Warnf("Maximum HA Assets limit reached: %d", maxHaAssets)
			// Since the meta leader assigned this, send a statsz update to them to get them up to date.
			go s.sendStatszUpdate()
			return nil, errors.New("system limit reached")
		}
	}

	// Register an in-flight sentinel so concurrent callers for the same group
	// will wait for us. Then drop js.mu around all the blocking work below
	// (file store creation, peer state read, snapshot replay, fsyncs) so we
	// don't serialize every stream/consumer assignment behind one disk fsync.
	if cc.creatingRaftGroups == nil {
		cc.creatingRaftGroups = make(map[string]chan struct{})

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Verify the server was intentionally shut down; if so no action is needed — retry the JetStream asset creation once the server is back up
2. Check server logs immediately before this message for the real shutdown cause (signal, config reload, supervisor restart)
3. If unexpected, inspect systemd/supervisor/orchestrator logs to see why the process was restarted
4. Ensure provisioning automation retries asset creation against remaining cluster members
5. Upgrade NATS server if the error appears without a corresponding shutdown event

Example fix

// before: blind retry loop against a shutting-down server
s.requestCreateStream(cfg)
// after: detect shutdown and re-provision against a live peer
if err := s.requestCreateStream(cfg); err != nil && strings.Contains(err.Error(), "shutting down") {
    time.Sleep(restartGrace)
    s.requestCreateStream(cfg) // retry on restarted or peer server
}

// Before issuing JetStream clustered APIs, confirm the target server is healthy
resp, _ := http.Get("http://server:8222/healthz")
if resp == nil || resp.StatusCode != 200 { /* pick another cluster member */ }

// Retry with backoff on transient shutdown race
for i := 0; i < 3; i++ {
  err := createStream(cfg)
  if err == nil || !strings.Contains(err.Error(), "shutting down") { break }
  time.Sleep(backoff(i))
}

When it happens

Trigger: A JetStream cluster meta leader assigns a new raft group (stream or consumer creation in clustered mode) to this server at the same moment the server is shutting down; `createRaftGroup` (server/jetstream_cluster.go:3488) finds the system account already gone and returns the error.

Common situations: Rolling upgrades of a NATS cluster where streams/consumers are being created concurrently; a server being gracefully removed while automation keeps provisioning assets; fast restart loops where shutdown overlaps in-flight JetStream API requests.

Related errors

• consumer write error: %v
• %w for stream '%s > %s'
• raft: illegal peer: %q
• server is not running
• JS_CONSUMER_OFFLINE

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/28d61140aad16091. Report an issue: GitHub.