nats-io/nats-server · error

system limit reached

Error message

system limit reached

What it means

This error is returned when the server would exceed its configured `max_ha_assets` JetStream limit: the number of existing raft nodes plus in-flight raft group creations would surpass the maximum count of HA (replicated) assets allowed on this server. Because the meta leader assigned this asset, the server proactively sends a statsz update to the leader so its view of HA asset counts is refreshed.

Source

Thrown at server/jetstream_cluster.go:3497

		rg.node = node
		return node, nil
	}

	s.Debugf("JetStream cluster creating raft group:%+v", rg)

	sysAcc := s.SystemAccount()
	if sysAcc == nil {
		s.Debugf("JetStream cluster detected shutdown processing raft group: %+v", rg)
		return nil, errors.New("shutting down")
	}

	// Check here to see if we have a max HA Assets limit set.
	if maxHaAssets := s.getOpts().JetStreamLimits.MaxHAAssets; maxHaAssets > 0 {
		if s.numRaftNodes()+len(cc.creatingRaftGroups) > maxHaAssets {
			s.Warnf("Maximum HA Assets limit reached: %d", maxHaAssets)
			// Since the meta leader assigned this, send a statsz update to them to get them up to date.
			go s.sendStatszUpdate()
			return nil, errors.New("system limit reached")
		}
	}

	// Register an in-flight sentinel so concurrent callers for the same group
	// will wait for us. Then drop js.mu around all the blocking work below
	// (file store creation, peer state read, snapshot replay, fsyncs) so we
	// don't serialize every stream/consumer assignment behind one disk fsync.
	if cc.creatingRaftGroups == nil {
		cc.creatingRaftGroups = make(map[string]chan struct{})
	}
	doneCh := make(chan struct{})
	cc.creatingRaftGroups[rg.Name] = doneCh

	// Snapshot rg fields; we drop js.mu below and rg is shared.
	rgName, rgScaleUp := rg.Name, rg.ScaleUp
	rgPeers := copyStrings(rg.Peers)
	storeDir := filepath.Join(js.config.StoreDir, sysAcc.Name, defaultStoreDirName, rg.Name)
	js.mu.Unlock()

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Count current HA assets (replicated streams and consumers) and either delete/compact unneeded ones or raise `max_ha_assets` in the server config and reload/restart
2. Run `nats stream info`/`consumer info` or check monitoring endpoints to see how many raft nodes exist on the server
3. If the limit was set for an old topology, update JetStreamLimits.MaxHAAssets to match current capacity
4. Check that the meta leader's statsz view is current; a stale leader may keep assigning assets — restart the meta leader if counts look wrong

Example fix

# before
jetstream {
  max_ha_assets = 20
}
# after: raise the limit to fit actual workload
jetstream {
  max_ha_assets = 100
}

// Check current HA asset usage against the limit before creating replicated assets
info, _ := nc.Request("$SYS.REQ.SERVER.PZ", nil, time.Second)
// parse raftnode/HA asset counts; abort creation if count >= max_ha_assets

When it happens

Trigger: Calling stream/consumer APIs that create a replicated (R>1 or R=nats cluster) asset when `s.numRaftNodes()+len(cc.creatingRaftGroups)+1 > max_ha_assets` in the server config; server/jetstream_cluster.go:3497.

Common situations: Operators setting `max_ha_assets` (often to 0-limited values to cap memory/filestore usage) and then creating more streams/consumers than budgeted; consolidating many accounts with replicated streams onto one server; forgetting the limit counts consumers too.

Related errors

• remote leafnode has same cluster name
• system account not setup
• duplicate server name
• JS_CONSUMER_OFFLINE
• JetStream cluster requires cluster name

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/84b0b87ccc1d89a0. Report an issue: GitHub.