{"record":{"id":"84b0b87ccc1d89a0","repo":"nats-io/nats-server","slug":"system-limit-reached","errorCode":null,"errorMessage":"system limit reached","messagePattern":"system limit reached","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/jetstream_cluster.go","lineNumber":3497,"sourceCode":"\t\trg.node = node\n\t\treturn node, nil\n\t}\n\n\ts.Debugf(\"JetStream cluster creating raft group:%+v\", rg)\n\n\tsysAcc := s.SystemAccount()\n\tif sysAcc == nil {\n\t\ts.Debugf(\"JetStream cluster detected shutdown processing raft group: %+v\", rg)\n\t\treturn nil, errors.New(\"shutting down\")\n\t}\n\n\t// Check here to see if we have a max HA Assets limit set.\n\tif maxHaAssets := s.getOpts().JetStreamLimits.MaxHAAssets; maxHaAssets > 0 {\n\t\tif s.numRaftNodes()+len(cc.creatingRaftGroups) > maxHaAssets {\n\t\t\ts.Warnf(\"Maximum HA Assets limit reached: %d\", maxHaAssets)\n\t\t\t// Since the meta leader assigned this, send a statsz update to them to get them up to date.\n\t\t\tgo s.sendStatszUpdate()\n\t\t\treturn nil, errors.New(\"system limit reached\")\n\t\t}\n\t}\n\n\t// Register an in-flight sentinel so concurrent callers for the same group\n\t// will wait for us. Then drop js.mu around all the blocking work below\n\t// (file store creation, peer state read, snapshot replay, fsyncs) so we\n\t// don't serialize every stream/consumer assignment behind one disk fsync.\n\tif cc.creatingRaftGroups == nil {\n\t\tcc.creatingRaftGroups = make(map[string]chan struct{})\n\t}\n\tdoneCh := make(chan struct{})\n\tcc.creatingRaftGroups[rg.Name] = doneCh\n\n\t// Snapshot rg fields; we drop js.mu below and rg is shared.\n\trgName, rgScaleUp := rg.Name, rg.ScaleUp\n\trgPeers := copyStrings(rg.Peers)\n\tstoreDir := filepath.Join(js.config.StoreDir, sysAcc.Name, defaultStoreDirName, rg.Name)\n\tjs.mu.Unlock()","sourceCodeStart":3479,"sourceCodeEnd":3515,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/jetstream_cluster.go#L3479-L3515","documentation":"This error is returned when the server would exceed its configured `max_ha_assets` JetStream limit: the number of existing raft nodes plus in-flight raft group creations would surpass the maximum count of HA (replicated) assets allowed on this server. Because the meta leader assigned this asset, the server proactively sends a statsz update to the leader so its view of HA asset counts is refreshed.","triggerScenarios":"Calling stream/consumer APIs that create a replicated (R>1 or R=nats cluster) asset when `s.numRaftNodes()+len(cc.creatingRaftGroups)+1 > max_ha_assets` in the server config; server/jetstream_cluster.go:3497.","commonSituations":"Operators setting `max_ha_assets` (often to 0-limited values to cap memory/filestore usage) and then creating more streams/consumers than budgeted; consolidating many accounts with replicated streams onto one server; forgetting the limit counts consumers too.","solutions":["Count current HA assets (replicated streams and consumers) and either delete/compact unneeded ones or raise `max_ha_assets` in the server config and reload/restart","Run `nats stream info`/`consumer info` or check monitoring endpoints to see how many raft nodes exist on the server","If the limit was set for an old topology, update JetStreamLimits.MaxHAAssets to match current capacity","Check that the meta leader's statsz view is current; a stale leader may keep assigning assets — restart the meta leader if counts look wrong"],"exampleFix":"# before\njetstream {\n  max_ha_assets = 20\n}\n# after: raise the limit to fit actual workload\njetstream {\n  max_ha_assets = 100\n}","handlingStrategy":"validation","validationCode":"// Check current HA asset usage against the limit before creating replicated assets\ninfo, _ := nc.Request(\"$SYS.REQ.SERVER.PZ\", nil, time.Second)\n// parse raftnode/HA asset counts; abort creation if count >= max_ha_assets","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Budget max_ha_assets as streams+consumers combined, per server","Monitor num raft nodes via monitoring endpoints and alert at 80% of the limit","Clean up orphaned consumers which count toward the limit"],"tags":["jetstream","cluster","limits","ha-assets","configuration"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}