hashicorp/nomad · error

rpc_max_conns_per_client must be > %d; found: %d

Error message

rpc_max_conns_per_client must be > %d; found: %d

What it means

limits.rpc_max_conns_per_client caps non-streaming RPC connections per client. Nomad reserves room for streaming RPCs, so the value must exceed config.LimitsNonStreamingConnsPerClient + 5; a positive value at or below that minimum is rejected at startup/reload. Setting nil or 0 means unlimited and is allowed.

Source

Thrown at command/agent/agent.go:609

	conf.DisableDispatchedJobSummaryMetrics = agentConfig.Telemetry.DisableDispatchedJobSummaryMetrics
	conf.DisableQuotaUtilizationMetrics = agentConfig.Telemetry.DisableQuotaUtilizationMetrics
	conf.DisableRPCRateMetricsLabels = agentConfig.Telemetry.DisableRPCRateMetricsLabels

	if d, err := time.ParseDuration(agentConfig.Limits.RPCHandshakeTimeout); err != nil {
		return nil, fmt.Errorf("error parsing rpc_handshake_timeout: %v", err)
	} else if d < 0 {
		return nil, fmt.Errorf("rpc_handshake_timeout must be >= 0")
	} else {
		conf.RPCHandshakeTimeout = d
	}

	// Set max rpc conns; nil/0 == unlimited
	// Leave a little room for streaming RPCs
	minLimit := config.LimitsNonStreamingConnsPerClient + 5
	if agentConfig.Limits.RPCMaxConnsPerClient == nil || *agentConfig.Limits.RPCMaxConnsPerClient == 0 {
		conf.RPCMaxConnsPerClient = 0
	} else if limit := *agentConfig.Limits.RPCMaxConnsPerClient; limit <= minLimit {
		return nil, fmt.Errorf("rpc_max_conns_per_client must be > %d; found: %d", minLimit, limit)
	} else {
		conf.RPCMaxConnsPerClient = limit
	}

	// Set deployment rate limit
	if rate := agentConfig.Server.DeploymentQueryRateLimit; rate == 0 {
		conf.DeploymentQueryRateLimit = deploymentwatcher.LimitStateQueriesPerSecond
	} else if rate > 0 {
		conf.DeploymentQueryRateLimit = rate
	} else {
		return nil, fmt.Errorf("deploy_query_rate_limit must be greater than 0")
	}

	// Set plan rejection tracker configuration.
	if planRejectConf := agentConfig.Server.PlanRejectionTracker; planRejectConf != nil {
		if planRejectConf.Enabled != nil {
			conf.NodePlanRejectionEnabled = *planRejectConf.Enabled
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Raise rpc_max_conns_per_client above the printed minimum (the error message states the required minLimit value)
2. Remove the setting or set it to 0 for unlimited connections
3. Scale down the number of clients instead of the per-client connection cap if resources are constrained
4. Check the Nomad version's config.LimitsNonStreamingConnsPerClient constant to know the exact floor

Example fix

// before
limits {
  rpc_max_conns_per_client = 10
}
// after
limits {
  rpc_max_conns_per_client = 100
}

// Go: check the value against the known floor before applying
const minLimit = config.LimitsNonStreamingConnsPerClient + 5
if p := cfg.Limits.RPCMaxConnsPerClient; p != nil && *p != 0 && *p <= minLimit {
    return fmt.Errorf("rpc_max_conns_per_client must be > %d", minLimit)
}

func validMaxConns(p *int) bool {
    return p == nil || *p == 0 || *p > config.LimitsNonStreamingConnsPerClient+5
}

When it happens

Trigger: Setting limits { rpc_max_conns_per_client = 10 } (or any small positive number <= minLimit) in a server agent config and starting the agent or reloading config via handleReload.

Common situations: Operators aggressively lowering connection limits on small clusters without knowing the internal floor; configs tuned from older versions where the minimum was lower; copy-paste of small round numbers like 5 or 10.

Related errors

• wait config is nil or empty
• missing datacenter for client registration
• default_identity_ttl must be greater than 0
• max_identity_ttl must be greater than 0
• max_identity_ttl must be greater than or equal to default_id

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/434173c2e714661a. Report an issue: GitHub.