hashicorp/nomad · warning · noServersErr

no servers

Error message

no servers

What it means

noServersErr is a sentinel error returned by the Nomad client's RPC method when the client has no servers configured or discovered to talk to. It is not a hard failure: the client uses it as an internal signal to trigger Consul-based server discovery. Callers are expected to check for it with errors.Is and retry once servers become available.

Source

Thrown at client/client.go:352

	// users is a pool of dynamic workload users
	users dynamic.Pool

	// identity is the node identity token that has been generated and signed by
	// the servers. This is used to authenticate the client to the servers when
	// performing RPC calls.
	identity atomic.Value

	// identityForceRenewal is used to force the client to renew its identity
	// at the next heartbeat. It is set by an operator calling the node identity
	// renew RPC method.
	identityForceRenewal atomic.Bool
}

var (
	// noServersErr is returned by the RPC method when the client has no
	// configured servers. This is used to trigger Consul discovery if
	// enabled.
	noServersErr = errors.New("no servers")
)

// NewClient is used to create a new client from the given configuration.
// `rpcs` is a map of RPC names to RPC structs that, if non-nil, will be
// registered via https://golang.org/pkg/net/rpc/#Server.RegisterName in place
// of the client's normal RPC handlers. This allows server tests to override
// the behavior of the client.
func NewClient(cfg *config.Config, consulCatalog consul.CatalogAPI, consulProxiesFunc consulApiShim.SupportedProxiesAPIFunc, consulServices serviceregistration.Handler, rpcs map[string]any) (*Client, error) {
	// Create the tls wrapper
	var tlsWrap tlsutil.RegionWrapper
	if cfg.TLSConfig.EnableRPC {
		tw, err := tlsutil.NewTLSConfiguration(cfg.TLSConfig, true, true)
		if err != nil {
			return nil, err
		}
		tlsWrap, err = tw.OutgoingTLSWrapper()
		if err != nil {
			return nil, err

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Configure servers explicitly via servers block or server_join.retry_join in the client config
  2. Verify Consul is reachable and nomad services are registered if auto-discovery is used
  3. Check network/firewall connectivity to servers on the RPC port (4647)
  4. Wait/retry: the client automatically retries registration and triggers discovery when it sees this error

Example fix

// before
servers = []
// after (HCL client config)
server_join {
  retry_join = ["10.0.0.10", "10.0.0.11"]
  retry_max = 0
}
Defensive patterns

Strategy: retry

Validate before calling

// before issuing RPCs, check the client has servers
if len(client.GetServers()) == 0 {
    // wait or trigger discovery; skip RPC call
    return fmt.Errorf("client has no servers yet")
}

Try / catch

res, err := client.RPC(method, args, reply)
if errors.Is(err, noServersErr) {
    client.TriggerDiscovery()
    // retry with backoff until servers are found
    time.Sleep(registerRetryIntv)
    return retryRPC(...)
}
if err != nil {
    return err
}

Prevention

When it happens

Trigger: Client.RPC() returns it when c.servers is empty (no server list from config, no Consul discovery results yet, or servers previously failed health checks and were removed). Also surfaces in registration flows where the client cannot find any region servers to register with.

Common situations: Nomad agent started in client mode with server list omitted and Consul auto-discovery disabled or Consul unreachable; all configured servers are down or unreachable; retry_join still resolving; clients started before servers come up in rolling deployments.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/75664beb2726f0d9. Report an issue: GitHub.