hashicorp/nomad · critical

no Nomad Servers advertising service %q in Consul datacenter

Error message

no Nomad Servers advertising service %q in Consul datacenters: %+q

What it means

After querying all Consul datacenters for the `nomad` RPC service, `consulDiscoveryImpl` throws "no Nomad Servers advertising service %q in Consul datacenters: %+q" when zero servers were found. If individual DC queries errored, the aggregated multi-error is returned instead. This means Consul knows the datacenters but no healthy Nomad server instances are registered under the service name.

Source

Thrown at client/client.go:3186

			if err != nil {
				mErr.Errors = append(mErr.Errors, err)
				continue
			}

			srv := &servers.Server{Addr: addr}
			nomadServers = append(nomadServers, srv)
		}

		if len(nomadServers) > 0 {
			break DISCOLOOP
		}

	}
	if len(nomadServers) == 0 {
		if len(mErr.Errors) > 0 {
			return mErr.ErrorOrNil()
		}
		return fmt.Errorf("no Nomad Servers advertising service %q in Consul datacenters: %+q", serviceName, dcs)
	}

	consulLogger.Info("discovered following servers", "servers", nomadServers)

	// Fire the retry trigger if we have updated the set of servers.
	if c.servers.SetServers(nomadServers) {
		// Start rebalancing
		c.servers.RebalanceServers()

		// Notify waiting rpc calls. If a goroutine just failed an RPC call and
		// isn't receiving on this chan yet they'll still retry eventually.
		// This is a shortcircuit for the longer retry intervals.
		c.fireRpcRetryWatcher()
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check that Nomad servers are running and registered in Consul (`consul catalog services` / health of the nomad service)
2. Ensure server and client consul service-name configs match
3. Fix failing Consul health checks (verify server RPC port 4647 is reachable from Consul)
4. Retry after servers re-register; discovery is retried on the client's retry loop

Example fix

// nomad server config
// before
consul { service {
  name = "nomad-server"  // clients look for "nomad"
} }
// after
consul { service {
  name = "nomad"
} }

entries, _, err := consul.Health().Service("nomad", "", true, nil)
if err != nil || len(entries) == 0 {
    log.Println("no healthy nomad servers registered in Consul")
}

err := client.consulDiscoveryImpl()
if err != nil {
    if strings.Contains(err.Error(), "no Nomad Servers advertising") {
        // fall back to static server list or wait for servers to re-register
        useStaticServerList()
    }
}

When it happens

Trigger: All Nomad servers down or deregistered from Consul; servers registered under a different service name than the client's discovery config; health checks failing so Consul marks all instances unavailable.

Common situations: Server outage or rolling restart draining all registrations; mismatch between server `consul.service[0].name` and client discovery service name; Consul health checks failing due to firewall blocking the server RPC port (4647).

Related errors

• client.consul: unable to query Consul datacenters: %v
• unable to query service %+q from Consul datacenter %+q: %v
• failed to retrieve services from consul: %w
• error creating bootstrap configuration for Connect proxy sid
• client stopped and may not longer create config entries

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/43c7a11fcda73fdf. Report an issue: GitHub.