no Nomad Servers advertising service

Error message

no Nomad Servers advertising service %q in Consul datacenters: %+q

What it means

After querying all Consul datacenters for the `nomad` RPC service, `consulDiscoveryImpl` throws "no Nomad Servers advertising service %q in Consul datacenters: %+q" when zero servers were found. If individual DC queries errored, the aggregated multi-error is returned instead. This means Consul knows the datacenters but no healthy Nomad server instances are registered under the service name.

Solutions

Check that Nomad servers are running and registered in Consul (`consul catalog services` / health of the nomad service)
Ensure server and client consul service-name configs match
Fix failing Consul health checks (verify server RPC port 4647 is reachable from Consul)
Retry after servers re-register; discovery is retried on the client's retry loop

Example fix

// nomad server config
// before
consul { service {
  name = "nomad-server"  // clients look for "nomad"
} }
// after
consul { service {
  name = "nomad"
} }

Defensive patterns

Strategy: fallback

Validate before calling

entries, _, err := consul.Health().Service("nomad", "", true, nil)
if err != nil || len(entries) == 0 {
    log.Println("no healthy nomad servers registered in Consul")
}

Try / catch

err := client.consulDiscoveryImpl()
if err != nil {
    if strings.Contains(err.Error(), "no Nomad Servers advertising") {
        // fall back to static server list or wait for servers to re-register
        useStaticServerList()
    }
}

Prevention

Keep at least one healthy Nomad server registered with passing Consul health checks
Match server and client consul service name configuration
Ensure server RPC port (4647) is reachable so health checks pass
Configure a static server fallback in client config for resilience

When it happens

Trigger: All Nomad servers down or deregistered from Consul; servers registered under a different service name than the client's discovery config; health checks failing so Consul marks all instances unavailable.

Common situations: Server outage or rolling restart draining all registrations; mismatch between server `consul.service[0].name` and client discovery service name; Consul health checks failing due to firewall blocking the server RPC port (4647).

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/43c7a11fcda73fdf. Report an issue: GitHub.

Appendix: source

Thrown at client/client.go:3186

			if err != nil {
				mErr.Errors = append(mErr.Errors, err)
				continue
			}

			srv := &servers.Server{Addr: addr}
			nomadServers = append(nomadServers, srv)
		}

		if len(nomadServers) > 0 {
			break DISCOLOOP
		}

	}
	if len(nomadServers) == 0 {
		if len(mErr.Errors) > 0 {
			return mErr.ErrorOrNil()
		}
		return fmt.Errorf("no Nomad Servers advertising service %q in Consul datacenters: %+q", serviceName, dcs)
	}

	consulLogger.Info("discovered following servers", "servers", nomadServers)

	// Fire the retry trigger if we have updated the set of servers.
	if c.servers.SetServers(nomadServers) {
		// Start rebalancing
		c.servers.RebalanceServers()

		// Notify waiting rpc calls. If a goroutine just failed an RPC call and
		// isn't receiving on this chan yet they'll still retry eventually.
		// This is a shortcircuit for the longer retry intervals.
		c.fireRpcRetryWatcher()
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)