hashicorp/nomad · warning

contacted %d Nomad Servers: %v

Error message

contacted %d Nomad Servers: %v

What it means

In the fallback bootstrap path, the server collects discovered Nomad servers from Consul and calls s.Join() to have Raft contact them. If Join fails (or contacts 0 usable servers), this error reports how many servers were contacted along with the underlying error, and the poll timer is reset to retry. It signals the server still cannot find/join a quorum via Consul.

Source

Thrown at nomad/server.go:1108

		if len(nomadServerServices) == 0 {
			if len(mErr.Errors) > 0 {
				peersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))
				return mErr.ErrorOrNil()
			}

			// Log the error and return nil so future handlers
			// can attempt to register the `nomad` service.
			pollInterval := peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor)
			s.logger.Trace("no Nomad Servers advertising Nomad service in Consul datacenters", "service_name", nomadServerServiceName, "datacenters", dcs, "retry", pollInterval)
			peersTimeout.Reset(pollInterval)
			return nil
		}

		numServersContacted, err := s.Join(nomadServerServices)
		if err != nil {
			peersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))
			return fmt.Errorf("contacted %d Nomad Servers: %v", numServersContacted, err)
		}

		peersTimeout.Reset(maxStaleLeadership)
		s.logger.Info("successfully contacted Nomad servers", "num_servers", numServersContacted)

		return nil
	}

	// Hacky replacement for old ConsulSyncer Periodic Handler.
	go func() {
		lastOk := true
		sync := time.NewTimer(0)
		for {
			select {
			case <-sync.C:
				d := defaultConsulDiscoveryInterval
				if err := bootstrapFn(); err != nil {
					// Only log if it worked last time

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify connectivity to the contacted servers on the serf (4648) and RPC (4647) ports from this node.
2. Purge stale nomad service entries in Consul (dead members still registered).
3. Confirm gossip encryption key and TLS config match across servers.
4. Ensure at least a quorum of original servers is running, or use `nomad server force-leave` on dead members and restart.

Example fix

// before
# security group: only 4646 open
// after
# allow inbound tcp 4647 and both tcp/udp 4648 between nomad servers

// preflight: can this server reach serf/RPC ports on each discovered peer?
for _, peer := range discoveredServers {
    if err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", peer.Address, serfPort), 2*time.Second); err != nil {
        log.Printf("peer %s unreachable: %v", peer.Name, err)
    }
}

// Join is retried by the bootstrap poll; automation should wait and re-check
waitForQuorumPeers("/v1/status/peers", expectedQuorum, 10*time.Minute)

When it happens

Trigger: Quorum lost, Consul discovery returns nomad service entries, but s.Join fails — discovered servers are unreachable on the serf/RPC ports, TLS/gossip keys mismatch, firewall blocking 4647/4648, or all listed servers are themselves dead.

Common situations: Firewall/security-group rules blocking RPC between nodes; stale Consul entries pointing at decommissioned servers; raft_version/gossip encryption key mismatch after upgrade; network partition healing slowly.

Related errors

• resp.Warning
• error renewing the lease: %w
• failed to derive Consul token for task %s: %v
• failed to derive Consul token for service %s: %v
• unable to create unix socket for Consul HTTP endpoint: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/97a6012153321f04. Report an issue: GitHub.