hashicorp/nomad · error

Server at address %s failed ping: %v

Error message

Server at address %s failed ping: %v

What it means

While refreshing its server list, the client pings each candidate server address to verify it is a real, reachable server. If c.Ping(addr) fails for a candidate, the address is recorded as a multi-error entry with this message. Depending on the force flag, the candidate may be discarded or still injected.

Source

Thrown at client/client.go:1313

	endpoints := make([]*servers.Server, 0, len(in))
	wg.Add(len(in))

	for _, s := range in {
		go func(srv string) {
			defer wg.Done()
			addr, err := resolveServer(srv)
			if err != nil {
				mu.Lock()
				c.logger.Debug("ignoring server due to resolution error", "error", err, "server", srv)
				merr.Errors = append(merr.Errors, err)
				mu.Unlock()
				return
			}

			// Try to ping to check if it is a real server
			if err := c.Ping(addr); err != nil {
				mu.Lock()
				merr.Errors = append(merr.Errors, fmt.Errorf("Server at address %s failed ping: %v", addr, err))
				mu.Unlock()

				// If we are forcing the setting of the servers, inject it to
				// the serverlist even if we can't ping immediately.
				if !force {
					return
				}
			}

			mu.Lock()
			endpoints = append(endpoints, &servers.Server{Addr: addr})
			mu.Unlock()
		}(s)
	}

	wg.Wait()

	// Only return errors if no servers are valid

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped %v ping error to distinguish connection-refused/timeout from TLS errors and fix the underlying cause.
2. Verify the Nomad servers' RPC port (default 4647) is reachable from the client: nc -zv <server> 4647.
3. Confirm DNS/addresses in the servers list resolve to live Nomad server nodes; remove stale entries.
4. If TLS is enabled, ensure the client's CA cert and server certs are valid and match on both sides.

Example fix

// before (client.hcl with unreachable server)
client { servers = ["10.0.0.5:4647"] }  // server decommissioned
// after
client { servers = ["10.0.0.10:4647", "10.0.0.11:4647"] }  # live servers
# plus: open firewall
# sudo ufw allow 4647/tcp

for _, addr := range servers {
    conn, err := net.DialTimeout("tcp", addr, 2*time.Second)
    if err != nil {
        log.Printf("server %s unreachable before join: %v", addr, err)
    } else {
        conn.Close()
    }
}

if err := client.Ping(addr); err != nil {
    merr.Errors = append(merr.Errors, err)
    // retry with backoff, then try next server in the list
    if !retryable(err) { return merr }
    time.Sleep(backoff)
    continue
}

When it happens

Trigger: Client heartbeat/server-list update loop: c.Ping(addr) returns an error (connection refused, timeout, TLS mismatch) for a server address in the returned server list.

Common situations: Server down or restarted, firewall/security group blocking the RPC port (4647), stale DNS entries pointing at decommissioned servers, network partition, or clients and servers with mismatched TLS/CA configuration.

Related errors

• ack.Error
• No path to node
• Region %q: %v
• failed to get conn: %v
• rpc error: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e7ac5f71b6cc66d1. Report an issue: GitHub.