hashicorp/nomad · error

failed querying server %q: %v

Error message

failed querying server %q: %v

What it means

serverWithNodeConn iterates servers asking each via Status.HasNodeConn whether it holds a multiplexed connection to the target node; failures querying an individual server are accumulated into rpcErr with this message including the server address. It signals a per-server RPC failure while searching for the best server to forward a client-targeted streaming/exec RPC.

Source

Thrown at nomad/client_rpc.go:184

		}
	}

	// connections is used to store the servers that have connections to the
	// requested node.
	var mostRecentServer *peers.Parts
	var mostRecent time.Time

	var rpcErr multierror.Error
	for _, server := range targets {
		if server.Addr.String() == selfAddr {
			continue
		}

		// Make the RPC
		var resp structs.NodeConnQueryResponse
		err := s.connPool.RPC(s.config.Region, server.Addr, "Status.HasNodeConn", &req, &resp)
		if err != nil {
			multierror.Append(&rpcErr, fmt.Errorf("failed querying server %q: %v", server.Addr.String(), err))
			continue
		}

		if resp.Connected && resp.Established.After(mostRecent) {
			mostRecentServer = server
			mostRecent = resp.Established
		}
	}

	// Return an error if there is no route to the node.
	if mostRecentServer == nil {
		if err := rpcErr.ErrorOrNil(); err != nil {
			return nil, err
		}

		return nil, structs.ErrNoNodeConn
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Check the wrapped %v for the actual cause (timeout, TLS handshake, connection refused).
  2. Verify all servers are healthy: `nomad server members` and `nomad server peers`.
  3. Confirm server-to-server connectivity on the RPC port (4647) and matching TLS CA/certificates.
  4. If one specific server is bad (shown in %q), restart or remove it from the Raft peers.

Example fix

// before
server 10.0.0.5:4647 unreachable (mTLS)
// after
nomad agent -server -ca-file=ca.pem -cert-file=server.pem -key-file=server-key.pem ...  # matching certs on all servers
Defensive patterns

Strategy: retry

Validate before calling

// health-check peers first
for _, m := range client.Agent().Members().ServerMembers {
    if m.Status != "alive" { log.Printf("server %s not alive", m.Name) }
}

Try / catch

if strings.Contains(err.Error(), "failed querying server") {
    // transient per-server failure; retry — iteration covers other peers
    time.Sleep(2 * time.Second)
    return findNodeConnAndForward(...) // retry
}

Prevention

When it happens

Trigger: Called from findClientConn / exec / streaming fs handlers / findNodeConnAndForward when the connPool.RPC to Status.HasNodeConn on some server fails: server down, TLS/mTLS misconfiguration, wrong region advertise address, network partition, or RPC timeout.

Common situations: A peer server crashed or is being replaced; TLS certificate mismatch between servers; firewall blocking RPC port (default 4647) between servers; servers in different regions/datacenters with broken WAN gossip.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/11f05e55db99b869. Report an issue: GitHub.