hashicorp/nomad · error

failed to get conn: %v

Error message

failed to get conn: %v

What it means

getRPCClient first acquires a connection from the pool; any error from acquire (shutdown, lead thread failure, dial errors) is wrapped as 'failed to get conn: <err>'. This is the outer wrapper for all pool-acquisition failures on the normal RPC path.

Source

Thrown at helper/pool/pool.go:472

	if c, ok := p.pool[conn.addr.String()]; ok && c == conn {
		delete(p.pool, conn.addr.String())
	}
	p.Unlock()

	// Close down immediately if idle
	if refCount := atomic.LoadInt32(&conn.refCount); refCount == 0 {
		conn.Close()
	}
}

// getClient is used to get a usable client for an address
func (p *ConnPool) getRPCClient(region string, addr net.Addr) (*Conn, *StreamClient, error) {
	retries := 0
START:
	// Try to get a conn first
	conn, err := p.acquire(region, addr)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to get conn: %v", err)
	}

	// Get a client
	client, err := conn.getRPCClient()
	if err != nil {
		p.clearConn(conn)
		conn.releaseUse()

		// Try to redial, possible that the TCP session closed due to timeout
		if retries == 0 {
			retries++
			goto START
		}
		return nil, nil, fmt.Errorf("failed to start stream: %v", err)
	}
	return conn, client, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the inner error: dial refused/timeout means fix connectivity or the server address
2. Verify the Nomad server is running and its advertise RPC address is correct
3. Check TLS/CA configuration if connections are rejected
4. Retry the RPC; the pool transparently redials on transient failures

Example fix

// before: hardcoding a stale server address
addr := "/ip/10.0.0.1/port/4647" // server moved
// after: resolve from client config
addr := nodeAddrFromServers(srv)

// precheck connectivity before RPC
c, err := net.DialTimeout("tcp", addr.String(), 2*time.Second)
if err != nil { return fmt.Errorf("server unreachable: %w", err) }
c.Close()

err := pool.RPC(region, addr, method, args, out)
if err != nil {
    var ne net.Error
    if errors.As(err, &ne) || strings.Contains(err.Error(), "failed to get conn") {
        // backoff + retry; also try another server from the server list
    }
    return err
}

When it happens

Trigger: pool.RPC -> getRPCClient where p.acquire fails: pool shut down, TCP dial timeout/refused, or lead-thread race error.

Common situations: Server down or restarting, wrong advertise address in config, network partition, TLS/mTLS mismatch preventing dial, agent shutdown in flight.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• ack.Error
• No path to node
• Server at address %s failed ping: %v
• Region %q: %v
• rpc error: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ca1f3955bfcccfe2. Report an issue: GitHub.