hashicorp/nomad · error

rpc error: lead thread didn't get connection

Error message

rpc error: lead thread didn't get connection

What it means

In ConnPool.acquire, after waiting for the lead thread, the pool lock is released and no connection was found in the pool for the address — the lead thread failed to establish/store one without waking waiters with it. The loader returns 'rpc error: lead thread didn't get connection'.

Source

Thrown at helper/pool/pool.go:384

	// Otherwise, wait for the lead thread to attempt the connection
	// and use what's in the pool at that point.
	select {
	case <-p.shutdownCh:
		return nil, fmt.Errorf("rpc error: shutdown")
	case <-wait:
	}

	// See if the lead thread was able to get us a connection.
	p.Lock()
	if c := p.pool[addr.String()]; c != nil {
		c.markForUse()
		p.Unlock()
		return c, nil
	}

	p.Unlock()
	return nil, fmt.Errorf("rpc error: lead thread didn't get connection")
}

// getNewConn is used to return a new connection
func (p *ConnPool) getNewConn(region string, addr net.Addr) (*Conn, error) {
	// Try to dial the conn
	conn, err := net.DialTimeout("tcp", addr.String(), p.dialTimeout)
	if err != nil {
		return nil, err
	}

	// Cast to TCPConn
	if tcp, ok := conn.(*net.TCPConn); ok {
		tcp.SetKeepAlive(true)
		tcp.SetNoDelay(true)
	}

	// Check if TLS is enabled
	if p.tlsWrap != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check connectivity to the target addr/port (is the Nomad server up?)
2. Retry the RPC; the next attempt may establish a fresh connection
3. Inspect server logs and firewall rules if the dial keeps failing

err := pool.RPC(region, addr, method, args, out)
if err != nil && strings.Contains(err.Error(), "lead thread didn't get connection") {
    // brief backoff then retry once; next dial may succeed
    time.Sleep(100 * time.Millisecond)
    return pool.RPC(region, addr, method, args, out)
}

When it happens

Trigger: getRPCClient/StreamingRPC -> acquire on an addr where the lead thread's dial failed and the pool entry remained nil at wake time.

Common situations: Server unreachable (crashed, network partition) so the lead dial fails; race during connection teardown between lead thread and waiters.

Related errors

• rpc error: shutdown
• no servers
• ack.Error
• rcp.accept_backlog interval must be greater than zero
• rcp.keep_alive_interval must be greater than zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d25cce559774153a. Report an issue: GitHub.