What causes "rpc error: shutdown" in hashicorp/nomad?

Treat as expected during shutdown; stop issuing RPCs once Shutdown was called

rpc error: shutdown

Error message

rpc error: shutdown

What it means

ConnPool.acquire waits on either the pool's shutdown channel or the lead thread's connection attempt. If the connection pool is shut down first, it returns 'rpc error: shutdown'. This happens when the agent/node is shutting down while an RPC client request is still being made.

Source

Thrown at helper/pool/pool.go:371

		p.pool[addr.String()] = c

		// If there is a connection listener, notify them of the new connection.
		if p.connListener != nil {
			select {
			case p.connListener <- c:
			default:
			}
		}

		p.Unlock()
		return c, nil
	}

	// Otherwise, wait for the lead thread to attempt the connection
	// and use what's in the pool at that point.
	select {
	case <-p.shutdownCh:
		return nil, fmt.Errorf("rpc error: shutdown")
	case <-wait:
	}

	// See if the lead thread was able to get us a connection.
	p.Lock()
	if c := p.pool[addr.String()]; c != nil {
		c.markForUse()
		p.Unlock()
		return c, nil
	}

	p.Unlock()
	return nil, fmt.Errorf("rpc error: lead thread didn't get connection")
}

// getNewConn is used to return a new connection
func (p *ConnPool) getNewConn(region string, addr net.Addr) (*Conn, error) {
	// Try to dial the conn

View on GitHub (pinned to 482b49bf1a)

Solutions

Treat as expected during shutdown; stop issuing RPCs once Shutdown was called
Retain/retry the RPC on a different pool/agent if failover is required
Order application teardown so RPC users stop before the pool shuts down

Example fix

// before
err := pool.RPC(region, addr, method, args, out) // ignores shutdown
// after
select {
case <-shutdownCh:
    return // stop issuing RPCs before pool shutdown
default:
}
err := pool.RPC(region, addr, method, args, out)

Defensive patterns

Strategy: try-catch

Validate before calling

// check pool shutdown state before issuing RPCs
select {
case <-poolShutdownCh:
    return fmt.Errorf("pool shut down; skip RPC")
default:
}

Try / catch

err := pool.RPC(region, addr, method, args, out)
if err != nil && strings.Contains(err.Error(), "rpc error: shutdown") {
    return err // expected during shutdown; do not retry on this pool
}

Prevention

Stop RPC users before calling pool.Shutdown() in teardown
Use a context/cancellation that follows agent shutdown
In tests, close pools after all RPC users finish

When it happens

Trigger: Calling pool.RPC or StreamingRPC (via getRPCClient -> acquire) after p.Shutdown() was invoked, i.e. during agent shutdown or pool teardown.

Common situations: Client agent shutting down while a driver or job still issues RPCs to the server; test teardown ordering; node shutdown racing in-flight requests.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/934e333980799ec3. Report an issue: GitHub.