hashicorp/nomad · error · errSocketProxyTimeout

timed out waiting for socket proxy to exit

Error message

timed out waiting for socket proxy to exit

What it means

consul_grpc_sock_hook.go declares errSocketProxyTimeout to signal that a socket proxy goroutine (proxying Consul gRPC traffic into the allocation's netns) did not exit within socketProxyStopWaitTime after a stop was requested. stop() selects on the proxy's done channel versus a timeout; hitting the timeout means the proxy is wedged or still serving connections. The same sentinel is reused by the sibling consul_http_sock_hook.go.

Source

Thrown at client/allocrunner/consul_grpc_sock_hook.go:43

	"github.com/hashicorp/nomad/nomad/structs"
	"github.com/hashicorp/nomad/nomad/structs/config"
)

const (
	consulGRPCSockHookName = "consul_grpc_socket"

	// socketProxyStopWaitTime is the amount of time to wait for a socket proxy
	// to stop before assuming something went awry and return a timeout error.
	socketProxyStopWaitTime = 3 * time.Second

	// consulGRPCFallbackPort is the last resort fallback port to use in
	// combination with the Consul HTTP config address when creating the
	// socket.
	consulGRPCFallbackPort = "8502"
)

var (
	errSocketProxyTimeout = errors.New("timed out waiting for socket proxy to exit")
)

// consulGRPCSocketHook creates Unix sockets to allow communication from inside a
// netns to Consul gRPC endpoint.
//
// Noop for allocations without a group Connect block using bridge networking.
type consulGRPCSocketHook struct {
	logger hclog.Logger

	// mu synchronizes proxy and alloc which may be mutated and read concurrently
	// via Prerun, Update, Postrun.
	mu      sync.Mutex
	alloc   *structs.Allocation
	proxies map[string]*grpcSocketProxy
}

func newConsulGRPCSocketHook(
	logger hclog.Logger,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the Nomad client agent to clean up the stuck socket proxy goroutine
2. Check Consul agent health/connectivity on the host (a blocked upstream keeps connections open)
3. Upgrade Nomad — later versions tuned proxy shutdown and connection draining
4. Investigate the workload for long-lived connections to the Unix socket that prevent proxy exit

func isSocketProxyTimeout(err error) bool {
  return errors.Is(err, errSocketProxyTimeout)
}

if err := hook.stop(); err != nil {
  if errors.Is(err, errSocketProxyTimeout) {
    // proxy did not stop in time; log and continue cleanup, force-close socket
    logger.Warn("socket proxy did not exit in time", "err", err)
  } else {
    return err
  }
}

When it happens

Trigger: Stopping a Connect-enabled allocation (bridge networking) whose Consul gRPC (or HTTP) socket proxy does not shut down before socketProxyStopWaitTime elapses — e.g. leaked open connections, blocked proxy loop, or task cleanup during node shutdown.

Common situations: Allocation stops/garbage collection hanging on Connect services; Consul agent unresponsive causing proxy connections to linger; node drain/shutdown delays with Connect workloads.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• consul address must be set on nomad client
• Service with provider nomad cannot include Connect blocks
• Connect proxy task must not have a service block
• error creating bootstrap configuration for Connect proxy sid
• cannot update alloc to Connect in-place

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9901df904eec8d37. Report an issue: GitHub.