{"record":{"id":"9901df904eec8d37","repo":"hashicorp/nomad","slug":"timed-out-waiting-for-socket-proxy-to-exit","errorCode":null,"errorMessage":"timed out waiting for socket proxy to exit","messagePattern":"timed out waiting for socket proxy to exit","errorType":"exception","errorClass":"errSocketProxyTimeout","httpStatus":null,"severity":"error","filePath":"client/allocrunner/consul_grpc_sock_hook.go","lineNumber":43,"sourceCode":"\t\"github.com/hashicorp/nomad/nomad/structs\"\n\t\"github.com/hashicorp/nomad/nomad/structs/config\"\n)\n\nconst (\n\tconsulGRPCSockHookName = \"consul_grpc_socket\"\n\n\t// socketProxyStopWaitTime is the amount of time to wait for a socket proxy\n\t// to stop before assuming something went awry and return a timeout error.\n\tsocketProxyStopWaitTime = 3 * time.Second\n\n\t// consulGRPCFallbackPort is the last resort fallback port to use in\n\t// combination with the Consul HTTP config address when creating the\n\t// socket.\n\tconsulGRPCFallbackPort = \"8502\"\n)\n\nvar (\n\terrSocketProxyTimeout = errors.New(\"timed out waiting for socket proxy to exit\")\n)\n\n// consulGRPCSocketHook creates Unix sockets to allow communication from inside a\n// netns to Consul gRPC endpoint.\n//\n// Noop for allocations without a group Connect block using bridge networking.\ntype consulGRPCSocketHook struct {\n\tlogger hclog.Logger\n\n\t// mu synchronizes proxy and alloc which may be mutated and read concurrently\n\t// via Prerun, Update, Postrun.\n\tmu      sync.Mutex\n\talloc   *structs.Allocation\n\tproxies map[string]*grpcSocketProxy\n}\n\nfunc newConsulGRPCSocketHook(\n\tlogger hclog.Logger,","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/consul_grpc_sock_hook.go#L25-L61","documentation":"consul_grpc_sock_hook.go declares errSocketProxyTimeout to signal that a socket proxy goroutine (proxying Consul gRPC traffic into the allocation's netns) did not exit within socketProxyStopWaitTime after a stop was requested. stop() selects on the proxy's done channel versus a timeout; hitting the timeout means the proxy is wedged or still serving connections. The same sentinel is reused by the sibling consul_http_sock_hook.go.","triggerScenarios":"Stopping a Connect-enabled allocation (bridge networking) whose Consul gRPC (or HTTP) socket proxy does not shut down before socketProxyStopWaitTime elapses — e.g. leaked open connections, blocked proxy loop, or task cleanup during node shutdown.","commonSituations":"Allocation stops/garbage collection hanging on Connect services; Consul agent unresponsive causing proxy connections to linger; node drain/shutdown delays with Connect workloads.","solutions":["Restart the Nomad client agent to clean up the stuck socket proxy goroutine","Check Consul agent health/connectivity on the host (a blocked upstream keeps connections open)","Upgrade Nomad — later versions tuned proxy shutdown and connection draining","Investigate the workload for long-lived connections to the Unix socket that prevent proxy exit"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"func isSocketProxyTimeout(err error) bool {\n  return errors.Is(err, errSocketProxyTimeout)\n}","tryCatchPattern":"if err := hook.stop(); err != nil {\n  if errors.Is(err, errSocketProxyTimeout) {\n    // proxy did not stop in time; log and continue cleanup, force-close socket\n    logger.Warn(\"socket proxy did not exit in time\", \"err\", err)\n  } else {\n    return err\n  }\n}","preventionTips":["Keep the Consul agent healthy so proxy connections can close promptly","Avoid long-lived idle connections through the netns socket during shutdown","Keep Nomad client versions current; shutdown handling for socket proxies has been improved","Monitor node drain/shutdown timing when running many Connect workloads"],"tags":["consul","connect","service-mesh","shutdown-timeout"],"backgroundTag":"shutdown-timeout","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}