{"record":{"id":"934e333980799ec3","repo":"hashicorp/nomad","slug":"rpc-error-shutdown","errorCode":null,"errorMessage":"rpc error: shutdown","messagePattern":"rpc error: shutdown","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/pool/pool.go","lineNumber":371,"sourceCode":"\t\tp.pool[addr.String()] = c\n\n\t\t// If there is a connection listener, notify them of the new connection.\n\t\tif p.connListener != nil {\n\t\t\tselect {\n\t\t\tcase p.connListener <- c:\n\t\t\tdefault:\n\t\t\t}\n\t\t}\n\n\t\tp.Unlock()\n\t\treturn c, nil\n\t}\n\n\t// Otherwise, wait for the lead thread to attempt the connection\n\t// and use what's in the pool at that point.\n\tselect {\n\tcase <-p.shutdownCh:\n\t\treturn nil, fmt.Errorf(\"rpc error: shutdown\")\n\tcase <-wait:\n\t}\n\n\t// See if the lead thread was able to get us a connection.\n\tp.Lock()\n\tif c := p.pool[addr.String()]; c != nil {\n\t\tc.markForUse()\n\t\tp.Unlock()\n\t\treturn c, nil\n\t}\n\n\tp.Unlock()\n\treturn nil, fmt.Errorf(\"rpc error: lead thread didn't get connection\")\n}\n\n// getNewConn is used to return a new connection\nfunc (p *ConnPool) getNewConn(region string, addr net.Addr) (*Conn, error) {\n\t// Try to dial the conn","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/pool/pool.go#L353-L389","documentation":"ConnPool.acquire waits on either the pool's shutdown channel or the lead thread's connection attempt. If the connection pool is shut down first, it returns 'rpc error: shutdown'. This happens when the agent/node is shutting down while an RPC client request is still being made.","triggerScenarios":"Calling pool.RPC or StreamingRPC (via getRPCClient -> acquire) after p.Shutdown() was invoked, i.e. during agent shutdown or pool teardown.","commonSituations":"Client agent shutting down while a driver or job still issues RPCs to the server; test teardown ordering; node shutdown racing in-flight requests.","solutions":["Treat as expected during shutdown; stop issuing RPCs once Shutdown was called","Retain/retry the RPC on a different pool/agent if failover is required","Order application teardown so RPC users stop before the pool shuts down"],"exampleFix":"// before\nerr := pool.RPC(region, addr, method, args, out) // ignores shutdown\n// after\nselect {\ncase <-shutdownCh:\n    return // stop issuing RPCs before pool shutdown\ndefault:\n}\nerr := pool.RPC(region, addr, method, args, out)","handlingStrategy":"try-catch","validationCode":"// check pool shutdown state before issuing RPCs\nselect {\ncase <-poolShutdownCh:\n    return fmt.Errorf(\"pool shut down; skip RPC\")\ndefault:\n}","typeGuard":null,"tryCatchPattern":"err := pool.RPC(region, addr, method, args, out)\nif err != nil && strings.Contains(err.Error(), \"rpc error: shutdown\") {\n    return err // expected during shutdown; do not retry on this pool\n}","preventionTips":["Stop RPC users before calling pool.Shutdown() in teardown","Use a context/cancellation that follows agent shutdown","In tests, close pools after all RPC users finish"],"tags":["rpc","connection-pool","shutdown"],"backgroundTag":"rpc-pool-shutdown","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}