{"record":{"id":"d25cce559774153a","repo":"hashicorp/nomad","slug":"rpc-error-lead-thread-didn-t-get-connection","errorCode":null,"errorMessage":"rpc error: lead thread didn't get connection","messagePattern":"rpc error: lead thread didn't get connection","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/pool/pool.go","lineNumber":384,"sourceCode":"\n\t// Otherwise, wait for the lead thread to attempt the connection\n\t// and use what's in the pool at that point.\n\tselect {\n\tcase <-p.shutdownCh:\n\t\treturn nil, fmt.Errorf(\"rpc error: shutdown\")\n\tcase <-wait:\n\t}\n\n\t// See if the lead thread was able to get us a connection.\n\tp.Lock()\n\tif c := p.pool[addr.String()]; c != nil {\n\t\tc.markForUse()\n\t\tp.Unlock()\n\t\treturn c, nil\n\t}\n\n\tp.Unlock()\n\treturn nil, fmt.Errorf(\"rpc error: lead thread didn't get connection\")\n}\n\n// getNewConn is used to return a new connection\nfunc (p *ConnPool) getNewConn(region string, addr net.Addr) (*Conn, error) {\n\t// Try to dial the conn\n\tconn, err := net.DialTimeout(\"tcp\", addr.String(), p.dialTimeout)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Cast to TCPConn\n\tif tcp, ok := conn.(*net.TCPConn); ok {\n\t\ttcp.SetKeepAlive(true)\n\t\ttcp.SetNoDelay(true)\n\t}\n\n\t// Check if TLS is enabled\n\tif p.tlsWrap != nil {","sourceCodeStart":366,"sourceCodeEnd":402,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/pool/pool.go#L366-L402","documentation":"In ConnPool.acquire, after waiting for the lead thread, the pool lock is released and no connection was found in the pool for the address — the lead thread failed to establish/store one without waking waiters with it. The loader returns 'rpc error: lead thread didn't get connection'.","triggerScenarios":"getRPCClient/StreamingRPC -> acquire on an addr where the lead thread's dial failed and the pool entry remained nil at wake time.","commonSituations":"Server unreachable (crashed, network partition) so the lead dial fails; race during connection teardown between lead thread and waiters.","solutions":["Check connectivity to the target addr/port (is the Nomad server up?)","Retry the RPC; the next attempt may establish a fresh connection","Inspect server logs and firewall rules if the dial keeps failing"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := pool.RPC(region, addr, method, args, out)\nif err != nil && strings.Contains(err.Error(), \"lead thread didn't get connection\") {\n    // brief backoff then retry once; next dial may succeed\n    time.Sleep(100 * time.Millisecond)\n    return pool.RPC(region, addr, method, args, out)\n}","preventionTips":["Keep servers reachable; monitor the RPC port (default 4647)","Retry transient pool errors at the application level","Investigate if it occurs persistently (dial failures racing)"],"tags":["rpc","connection-pool","race"],"backgroundTag":"rpc-connection-pool-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}