{"record":{"id":"2d4e0ba1646d6d22","repo":"grpc/grpc-go","slug":"all-subconns-are-in-transientfailure-last-connect","errorCode":null,"errorMessage":"all SubConns are in TransientFailure, last connection error: %v","messagePattern":"all SubConns are in TransientFailure, last connection error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"balancer/grpclb/grpclb.go","lineNumber":249,"sourceCode":"\tremoteBalancerConnected bool\n\tserverListReceived      bool\n\tinFallback              bool\n\t// resolvedBackendAddrs is resolvedAddrs minus remote balancers. It's set\n\t// when resolved address updates are received, and read in the goroutine\n\t// handling fallback.\n\tresolvedBackendAddrs []resolver.Address\n\tconnErr              error // the last connection error\n}\n\n// regeneratePicker takes a snapshot of the balancer, and generates a picker from\n// it. The picker\n//   - always returns ErrTransientFailure if the balancer is in TransientFailure,\n//   - does two layer roundrobin pick otherwise.\n//\n// Caller must hold lb.mu.\nfunc (lb *lbBalancer) regeneratePicker(resetDrop bool) {\n\tif lb.state == connectivity.TransientFailure {\n\t\tlb.picker = base.NewErrPicker(fmt.Errorf(\"all SubConns are in TransientFailure, last connection error: %v\", lb.connErr))\n\t\treturn\n\t}\n\n\tif lb.state == connectivity.Connecting {\n\t\tlb.picker = base.NewErrPicker(balancer.ErrNoSubConnAvailable)\n\t\treturn\n\t}\n\n\tvar readySCs []balancer.SubConn\n\tif lb.usePickFirst {\n\t\tfor _, sc := range lb.subConns {\n\t\t\treadySCs = append(readySCs, sc)\n\t\t\tbreak\n\t\t}\n\t} else {\n\t\tfor _, a := range lb.backendAddrsWithoutMetadata {\n\t\t\tif sc, ok := lb.subConns[a]; ok {\n\t\t\t\tif st, ok := lb.scStates[sc]; ok && st == connectivity.Ready {","sourceCodeStart":231,"sourceCodeEnd":267,"githubUrl":"https://github.com/grpc/grpc-go/blob/0c51461d27177d997e14c642fe18c11668fc09a3/balancer/grpclb/grpclb.go#L231-L267","documentation":"Produced by grpclb's regeneratePicker() (balancer/grpclb/grpclb.go:249). When aggregateSubConnStats() returns TransientFailure (no SubConn is Ready, Connecting, or Idle), the picker is set to an errPicker wrapping this message with lb.connErr. This is specific to the deprecated grpclb policy: every backend SubConn — whether from the remote load balancer server list or the fallback list — has failed.","triggerScenarios":"grpclb balancer's aggregateSubConnStats() returns TransientFailure because every entry in lb.subConns is in TransientFailure, with none Ready/Connecting/Idle. regeneratePicker() (line 248-250) builds the error picker. Happens after all backend connections fail and none are mid-reconnect.","commonSituations":"Remote load balancer (grpclb server) returned a server list whose backends are all unreachable; fallback backends (resolved directly) are also down; TLS to backends is misconfigured; the cluster is fully offline. grpclb itself is deprecated in favor of xDS, so this may also indicate an outdated deployment.","solutions":["Check the embedded connection error (lb.connErr) for the specific transport failure.","Verify the backend addresses delivered by the grpclb server (or fallback resolver) are reachable.","Migrate off the deprecated grpclb policy to xDS-based load balancing (ring_hash / weighted_target) where possible.","Ensure at least one backend in the server list is healthy and listening."],"exampleFix":"// before: using deprecated grpclb with all backends down\nimport _ \"google.golang.org/grpc/balancer/grpclb\"\nconn, _ := grpc.NewClient(\"dns:///my-svc:50051\", grpc.WithDefaultServiceConfig(`{\"loadBalancingConfig\":[{\"grpclb\":{}}]}`))\n\n// after: use round_robin (no remote LB needed) and fix backend reachability\nconn, _ := grpc.NewClient(\"dns:///my-svc:50051\", grpc.WithDefaultServiceConfig(`{\"loadBalancingConfig\":[{\"round_robin\":{}}]}`))","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// grpclb all-SubConn failure → Unavailable. Retry with backoff.\nfor {\n    resp, err := c.Call(ctx, in, grpc.WaitForReady(true))\n    if err == nil { return resp, nil }\n    if status.Code(err) != codes.Unavailable { return nil, err }\n    select {\n    case <-time.After(backoff):\n    case <-ctx.Done(): return nil, ctx.Err()\n    }\n}","preventionTips":["Migrate off the deprecated grpclb policy to xDS-based load balancing.","Ensure the grpclb remote balancer returns at least one healthy backend.","Monitor the grpclb server list for staleness and backend reachability."],"tags":["grpc","balancer","grpclb","connection","transient-failure","network","deprecated"],"backgroundTag":null,"analyzedSha":"0c51461d27177d997e14c642fe18c11668fc09a3","analyzedAt":"2026-08-11T14:49:15.055Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}