{"record":{"id":"f0d146340509ac79","repo":"vitessio/vitess","slug":"grpc-connection-wait-time-exceeded","errorCode":null,"errorMessage":"gRPC connection wait time exceeded","messagePattern":"gRPC connection wait time exceeded","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/grpcvtctldclient/client.go","lineNumber":38,"sourceCode":"\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\n\t\"google.golang.org/grpc\"\n\t\"google.golang.org/grpc/connectivity\"\n\n\t\"vitess.io/vitess/go/vt/grpcclient\"\n\t\"vitess.io/vitess/go/vt/vtctl/grpcclientcommon\"\n\t\"vitess.io/vitess/go/vt/vtctl/vtctldclient\"\n\n\tvtctlservicepb \"vitess.io/vitess/go/vt/proto/vtctlservice\"\n)\n\nvar (\n\tErrConnectionShutdown = errors.New(\"gRPCVtctldClient in a SHUTDOWN state\")\n\tErrConnectionTimeout  = errors.New(\"gRPC connection wait time exceeded\")\n)\n\nconst connClosedMsg = \"grpc: the client connection is closed\"\n\ntype gRPCVtctldClient struct {\n\tcc *grpc.ClientConn\n\tc  vtctlservicepb.VtctldClient\n}\n\n//go:generate -command grpcvtctldclient go run ../vtctldclient/codegen\n//go:generate grpcvtctldclient --out client_gen.go\n\nfunc gRPCVtctldClientFactory(ctx context.Context, addr string) (vtctldclient.VtctldClient, error) {\n\topt, err := grpcclientcommon.SecureDialOption()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldclient/client.go#L20-L56","documentation":"gRPCVtctldClient exposes ErrConnectionTimeout, returned by WaitForReady when the context deadline expires before the gRPC channel becomes ready. It signals that the vtctld server did not become reachable within the allotted wait time, as opposed to a client-side shutdown.","triggerScenarios":"Calling WaitForReady (or any RPC that waits for readiness) with a context whose deadline lapses before the gRPC connection to vtctld reaches Ready state.","commonSituations":"vtctld server down or unreachable (wrong host/port, firewall); slow network or overloaded server exceeding the caller's context timeout; DNS resolution delays in Kubernetes during startup races.","solutions":["Verify the vtctld server is running and the --vtctld address/port is correct","Increase the context deadline used around vtctldclient calls","Add retry with backoff around WaitForReady for transient startup races","Check network reachability (DNS, firewall, service endpoints) between client and vtctld"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)\nerr := client.WaitForReady(ctx) // ErrConnectionTimeout on slow startup\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\nfor {\n    if err := client.WaitForReady(ctx); err == nil || !errors.Is(err, grpcvtctldclient.ErrConnectionTimeout) {\n        break\n    }\n    time.Sleep(500 * time.Millisecond)\n}","handlingStrategy":"retry","validationCode":"// check reachability before the RPC\nconn, err := net.DialTimeout(\"tcp\", vtctldAddr, 2*time.Second)\nif err != nil {\n    return fmt.Errorf(\"vtctld unreachable at %s: %w\", vtctldAddr, err)\n}\nconn.Close()","typeGuard":"func isConnectionTimeoutErr(err error) bool {\n    return errors.Is(err, grpcvtctldclient.ErrConnectionTimeout)\n}","tryCatchPattern":"ctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\nif err := client.WaitForReady(ctx); err != nil {\n    if errors.Is(err, grpcvtctldclient.ErrConnectionTimeout) {\n        return retry.WithBackoff(ctx, func() error {\n            return client.WaitForReady(ctx)\n        })\n    }\n    return err\n}","preventionTips":["Use generous context deadlines (30s+) for vtctld startup scenarios","Retry with backoff on ErrConnectionTimeout during service startup races","Verify vtctld host/port and network reachability before calls","Differentiate timeout (ErrConnectionTimeout) from shutdown (ErrConnectionShutdown) when handling"],"tags":["grpc","timeout","vtctldclient","network"],"backgroundTag":"connection-timeout","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}