vitessio/vitess · error

gRPC connection wait time exceeded

Error message

gRPC connection wait time exceeded

What it means

gRPCVtctldClient exposes ErrConnectionTimeout, returned by WaitForReady when the context deadline expires before the gRPC channel becomes ready. It signals that the vtctld server did not become reachable within the allotted wait time, as opposed to a client-side shutdown.

Source

Thrown at go/vt/vtctl/grpcvtctldclient/client.go:38

import (
	"context"
	"errors"
	"fmt"

	"google.golang.org/grpc"
	"google.golang.org/grpc/connectivity"

	"vitess.io/vitess/go/vt/grpcclient"
	"vitess.io/vitess/go/vt/vtctl/grpcclientcommon"
	"vitess.io/vitess/go/vt/vtctl/vtctldclient"

	vtctlservicepb "vitess.io/vitess/go/vt/proto/vtctlservice"
)

var (
	ErrConnectionShutdown = errors.New("gRPCVtctldClient in a SHUTDOWN state")
	ErrConnectionTimeout  = errors.New("gRPC connection wait time exceeded")
)

const connClosedMsg = "grpc: the client connection is closed"

type gRPCVtctldClient struct {
	cc *grpc.ClientConn
	c  vtctlservicepb.VtctldClient
}

//go:generate -command grpcvtctldclient go run ../vtctldclient/codegen
//go:generate grpcvtctldclient --out client_gen.go

func gRPCVtctldClientFactory(ctx context.Context, addr string) (vtctldclient.VtctldClient, error) {
	opt, err := grpcclientcommon.SecureDialOption()
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Verify the vtctld server is running and the --vtctld address/port is correct
2. Increase the context deadline used around vtctldclient calls
3. Add retry with backoff around WaitForReady for transient startup races
4. Check network reachability (DNS, firewall, service endpoints) between client and vtctld

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
err := client.WaitForReady(ctx) // ErrConnectionTimeout on slow startup
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
for {
    if err := client.WaitForReady(ctx); err == nil || !errors.Is(err, grpcvtctldclient.ErrConnectionTimeout) {
        break
    }
    time.Sleep(500 * time.Millisecond)
}

// check reachability before the RPC
conn, err := net.DialTimeout("tcp", vtctldAddr, 2*time.Second)
if err != nil {
    return fmt.Errorf("vtctld unreachable at %s: %w", vtctldAddr, err)
}
conn.Close()

func isConnectionTimeoutErr(err error) bool {
    return errors.Is(err, grpcvtctldclient.ErrConnectionTimeout)
}

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if err := client.WaitForReady(ctx); err != nil {
    if errors.Is(err, grpcvtctldclient.ErrConnectionTimeout) {
        return retry.WithBackoff(ctx, func() error {
            return client.WaitForReady(ctx)
        })
    }
    return err
}

When it happens

Trigger: Calling WaitForReady (or any RPC that waits for readiness) with a context whose deadline lapses before the gRPC connection to vtctld reaches Ready state.

Common situations: vtctld server down or unreachable (wrong host/port, firewall); slow network or overloaded server exceeding the caller's context timeout; DNS resolution delays in Kubernetes during startup races.

Related errors

• failed to discover %ss (cluster %s): %w
• gRPCVtctldClient in a SHUTDOWN state
• no function closure for Event stream specified
• error dialing binlog server: %v
• error received from Stream %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f0d146340509ac79. Report an issue: GitHub.