What causes "gRPCVtctldClient in a SHUTDOWN state" in vitessio/vitess?

Create a new vtctldclient instance via vtctldclient.New instead of reusing a closed one

gRPCVtctldClient in a SHUTDOWN state

Error message

gRPCVtctldClient in a SHUTDOWN state

What it means

gRPCVtctldClient exposes ErrConnectionShutdown, a sentinel error returned by WaitForReady when the client is in a SHUTDOWN state — i.e., Close was called and the underlying gRPC connection can no longer be used for RPCs. The wrapper also detects the raw 'grpc: the client connection is closed' message and maps it to this sentinel.

Source

Thrown at go/vt/vtctl/grpcvtctldclient/client.go:37

package grpcvtctldclient

import (
	"context"
	"errors"
	"fmt"

	"google.golang.org/grpc"
	"google.golang.org/grpc/connectivity"

	"vitess.io/vitess/go/vt/grpcclient"
	"vitess.io/vitess/go/vt/vtctl/grpcclientcommon"
	"vitess.io/vitess/go/vt/vtctl/vtctldclient"

	vtctlservicepb "vitess.io/vitess/go/vt/proto/vtctlservice"
)

var (
	ErrConnectionShutdown = errors.New("gRPCVtctldClient in a SHUTDOWN state")
	ErrConnectionTimeout  = errors.New("gRPC connection wait time exceeded")
)

const connClosedMsg = "grpc: the client connection is closed"

type gRPCVtctldClient struct {
	cc *grpc.ClientConn
	c  vtctlservicepb.VtctldClient
}

//go:generate -command grpcvtctldclient go run ../vtctldclient/codegen
//go:generate grpcvtctldclient --out client_gen.go

func gRPCVtctldClientFactory(ctx context.Context, addr string) (vtctldclient.VtctldClient, error) {
	opt, err := grpcclientcommon.SecureDialOption()
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

Create a new vtctldclient instance via vtctldclient.New instead of reusing a closed one
Fix control flow so Close is the last operation on the client (no RPCs after deferred Close)
Guard concurrent use of the client with synchronization or per-goroutine clients
Check errors.Is(err, grpcvtctldclient.ErrConnectionShutdown) to distinguish this from network failures

Example fix

// before
client, _ := vtctldclient.New(...)
defer client.Close()
// ... client reused later in another function -> ErrConnectionShutdown
// after
client, _ := vtctldclient.New(...)
if err := client.WaitForReady(ctx); err != nil {
    if errors.Is(err, grpcvtctldclient.ErrConnectionShutdown) {
        client, err = vtctldclient.New(...) // recreate and retry
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

if client == nil || isClosed(client) {
    client, err = vtctldclient.New(addr)
    if err != nil {
        return err
    }
}

Type guard

func isConnectionShutdownErr(err error) bool {
    return errors.Is(err, grpcvtctldclient.ErrConnectionShutdown) ||
        strings.Contains(err.Error(), "grpc: the client connection is closed")
}

Try / catch

if err := client.WaitForReady(ctx); err != nil {
    if errors.Is(err, grpcvtctldclient.ErrConnectionShutdown) {
        client, err = vtctldclient.New(addr)
        if err != nil {
            return err
        }
        return client.WaitForReady(ctx)
    }
    return err
}

Prevention

Never issue RPCs after client.Close(); make Close the final call
Do not share one vtctldclient across goroutines without synchronization
Avoid deferring Close in setup functions if the client outlives them
Check for ErrConnectionShutdown with errors.Is to recreate the client

When it happens

Trigger: Calling WaitForReady (or any vtctldclient RPC backed by it) after Close() was invoked on the client; using a client whose Close was called by a deferred cleanup or by a previous shutdown sequence in the same process.

Common situations: Reusing a vtctldclient created in a setup function whose defer Close already ran; calling RPCs during vtctld shutdown/race with Close; long-lived processes that Close the client on one code path and later call it again.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d00747f148a67b2e. Report an issue: GitHub.