What causes "grpc: the connection is drained" in grpc/grpc-go?

Retry the RPC on a fresh pick — the balancer will route to a remaining READY SubConn; for idempotent calls wrap with a retry interceptor.

grpc/grpc-go · error

grpc: the connection is drained

Error message

grpc: the connection is drained

What it means

Internal sentinel errConnDrain (clientconn.go:71-72) meaning the channel is draining and no longer accepts new RPCs. It is passed to addrConn.tearDown when a SubConn is shut down because the balancer removed its address (balancer_wrapper.go:355 removeAddrConn(... , errConnDrain)). On tearDown, if err==errConnDrain the transport is GracefulClose'd (clientconn.go:1713) so in-flight streams finish.

Source

Thrown at clientconn.go:72

	_ "google.golang.org/grpc/internal/resolver/passthrough" // To register passthrough resolver.
	_ "google.golang.org/grpc/internal/resolver/unix"        // To register unix resolver.
	_ "google.golang.org/grpc/resolver/dns"                  // To register dns resolver.
)

const (
	// minimum time to give a connection to complete
	minConnectTimeout = 20 * time.Second
)

var (
	// ErrClientConnClosing indicates that the operation is illegal because
	// the ClientConn is closing.
	//
	// Deprecated: this error should not be relied upon by users; use the status
	// code of Canceled instead.
	ErrClientConnClosing = status.Error(codes.Canceled, "grpc: the client connection is closing")
	// errConnDrain indicates that the connection starts to be drained and does not accept any new RPCs.
	errConnDrain = errors.New("grpc: the connection is drained")
	// errConnClosing indicates that the connection is closing.
	errConnClosing = errors.New("grpc: the connection is closing")
	// errConnIdling indicates the connection is being closed as the channel
	// is moving to an idle mode due to inactivity.
	errConnIdling = errors.New("grpc: the connection is closing due to channel idleness")
	// invalidDefaultServiceConfigErrPrefix is used to prefix the json parsing error for the default
	// service config.
	invalidDefaultServiceConfigErrPrefix = "grpc: the provided default service config is invalid"
	// PickFirstBalancerName is the name of the pick_first balancer.
	PickFirstBalancerName = pickfirst.Name
)

// The following errors are returned from Dial and DialContext
var (
	// errNoTransportSecurity indicates that there is no transport security
	// being set for ClientConn. Users should either set one or explicitly
	// call WithInsecure DialOption to disable security.
	errNoTransportSecurity = errors.New("grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials)")

View on GitHub (pinned to 03255a9237)

Solutions

Retry the RPC on a fresh pick — the balancer will route to a remaining READY SubConn; for idempotent calls wrap with a retry interceptor.
Verify the resolver/backend set still contains healthy addresses; if all are draining, wait for new ones or fail over.
Do not hold references to a specific SubConn across long-lived logic; rely on the picker instead.
Enable the retry policy in the service config for the affected method so transient drain failures are retried transparently.

Example fix

// before — single attempt on a draining backend
resp, err := client.Get(ctx, req)

// after — enable retry in service config (methodConfig retryPolicy)
{
  "methodConfig": [{
    "name": [{"service": "pkg.Svc", "method": "Get"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

Defensive patterns

Strategy: retry

Validate before calling

// Avoid issuing RPCs against a draining subchannel; rely on the picker
// (No direct subchannel liveness check is exposed — instead use retry policy.)

Try / catch

if errors.Is(err, errConnDrain) || status.Code(err) == codes.Unavailable {
    // retry; balancer will pick another READY subchannel
}

Prevention

Enable retry on UNAVAILABLE for idempotent methods.
Do not cache a specific SubConn in application code.
Expect drain noise during deploys; classify it as transient.

When it happens

Trigger: A SubConn is removed by the balancer (address list shrank, resolver dropped it, server sent GOAWAY causing address removal) and that SubConn's transport is torn down with errConnDrain. New RPCs that try to use it get this error.

Common situations: Resolver address list changed during a rolling deploy; server sent GOAWAY; xDS removed a locality/endpoint; backend scaled down; in-flight RPC issued against a SubConn the instant it was removed.

Related errors

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/71941429c67d7b53. Report an issue: GitHub.