What causes "conn was closed" in gravitational/teleport?

Check whether Client.Close() (or process shutdown) was called before this request; recreate the client with teleport.NewClient instead of reusing the closed one.

gravitational/teleport · error

conn was closed

Error message

conn was closed

What it means

waitForConnectionReady polls the client's gRPC connection state until it becomes Ready. The error "conn was closed" is returned when c.conn is nil, meaning the ClientConn field was never set or has been closed/nulled out (e.g. after Client.Close) while someone is still waiting for the connection to become ready. It indicates use of a client whose underlying gRPC channel is gone, not a transient connectivity problem.

Source

Thrown at api/client/client.go:611

	return func(ctx context.Context, addr string) (net.Conn, error) {
		if c.isClosed() {
			return nil, trace.ConnectionProblem(nil, "client is closed")
		}
		conn, err := c.dialer.DialContext(ctx, "tcp", addr)
		if err != nil {
			return nil, trace.ConnectionProblem(err, "failed to dial: %v", err)
		}
		return conn, nil
	}
}

// waitForConnectionReady waits for the client's grpc connection finish dialing, returning an error
// if the ctx is canceled or the client's gRPC connection enters an unexpected state. This can be used
// alongside the DialInBackground client config option to wait until background dialing has completed.
func (c *Client) waitForConnectionReady(ctx context.Context) error {
	for {
		if c.conn == nil {
			return errors.New("conn was closed")
		}
		switch state := c.conn.GetState(); state {
		case connectivity.Ready:
			return nil
		case connectivity.TransientFailure, connectivity.Connecting, connectivity.Idle:
			// Wait for expected state transitions. For details about grpc.ClientConn state changes
			// see https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md
			if !c.conn.WaitForStateChange(ctx, state) {
				// ctx canceled
				return trace.Wrap(ctx.Err())
			}
		case connectivity.Shutdown:
			return trace.Errorf("client gRPC connection entered an unexpected state: %v", state)
		}
	}
}

// Config contains configuration of the client

View on GitHub (pinned to 1283425b60)

Solutions

Check whether Client.Close() (or process shutdown) was called before this request; recreate the client with teleport.NewClient instead of reusing the closed one.
Ensure the Client is constructed normally (config not leaving conn nil); with DialInBackground, call waitForConnectionReady only on a live client instance.
Fix lifecycle ownership: don't share one client across goroutines where one may close it while others dial/wait; guard with a reference count or context cancellation.

Example fix

// before
cl, _ := client.New(ctx, cfg)
_ = cl.Close()
users, err := cl.GetUsers(ctx, false) // "conn was closed"

// after
cl, _ := client.New(ctx, cfg)
defer cl.Close()
users, err := cl.GetUsers(ctx, false)

Defensive patterns

Strategy: try-catch

Validate before calling

if cl == nil || cl.IsClosed() { cl, err = client.New(ctx, cfg) }

Type guard

func clientUsable(c *client.Client) bool { return c != nil && !c.IsClosed() }

Try / catch

users, err := cl.GetUsers(ctx, false)
if err != nil && strings.Contains(err.Error(), "conn was closed") {
    cl, err = client.New(ctx, cfg)
    if err != nil { return trace.Wrap(err) }
    users, err = cl.GetUsers(ctx, false)
}

Prevention

Use defer cl.Close() in the same scope that creates the client so it outlives all calls.
Cancel a shared context to stop in-flight work instead of closing the client concurrently.
Recreate the client on reuse-after-idle rather than caching a closed instance.

When it happens

Trigger: Calling any Client RPC (or using DialInBackground then waitForConnectionReady) after the client's gRPC connection has been closed via Client.Close(), or on a Client constructed without a valid connection so c.conn is nil.

Common situations: Long-lived processes reusing a Teleport client after shutdown/teardown; race between Close() and in-flight calls; DialInBackground usage where the dial never completes because the conn was torn down; tests or proxies closing clients while background work still runs.

Related errors

AI-assisted analysis of gravitational/teleport@1283425b60 (2026-09-02). Data as JSON: /api/errors/67aa85dd059c118c. Report an issue: GitHub.