hyperledger/fabric · warning

connection to %d(%s) is in state %s

Error message

connection to %d(%s) is in state %s

What it means

In createRemoteContext, the probeConnection helper checks the gRPC ClientConn state right after dialing; if the connection is still in connectivity.Connecting, it refuses to hand back a RemoteContext. This is a transient condition: the TCP/TLS handshake to the remote orderer has not completed yet, so the stream factory would be backed by a not-ready connection.

Source

Thrown at orderer/common/cluster/comm.go:292

		cert, err := x509.ParseCertificate(stub.ServerTLSCert)
		if err != nil {
			pemString := string(pem.EncodeToMemory(&pem.Block{Bytes: stub.ServerTLSCert}))
			c.Logger.Errorf("Invalid DER for channel %s, endpoint %s, ID %d: %v", channel, stub.Endpoint, stub.ID, pemString)
			return nil, errors.Wrap(err, "invalid certificate DER")
		}

		c.Logger.Debug("Connecting to", stub.RemoteNode, "for channel", channel)

		conn, err := c.Connections.Connection(stub.Endpoint, stub.ServerTLSCert)
		if err != nil {
			c.Logger.Warningf("Unable to obtain connection to %d(%s) (channel %s): %v", stub.ID, stub.Endpoint, channel, err)
			return nil, err
		}

		probeConnection := func(conn *grpc.ClientConn) error {
			connState := conn.GetState()
			if connState == connectivity.Connecting {
				return errors.Errorf("connection to %d(%s) is in state %s", stub.ID, stub.Endpoint, connState)
			}
			return nil
		}

		clusterClient := orderer.NewClusterClient(conn)
		getStream := func(ctx context.Context) (StepClientStream, error) {
			stream, err := clusterClient.Step(ctx)
			if err != nil {
				return nil, err
			}
			stepClientStream := &CommClientStream{
				StepClient: stream,
			}
			return stepClientStream, nil
		}

		workerCountReporter := workerCountReporter{
			channel: channel,

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Retry the request after a short delay - the connection state usually transitions to Ready on its own.
2. Verify the remote orderer endpoint (host:port) is correct and the process is running and listening.
3. Check network latency/firewall rules between orderers that may slow or block the TLS handshake.
4. Increase gRPC dial/connect timeouts in the cluster configuration if handshakes are consistently slow.

Example fix

// before
remoteCtx, err := comm.Remote(channel, id) // may hit Connecting state
// after
remoteCtx, err := comm.Remote(channel, id)
if err != nil && strings.Contains(err.Error(), "is in state Connecting") {
    time.Sleep(500 * time.Millisecond)
    remoteCtx, err = comm.Remote(channel, id)
}

// Go: probe the conn state yourself before using the remote context
state := conn.GetState()
if state == connectivity.Connecting {
    // wait for readiness instead of failing
    conn.WaitForStateChange(context.Background(), state)
}

func connReady(conn *grpc.ClientConn) bool {
    return conn.GetState() == connectivity.Ready
}

var remoteCtx *cluster.RemoteContext
err := backoff.Retry(func() error {
    var err error
    remoteCtx, err = comm.Remote(channel, id)
    if err != nil && strings.Contains(err.Error(), "is in state Connecting") {
        return err // transient - retry
    }
    return backoff.Permanent(err) // other errors are fatal
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 5))

When it happens

Trigger: Dialing a remote cluster node whose gRPC channel is still in the Connecting state - typically slow network, TLS handshake in progress, or the remote endpoint not yet listening when the first request arrives.

Common situations: Cold start of the ordering service where nodes dial peers before all nodes are up; slow or lossy network between datacenters; DNS resolution delays; remote orderer still booting or restarting under load.

Related errors

• didn't receive a response within %v
• timeout waiting for channel creation
• failed to create new connection
• invalid request object
• response is of type %v, but expected a block

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/b944db8664a7d00e. Report an issue: GitHub.