hyperledger/fabric · warning

connection to ( ) is in state

Error message

connection to %d(%s) is in state %s

What it means

During stub activation, probeConnection checks the gRPC ClientConn state and rejects it if it is still in connectivity.Connecting - meaning the TCP/TLS connection to the remote cluster node was initiated but the handshake has not completed yet. The RemoteContext's ProbeConn is invoked before the connection is considered usable, so half-open connections are not handed to consensus.

Solutions

1. Check the remote orderer at that endpoint is running and its cluster (port 7050) listener is up: verify stub.Endpoint host:port.
2. Retry after a short delay - the state is transient and will become Ready or the underlying Connect will fail with a definitive error.
3. Verify TLS certs (ServerRootCA vs the remote server cert) and that the gRPC connection is not stuck in backoff due to failed handshakes.
4. Check network/DNS connectivity between orderers (firewall rules on the cluster port).

Example fix

// before
conn, _ := connections.Connect(endpoint, ca) // may be Connecting
rc, err := remoteContext() // probe fails: state Connecting
// after
// retry with backoff until state is Ready or a hard error surfaces
for attempts := 0; attempts < 5; attempts++ {
    if conn.GetState() == connectivity.Ready { break }
    time.Sleep(backoff)
}
rc, err := remoteContext()

func connectionReady(conn *grpc.ClientConn) bool {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    return conn.WaitForStateChange(ctx, conn.GetState()) && conn.GetState() == connectivity.Ready
}

func isConnecting(state connectivity.State) bool {
    return state == connectivity.Connecting
}

rc, err := comm.Remote(channel, id)
if err != nil {
    if strings.Contains(err.Error(), "is in state Connecting") {
        // transient: probe caught the conn mid-handshake; retry with backoff
        time.Sleep(500 * time.Millisecond)
        rc, err = comm.Remote(channel, id)
    }
    return err
}

When it happens

Trigger: createRemoteContext successfully obtained a ClientConn via Connections.Connect, but conn.GetState() returned connectivity.Connecting when probed - typically when the connection attempt is still in flight (slow network, remote peer not yet listening, TLS handshake pending) at the moment the stub is activated and probed.

Common situations: Remote orderer is down or still starting; network latency or DNS slowness delaying the gRPC connect; TLS certificate problems causing a long/hanging handshake; transient race right after channel configure when stubs are activated en masse.

Related errors

• Error getting broadcast client
• failed connecting to discovery service
• received nil proposal response
• badly formatted message, cannot extract channel
• Cannot read channels list response

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/f8714ec88e62a348. Report an issue: GitHub.

Appendix: source

Thrown at orderer/common/cluster/commauth.go:173

	// Activate the stub
	stub.Activate(ac.createRemoteContext(stub, channel))
}

func (ac *AuthCommMgr) createRemoteContext(stub *Stub, channel string) func() (*RemoteContext, error) {
	return func() (*RemoteContext, error) {
		ac.Logger.Debugf("Connecting to node: %v for channel: %v", stub.RemoteNode.NodeAddress, channel)

		conn, err := ac.Connections.Connect(stub.Endpoint, stub.RemoteNode.ServerRootCA)
		if err != nil {
			ac.Logger.Warningf("Unable to obtain connection to %d(%s) (channel %s): %v", stub.ID, stub.Endpoint, channel, err)
			return nil, err
		}

		probeConnection := func(conn *grpc.ClientConn) error {
			connState := conn.GetState()
			if connState == connectivity.Connecting {
				return errors.Errorf("connection to %d(%s) is in state %s", stub.ID, stub.Endpoint, connState)
			}
			return nil
		}

		clusterClient := orderer.NewClusterNodeServiceClient(conn)
		getStepClientStream := func(ctx context.Context) (StepClientStream, error) {
			stream, err := clusterClient.Step(ctx)
			if err != nil {
				return nil, err
			}

			membersMapping, exists := ac.Chan2Members[channel]
			if !exists {
				return nil, errors.Errorf("channel members not initialized")
			}
			nodeStub := membersMapping.LookupByIdentity(ac.NodeIdentity)
			if nodeStub == nil {
				return nil, errors.Errorf("node identity is missing in channel")

View on GitHub (pinned to 2736b63f8f)