hyperledger/fabric · warning

connection to %d(%s) is in state %s

Error message

connection to %d(%s) is in state %s

What it means

During stub activation, probeConnection checks the gRPC ClientConn state and rejects it if it is still in connectivity.Connecting - meaning the TCP/TLS connection to the remote cluster node was initiated but the handshake has not completed yet. The RemoteContext's ProbeConn is invoked before the connection is considered usable, so half-open connections are not handed to consensus.

Source

Thrown at orderer/common/cluster/commauth.go:173

	// Activate the stub
	stub.Activate(ac.createRemoteContext(stub, channel))
}

func (ac *AuthCommMgr) createRemoteContext(stub *Stub, channel string) func() (*RemoteContext, error) {
	return func() (*RemoteContext, error) {
		ac.Logger.Debugf("Connecting to node: %v for channel: %v", stub.RemoteNode.NodeAddress, channel)

		conn, err := ac.Connections.Connect(stub.Endpoint, stub.RemoteNode.ServerRootCA)
		if err != nil {
			ac.Logger.Warningf("Unable to obtain connection to %d(%s) (channel %s): %v", stub.ID, stub.Endpoint, channel, err)
			return nil, err
		}

		probeConnection := func(conn *grpc.ClientConn) error {
			connState := conn.GetState()
			if connState == connectivity.Connecting {
				return errors.Errorf("connection to %d(%s) is in state %s", stub.ID, stub.Endpoint, connState)
			}
			return nil
		}

		clusterClient := orderer.NewClusterNodeServiceClient(conn)
		getStepClientStream := func(ctx context.Context) (StepClientStream, error) {
			stream, err := clusterClient.Step(ctx)
			if err != nil {
				return nil, err
			}

			membersMapping, exists := ac.Chan2Members[channel]
			if !exists {
				return nil, errors.Errorf("channel members not initialized")
			}
			nodeStub := membersMapping.LookupByIdentity(ac.NodeIdentity)
			if nodeStub == nil {
				return nil, errors.Errorf("node identity is missing in channel")

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Check the remote orderer at that endpoint is running and its cluster (port 7050) listener is up: verify stub.Endpoint host:port.
2. Retry after a short delay - the state is transient and will become Ready or the underlying Connect will fail with a definitive error.
3. Verify TLS certs (ServerRootCA vs the remote server cert) and that the gRPC connection is not stuck in backoff due to failed handshakes.
4. Check network/DNS connectivity between orderers (firewall rules on the cluster port).

Example fix

// before
conn, _ := connections.Connect(endpoint, ca) // may be Connecting
rc, err := remoteContext() // probe fails: state Connecting
// after
// retry with backoff until state is Ready or a hard error surfaces
for attempts := 0; attempts < 5; attempts++ {
    if conn.GetState() == connectivity.Ready { break }
    time.Sleep(backoff)
}
rc, err := remoteContext()

func connectionReady(conn *grpc.ClientConn) bool {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    return conn.WaitForStateChange(ctx, conn.GetState()) && conn.GetState() == connectivity.Ready
}

func isConnecting(state connectivity.State) bool {
    return state == connectivity.Connecting
}

rc, err := comm.Remote(channel, id)
if err != nil {
    if strings.Contains(err.Error(), "is in state Connecting") {
        // transient: probe caught the conn mid-handshake; retry with backoff
        time.Sleep(500 * time.Millisecond)
        rc, err = comm.Remote(channel, id)
    }
    return err
}

When it happens

Trigger: createRemoteContext successfully obtained a ClientConn via Connections.Connect, but conn.GetState() returned connectivity.Connecting when probed - typically when the connection attempt is still in flight (slow network, remote peer not yet listening, TLS handshake pending) at the moment the stub is activated and probed.

Common situations: Remote orderer is down or still starting; network latency or DNS slowness delaying the gRPC connect; TLS certificate problems causing a long/hanging handshake; transient race right after channel configure when stubs are activated en masse.

Related errors

• failed connecting to discovery service
• Error getting broadcast client: %s
• received nil proposal response
• reading http response body: %s
• orderer `%s` hung up without sending status

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/f8714ec88e62a348. Report an issue: GitHub.