hyperledger/fabric · info

communication has been shut down

Error message

communication has been shut down

What it means

Comm.Remote is the public API for obtaining a RemoteContext to talk to a specific cluster node on a channel. If the Comm instance has been shut down (c.shutdown is true, typically after Comm.Stop/Serve termination), all outbound cluster communication is refused with this sentinel error instead of attempting dials on dead infrastructure.

Source

Thrown at orderer/common/cluster/comm.go:123

	stub := mapping.LookupByClientCert(cert)
	if stub == nil {
		return nil, errors.Errorf("certificate extracted from TLS connection isn't authorized")
	}
	return &requestContext{
		channel: channel,
		sender:  stub.ID,
	}, nil
}

// Remote obtains a RemoteContext linked to the destination node on the context
// of a given channel
func (c *Comm) Remote(channel string, id uint64) (*RemoteContext, error) {
	c.Lock.RLock()
	defer c.Lock.RUnlock()

	if c.shutdown {
		return nil, errors.New("communication has been shut down")
	}

	mapping, exists := c.Chan2Members[channel]
	if !exists {
		return nil, errors.Errorf("channel %s doesn't exist", channel)
	}
	stub := mapping.ByID(id)
	if stub == nil {
		return nil, errors.Errorf("node %d doesn't exist in channel %s's membership", id, channel)
	}

	if stub.Active() {
		return stub.RemoteContext, nil
	}

	err := stub.Activate(c.createRemoteContext(stub, channel))
	if err != nil {
		return nil, errors.WithStack(err)

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Check for shutdown before calling Remote and recreate the Comm if continued operation is needed
2. Fix lifecycle ordering so chains stop issuing RPCs before Comm.Stop() is invoked
3. In tests, create a fresh Comm instance after each Stop instead of reusing the shut-down one
4. Treat this error as terminal (no retry) and propagate cancellation of dependent work

Example fix

// before (reusing shut-down Comm)
remote, err := comm.Remote("mychannel", 1)
// after
if comm.IsShutdown() {
    comm = cluster.NewComm(...) // recreate
}
remote, err := comm.Remote("mychannel", 1)

// Check shutdown flag before using Remote (exposed via channel closure)
select {
case <-comm.ShutdownC:
    return errors.New("comm shut down; aborting Remote call")
default:
    // safe to proceed
}

func commIsLive(c *cluster.Comm) bool {
    select {
    case <-c.ShutdownC:
        return false
    default:
        return true
    }
}

remote, err := comm.Remote(channel, nodeID)
if err != nil {
    if err.Error() == "communication has been shut down" {
        // terminal: cancel work, do not retry on this Comm instance
        return nil, ErrCommTerminated
    }
    return nil, err // transient dial errors may be retried
}

When it happens

Trigger: Calling Comm.Remote(channel, id) after c.Stop() or after the Comm's ShutdownC channel has been fired — e.g. an RPC/probe initiated concurrently with orderer shutdown, or reusing a stale Comm reference after the chain was halted.

Common situations: Etcdraft chain still issuing requests during graceful orderer shutdown; a long-running component holding a Comm reference past the channel's closure; tests reusing a Comm after Stop without recreating it.

Related errors

• communication has been shut down
• OrdererOrg config does not allow sub-groups
• Attempted to set the batch size preferred max bytes (%v) gre
• Attempted to set the batch timeout to a invalid value: %s
• Attempted to set the batch timeout to a non-positive value:

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/ad6318eed599bd3d. Report an issue: GitHub.