What causes "communication has been shut down" in hyperledger/fabric?

Ensure all callers stop issuing Remote() requests before calling Shutdown() - coordinate via context cancellation or wait groups.

hyperledger/fabric · error

communication has been shut down

Error message

communication has been shut down

What it means

AuthCommMgr.Remote (orderer/common/cluster/commauth.go) checks an internal shutdown flag under a read lock before performing any work; if the manager has been shut down, it refuses all remote operations with this error. It signals the comm manager's lifecycle has ended - typically during orderer termination - and further calls are invalid.

Source

Thrown at orderer/common/cluster/commauth.go:48

	Lock           sync.RWMutex
	shutdown       bool
	shutdownSignal chan struct{}

	Chan2Members MembersByChannel
	Connections  *ConnectionsMgr

	SendBufferSize int
	NodeIdentity   []byte
	Signer         identity.Signer
}

func (ac *AuthCommMgr) Remote(channel string, id uint64) (*RemoteContext, error) {
	ac.Lock.RLock()
	defer ac.Lock.RUnlock()

	if ac.shutdown {
		return nil, errors.New("communication has been shut down")
	}

	mapping, exists := ac.Chan2Members[channel]
	if !exists {
		return nil, errors.Errorf("channel %s doesn't exist", channel)
	}
	stub := mapping.ByID(id)
	if stub == nil {
		return nil, errors.Errorf("node %d doesn't exist in channel %s's membership", id, channel)
	}

	if stub.Active() {
		return stub.RemoteContext, nil
	}

	err := stub.Activate(ac.createRemoteContext(stub, channel))
	if err != nil {
		return nil, errors.WithStack(err)

View on GitHub (pinned to 2736b63f8f)

Solutions

Ensure all callers stop issuing Remote() requests before calling Shutdown() - coordinate via context cancellation or wait groups.
Restart the orderer/servicer; after shutdown the manager cannot be revived, so a new instance is required.
Audit goroutine lifecycle so background tasks are joined/stopped before the comm manager is shut down.
In tests, close the manager only after all dependent operations complete (defer close last or use sync.WaitGroup).

Example fix

// before
go func() { commMgr.Remote(ch, id) }() // may run after Shutdown
commMgr.Shutdown()
// after
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); commMgr.Remote(ch, id) }()
wg.Wait() // drain callers first
commMgr.Shutdown()

Defensive patterns

Strategy: try-catch

Validate before calling

// Go: expose/track shutdown state and check it before calling Remote
if mgr.IsShutdown() { // if exposed; otherwise rely on the error
    return errors.New("comm manager already shut down")
}

Type guard

func (ac *AuthCommMgr) Available() bool {
    ac.Lock.RLock()
    defer ac.Lock.RUnlock()
    return !ac.shutdown
}

Try / catch

remote, err := authMgr.Remote(channel, id)
if err != nil {
    if err.Error() == "communication has been shut down" {
        // manager closed; recreate or propagate termination
        return errShutdown
    }
    return err
}

Prevention

Cancel dependent goroutines (via context) before calling Shutdown() on the comm manager.
Use sync.WaitGroup or errgroup to drain in-flight Remote calls before shutdown.
In tests, defer Shutdown() last so all operations complete first.
Never reuse a shut-down manager; construct a fresh instance if the service resumes.

When it happens

Trigger: Calling AuthCommMgr.Remote(channel, id) after Shutdown()/Close() was invoked on the manager, or while the orderer is shutting down and another goroutine still issues remote requests.

Common situations: Race during graceful shutdown where in-flight consensus or forwarding calls hit Remote after shutdown begins; tests tearing down the comm manager while background tasks still run; double-close patterns leaving dependent components calling into a dead manager.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/fd8863f98e03948d. Report an issue: GitHub.