out of nodes are alive, configuration will result in quorum…

Error message

%d out of %d nodes are alive, configuration will result in quorum loss

What it means

The etcdraft chain refuses config updates that would leave the raft cluster unable to maintain quorum (orderer/consensus/etcdraft/chain.go:1546). After computing membership changes, UnacceptableQuorumLoss compares currently alive nodes against the new effective quorum; if removing nodes would make the alive count insufficient, the update is rejected. This protects the cluster from permanent unavailability.

Solutions

Bring the remaining nodes back online so enough nodes are alive before submitting the update
Shrink the cluster in stages (e.g. 5→4→3), re-verifying liveness at each step
Remove the nodes from the config first while they are still running and reachable
Check ActiveNodes liveness via the orderer metrics/logs before submitting

Example fix

// before: removing 3 of 5 consenters while 2 are down
ConfigUpdate: remove orderer3, orderer4, orderer5 (only 2 alive)
// after: staged removal
Step 1: restart orderer4/orderer5, then remove orderer5 only; Step 2: remove orderer4 once quorum-verified

Defensive patterns

Strategy: validation

Validate before calling

active := c.ActiveNodes.Load().([]uint64)
if changes.UnacceptableQuorumLoss(active) {
    return fmt.Errorf("abort update: only %d of %d nodes alive would break quorum", len(active), len(changes.TotalNodes))
}

Type guard

func safeToRemove(currentAlive, currentTotal, removing int) bool {
    remaining := currentTotal - removing
    quorum := remaining/2 + 1 // majority quorum
    return currentAlive-0 >= quorum
}

Try / catch

if err := submitConfigUpdate(cfg); err != nil {
    if strings.Contains(err.Error(), "quorum loss") {
        // restart dead nodes or remove fewer consenters per step
    }
}

Prevention

Verify node liveness (metrics/logs) before submitting shrink updates
Shrink raft clusters one node at a time
Keep a majority of consenters running during maintenance windows

When it happens

Trigger: A config update removes consenters such that the number of currently responsive nodes would fall below the quorum required by the resulting cluster size (e.g. shrinking 5-node cluster to 3 while only 2 nodes are alive).

Common situations: Decommissioning several orderers at once during infrastructure cleanup; removing nodes whose containers are stopped; maintenance that temporarily shuts down nodes before submitting the shrink update.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/b9703a0cfe09dac5. Report an issue: GitHub.

Appendix: source

Thrown at orderer/consensus/etcdraft/chain.go:1546

	c.raftMetadataLock.RUnlock()

	dummyOldConsentersMap := CreateConsentersMap(dummyOldBlockMetadata, oldMetadata)
	changes, err := ComputeMembershipChanges(dummyOldBlockMetadata, dummyOldConsentersMap, newMetadata.GetConsenters())
	if err != nil {
		return err
	}

	// new config metadata was verified above. Additionally need to check new consenters for certificates expiration
	for _, c := range changes.AddedNodes {
		if err := validateConsenterTLSCerts(c, verifyOpts, false); err != nil {
			return errors.Wrapf(err, "consenter %s:%d has invalid certificates", c.GetHost(), c.GetPort())
		}
	}

	active := c.ActiveNodes.Load().([]uint64)
	if changes.UnacceptableQuorumLoss(active) {
		c.logger.Debugf("%d out of %d nodes are alive - %+v", len(active), len(dummyOldConsentersMap), active)
		return errors.Errorf("%d out of %d nodes are alive, configuration will result in quorum loss", len(active), len(dummyOldConsentersMap))
	}

	return nil
}

// StatusReport returns the ConsensusRelation & Status
func (c *Chain) StatusReport() (types.ConsensusRelation, types.Status) {
	c.statusReportMutex.Lock()
	defer c.statusReportMutex.Unlock()

	return c.consensusRelation, c.status
}

func (c *Chain) suspectEviction() bool {
	if c.isRunning() != nil {
		return false
	}

View on GitHub (pinned to 2736b63f8f)