apache/cassandra · error · java.lang.IllegalStateException

Too many nodes are currently DOWN to safely perform the reco

Error message

Too many nodes are currently DOWN to safely perform the reconfiguration

What it means

Thrown by PrepareCMSReconfiguration.verify() when preparing a new CMS membership (e.g. moving replicas to a new DC or RF change): after excluding DOWN nodes, the candidate new CMS group is smaller than a majority of the sum of the DC RFs. Committing would risk losing quorum for cluster metadata, so the transformation is rejected.

Source

Thrown at src/java/org/apache/cassandra/tcm/transformations/cms/PrepareCMSReconfiguration.java:121

        Set<NodeId> newCms = prepareNewCMS(dcRF, prev);
        if (newCms.equals(currentCms))
            return Diff.NOCHANGE;
        return diff(currentCms, newCms);
    }

    private Set<NodeId> prepareNewCMS(Map<String, Integer> dcRf, ClusterMetadata prev)
    {
        CMSPlacementStrategy placementStrategy = new CMSPlacementStrategy(dcRf, additionalFilteringPredicate(downNodes));
        return placementStrategy.reconfigure(prev);
    }

    public void verify(ClusterMetadata prev)
    {
        Map<String, Integer> dcRf = extractRf(newReplicationParams(prev));
        int expectedSize = dcRf.values().stream().mapToInt(Integer::intValue).sum();
        Set<NodeId> newCms = prepareNewCMS(dcRf, prev);
        if (newCms.size() < (expectedSize / 2) + 1)
            throw new IllegalStateException("Too many nodes are currently DOWN to safely perform the reconfiguration");
    }

    private static void serializeDownNodes(PrepareCMSReconfiguration transformation, DataOutputPlus out, Version version) throws IOException
    {
        out.writeUnsignedVInt32(transformation.downNodes.size());
        for (NodeId nodeId : transformation.downNodes)
            NodeId.serializer.serialize(nodeId, out, version);
    }

    private static Set<NodeId> deserializeDownNodes(DataInputPlus in, Version version) throws IOException
    {
        Set<NodeId> downNodes = new HashSet<>();
        int count = in.readUnsignedVInt32();
        for (int i = 0; i < count; i++)
            downNodes.add(NodeId.serializer.deserialize(in, version));
        return downNodes;
    }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

  1. Bring the DOWN nodes back up before running the reconfiguration
  2. If nodes are permanently lost, first remove them from the cluster (nodetool removenode) so the majority computation uses live members
  3. Re-run `cms reconfigure` once a majority of the target DCs' RF nodes is available

Example fix

# before
cms reconfigure --all-dcs  # fails while 3 of 5 CMS nodes are down
# after
nodetool start / repair the down nodes, then
cms reconfigure --all-dcs
Defensive patterns

Strategy: validation

Validate before calling

// before cms reconfigure, check liveness of CMS members
Map<String,Integer> dcRf = extractRf(newReplicationParams);
Set<NodeId> newCms = prepareNewCMS(dcRf, ClusterMetadata.current());
int expected = dcRf.values().stream().mapToInt(Integer::intValue).sum();
if (newCms.size() < expected / 2 + 1) { /* bring nodes up first */ }

Try / catch

try { cms.reconfigure(params); } catch (IllegalStateException e) { if (e.getMessage().contains("Too many nodes are currently DOWN")) { /* repair nodes or removenode, then retry */ } else throw e; }

Prevention

When it happens

Trigger: Calling cms reconfigure (newReplicationParams) while enough nodes in the relevant datacenters are DOWN that prepareNewCMS() yields fewer than (expectedSize/2)+1 live nodes.

Common situations: Reconfiguring CMS placement during a multi-node outage; dropping RF or migrating CMS to a new DC while replicas are offline; running reconfigure immediately after several node failures.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/97bac580422780af. Report an issue: GitHub.