{"record":{"id":"97bac580422780af","repo":"apache/cassandra","slug":"too-many-nodes-are-currently-down-to-safely-perfor","errorCode":null,"errorMessage":"Too many nodes are currently DOWN to safely perform the reconfiguration","messagePattern":"Too many nodes are currently DOWN to safely perform the reconfiguration","errorType":"exception","errorClass":"java.lang.IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/transformations/cms/PrepareCMSReconfiguration.java","lineNumber":121,"sourceCode":"        Set<NodeId> newCms = prepareNewCMS(dcRF, prev);\n        if (newCms.equals(currentCms))\n            return Diff.NOCHANGE;\n        return diff(currentCms, newCms);\n    }\n\n    private Set<NodeId> prepareNewCMS(Map<String, Integer> dcRf, ClusterMetadata prev)\n    {\n        CMSPlacementStrategy placementStrategy = new CMSPlacementStrategy(dcRf, additionalFilteringPredicate(downNodes));\n        return placementStrategy.reconfigure(prev);\n    }\n\n    public void verify(ClusterMetadata prev)\n    {\n        Map<String, Integer> dcRf = extractRf(newReplicationParams(prev));\n        int expectedSize = dcRf.values().stream().mapToInt(Integer::intValue).sum();\n        Set<NodeId> newCms = prepareNewCMS(dcRf, prev);\n        if (newCms.size() < (expectedSize / 2) + 1)\n            throw new IllegalStateException(\"Too many nodes are currently DOWN to safely perform the reconfiguration\");\n    }\n\n    private static void serializeDownNodes(PrepareCMSReconfiguration transformation, DataOutputPlus out, Version version) throws IOException\n    {\n        out.writeUnsignedVInt32(transformation.downNodes.size());\n        for (NodeId nodeId : transformation.downNodes)\n            NodeId.serializer.serialize(nodeId, out, version);\n    }\n\n    private static Set<NodeId> deserializeDownNodes(DataInputPlus in, Version version) throws IOException\n    {\n        Set<NodeId> downNodes = new HashSet<>();\n        int count = in.readUnsignedVInt32();\n        for (int i = 0; i < count; i++)\n            downNodes.add(NodeId.serializer.deserialize(in, version));\n        return downNodes;\n    }\n","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/transformations/cms/PrepareCMSReconfiguration.java#L103-L139","documentation":"Thrown by PrepareCMSReconfiguration.verify() when preparing a new CMS membership (e.g. moving replicas to a new DC or RF change): after excluding DOWN nodes, the candidate new CMS group is smaller than a majority of the sum of the DC RFs. Committing would risk losing quorum for cluster metadata, so the transformation is rejected.","triggerScenarios":"Calling cms reconfigure (newReplicationParams) while enough nodes in the relevant datacenters are DOWN that prepareNewCMS() yields fewer than (expectedSize/2)+1 live nodes.","commonSituations":"Reconfiguring CMS placement during a multi-node outage; dropping RF or migrating CMS to a new DC while replicas are offline; running reconfigure immediately after several node failures.","solutions":["Bring the DOWN nodes back up before running the reconfiguration","If nodes are permanently lost, first remove them from the cluster (nodetool removenode) so the majority computation uses live members","Re-run `cms reconfigure` once a majority of the target DCs' RF nodes is available"],"exampleFix":"# before\ncms reconfigure --all-dcs  # fails while 3 of 5 CMS nodes are down\n# after\nnodetool start / repair the down nodes, then\ncms reconfigure --all-dcs","handlingStrategy":"validation","validationCode":"// before cms reconfigure, check liveness of CMS members\nMap<String,Integer> dcRf = extractRf(newReplicationParams);\nSet<NodeId> newCms = prepareNewCMS(dcRf, ClusterMetadata.current());\nint expected = dcRf.values().stream().mapToInt(Integer::intValue).sum();\nif (newCms.size() < expected / 2 + 1) { /* bring nodes up first */ }","typeGuard":null,"tryCatchPattern":"try { cms.reconfigure(params); } catch (IllegalStateException e) { if (e.getMessage().contains(\"Too many nodes are currently DOWN\")) { /* repair nodes or removenode, then retry */ } else throw e; }","preventionTips":["Check node liveness (nodetool status) before any CMS reconfiguration","Remove permanently dead nodes before reconfiguring CMS placement","Schedule reconfigurations during full cluster availability"],"tags":["cluster-metadata","quorum","reconfiguration"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}