{"record":{"id":"7257ac6833848a7c","repo":"apache/cassandra","slug":"could-not-remove-s-from-cms","errorCode":null,"errorMessage":"Could not remove %s from CMS","messagePattern":"Could not remove (.+?) from CMS","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java","lineNumber":250,"sourceCode":"    }\n\n    public static void maybeReconfigureCMS(ClusterMetadata metadata, InetAddressAndPort toRemove)\n    {\n        if (!metadata.fullCMSMembers().contains(toRemove))\n            return;\n        Set<NodeId> downNodes = new HashSet<>();\n        for (InetAddressAndPort ep : metadata.directory.allJoinedEndpoints())\n            if (!FailureDetector.instance.isAlive(ep))\n                downNodes.add(metadata.directory.peerId(ep));\n\n        PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(metadata.directory.peerId(toRemove), downNodes);\n        transformation.verify(metadata);\n        // We can force removal from the CMS as it doesn't alter the size of the service\n        ClusterMetadataService.instance().commit(transformation);\n\n        InProgressSequences.finishInProgressSequences(SequenceKey.instance);\n        if (ClusterMetadata.current().isCMSMember(toRemove))\n            throw new IllegalStateException(String.format(\"Could not remove %s from CMS\", toRemove));\n    }\n\n    private static void initiateRemoteStreaming(Replica replicaForStreaming, Set<InetAddressAndPort> streamCandidates)\n    {\n        ClusterMetadata metadata = ClusterMetadata.current();\n        EndpointsForRange.Builder efr = EndpointsForRange.builder(entireRange);\n        streamCandidates.forEach(addr -> efr.add(new Replica(addr, entireRange, true)));\n\n        MovementMap movements = MovementMap.builder().put(ReplicationParams.meta(metadata),\n                                                          new EndpointsByReplica(Collections.singletonMap(replicaForStreaming, efr.build())))\n                                           .build();\n\n        String operationId = replicaForStreaming.toString();\n        DataMovements.ResponseTracker responseTracker = DataMovements.instance.registerMovements(RESTORE_REPLICA_COUNT, operationId, movements);\n        movements.byEndpoint().forEach((ep, epMovements) -> {\n            DataMovement msg = new DataMovement(operationId, RESTORE_REPLICA_COUNT.name(), epMovements);\n            MessagingService.instance().sendWithCallback(Message.out(Verb.INITIATE_DATA_MOVEMENTS_REQ, msg), ep, response -> {\n                logger.debug(\"Endpoint {} starting streams {}\", response.from(), epMovements);","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java#L232-L268","documentation":"Thrown by ReconfigureCMS.maybeReconfigureCMS after committing a RemoveNode transformation and finishing in-progress sequences: if the removed node is still a CMS member in current cluster metadata, the removal did not take effect and the post-condition invariant is violated.","triggerScenarios":"Removing a node that is a CMS member: the forced CMS-removal transformation commits, finishInProgressSequences runs, yet ClusterMetadata.current().isCMSMember(toRemove) still returns true afterwards.","commonSituations":"Decommissioning CMS members during scale-down; buggy or partially failed removal transformations where the new CMS membership was not applied; concurrent metadata updates overwriting the reconfiguration.","solutions":["Check ClusterMetadata.current().fullCMS membership and retry the CMS reconfiguration explicitly (ReconfigureCMS with an explicit new membership)","Verify the RemoveNode transformation actually committed (check the log) before assuming success","Run the removal again / finish in-progress sequences manually and re-check membership","Collect the cluster metadata log and report if the committed transformation left the node in the CMS"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// after removal, verify before proceeding\nif (ClusterMetadata.current().isCMSMember(toRemove))\n    new ReconfigureCMS(currentCMSWithout(toRemove)).execute(ClusterMetadata.current());","typeGuard":null,"tryCatchPattern":"try { maybeReconfigureCMS(toRemove); }\ncatch (IllegalStateException e) { if (e.getMessage().contains(\"Could not remove\")) retryExplicitReconfigureCMS(toRemove); else throw e; }","preventionTips":["After forced removals, always verify CMS membership reflects the change","Prefer explicit ReconfigureCMS with the intended new membership over relying on forced removal","Monitor CMS membership during decommission workflows"],"tags":["cms","tcm","node-removal","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}