apache/cassandra · error · IllegalStateException

Could not remove from CMS

Error message

Could not remove %s from CMS

What it means

Thrown by ReconfigureCMS.maybeReconfigureCMS after committing a RemoveNode transformation and finishing in-progress sequences: if the removed node is still a CMS member in current cluster metadata, the removal did not take effect and the post-condition invariant is violated.

Solutions

1. Check ClusterMetadata.current().fullCMS membership and retry the CMS reconfiguration explicitly (ReconfigureCMS with an explicit new membership)
2. Verify the RemoveNode transformation actually committed (check the log) before assuming success
3. Run the removal again / finish in-progress sequences manually and re-check membership
4. Collect the cluster metadata log and report if the committed transformation left the node in the CMS

// after removal, verify before proceeding
if (ClusterMetadata.current().isCMSMember(toRemove))
    new ReconfigureCMS(currentCMSWithout(toRemove)).execute(ClusterMetadata.current());

try { maybeReconfigureCMS(toRemove); }
catch (IllegalStateException e) { if (e.getMessage().contains("Could not remove")) retryExplicitReconfigureCMS(toRemove); else throw e; }

When it happens

Trigger: Removing a node that is a CMS member: the forced CMS-removal transformation commits, finishInProgressSequences runs, yet ClusterMetadata.current().isCMSMember(toRemove) still returns true afterwards.

Common situations: Decommissioning CMS members during scale-down; buggy or partially failed removal transformations where the new CMS membership was not applied; concurrent metadata updates overwriting the reconfiguration.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Bad CMS state:
• Can not advance in-progress sequence, since its kind is
• Can not apply in-progress sequence, since its kind is
• Could not catch up to epoch
• Could not initialize CMS lookup

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/7257ac6833848a7c. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java:250

    }

    public static void maybeReconfigureCMS(ClusterMetadata metadata, InetAddressAndPort toRemove)
    {
        if (!metadata.fullCMSMembers().contains(toRemove))
            return;
        Set<NodeId> downNodes = new HashSet<>();
        for (InetAddressAndPort ep : metadata.directory.allJoinedEndpoints())
            if (!FailureDetector.instance.isAlive(ep))
                downNodes.add(metadata.directory.peerId(ep));

        PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(metadata.directory.peerId(toRemove), downNodes);
        transformation.verify(metadata);
        // We can force removal from the CMS as it doesn't alter the size of the service
        ClusterMetadataService.instance().commit(transformation);

        InProgressSequences.finishInProgressSequences(SequenceKey.instance);
        if (ClusterMetadata.current().isCMSMember(toRemove))
            throw new IllegalStateException(String.format("Could not remove %s from CMS", toRemove));
    }

    private static void initiateRemoteStreaming(Replica replicaForStreaming, Set<InetAddressAndPort> streamCandidates)
    {
        ClusterMetadata metadata = ClusterMetadata.current();
        EndpointsForRange.Builder efr = EndpointsForRange.builder(entireRange);
        streamCandidates.forEach(addr -> efr.add(new Replica(addr, entireRange, true)));

        MovementMap movements = MovementMap.builder().put(ReplicationParams.meta(metadata),
                                                          new EndpointsByReplica(Collections.singletonMap(replicaForStreaming, efr.build())))
                                           .build();

        String operationId = replicaForStreaming.toString();
        DataMovements.ResponseTracker responseTracker = DataMovements.instance.registerMovements(RESTORE_REPLICA_COUNT, operationId, movements);
        movements.byEndpoint().forEach((ep, epMovements) -> {
            DataMovement msg = new DataMovement(operationId, RESTORE_REPLICA_COUNT.name(), epMovements);
            MessagingService.instance().sendWithCallback(Message.out(Verb.INITIATE_DATA_MOVEMENTS_REQ, msg), ep, response -> {
                logger.debug("Endpoint {} starting streams {}", response.from(), epMovements);

View on GitHub (pinned to 88fd0f6a0e)