apache/cassandra · error · IllegalStateException

Could not remove node ${nodeIpOrId} from CMS.

Error message

Could not remove node ${nodeIpOrId} from CMS.

What it means

In maybeReconfigureCMS the tool applies a PrepareCMSReconfiguration.Simple transformation and the in-progress ReconfigureCMS sequence, then asserts the target node is no longer a CMS member. If it still is, the reconfiguration silently did not take effect, so it throws this IllegalStateException to signal the invariant was violated.

Source

Thrown at src/java/org/apache/cassandra/tools/CMSOfflineTool.java:402

            writeMetadata(output, updatedMetadata, outputFilePath);
        }

        ClusterMetadata maybeReconfigureCMS(ClusterMetadata metadata, NodeId nodeId)
        {
            InetAddressAndPort addressAndPort = metadata.directory.endpoint(nodeId);
            if (!metadata.isCMSMember(addressAndPort))
            {
                return metadata;
            }

            // Ref: org.apache.cassandra.tcm.sequences.ReconfigureCMS.maybeReconfigureCMS
            PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, Set.of());
            ClusterMetadata updatedMetadata = transformation.execute(metadata).success().metadata;
            updatedMetadata = updatedMetadata.inProgressSequences.get(ReconfigureCMS.SequenceKey.instance)
                                                                 .applyTo(updatedMetadata).success().metadata;
            if (updatedMetadata.isCMSMember(addressAndPort))
            {
                throw new IllegalStateException("Could not remove node " + nodeIdentifierOption.getNodeIpOrId() +
                                                " from CMS.");
            }
            return updatedMetadata;
        }
    }

    /**
     * Replaces the entire CMS membership with a single node. All existing CMS replicas are
     * removed and the specified node becomes the sole CMS member. Useful for recovering from
     * a state where the CMS is unreachable.
     */
    @Command(name = "resetcms",
    description = "Replaces all CMS members with the specified node. WARNING: all existing CMS replicas are removed.")
    static class ResetCMS extends ClusterMetadataToolCmd
    {
        @CommandLine.ArgGroup(exclusive = true, multiplicity = "1")
        NodeIdentifierOption nodeIdentifierOption;

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Re-run the tool against a fresh ClusterMetadata snapshot.
2. Inspect the ReconfigureCMS in-progress sequence state and advance/cancel it before reconfiguring CMS membership.
3. Report/verify whether the transformation's members set actually excluded the node (Set.of() means remove all but empty placement); fix the parameters passed to PrepareCMSReconfiguration.Simple.

Example fix

// before
PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, Set.of());
// after
// ensure the new CMS member set intentionally excludes the node and includes enough replicas
PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, newCmsMembers);

try { tool.reconfigureCms(nodeId); } catch (IllegalStateException e) { if (e.getMessage().startsWith("Could not remove node")) { reloadFreshMetadataAndRetry(); } else throw e; }

When it happens

Trigger: Calling maybeReconfigureCMS from `execute` when, after applying the reconfiguration transformations to ClusterMetadata, updatedMetadata.isCMSMember(addressAndPort) still returns true.

Common situations: Metadata snapshot is stale or inconsistent so the applied transformation did not remove the member; a bug or unexpected sequence state in ReconfigureCMS.applyTo leaves the membership unchanged.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Too many nodes are currently DOWN to safely perform the reco
• Node ${nodeIpOrId} is in ${nodeState} state. Only a JOINED n
• CMS Membership differ: {} != {}
• Attempting to compact pending repair sstables with sstables
• Attempting to compact transient sstables with non transient

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/c072eeb58fdc23eb. Report an issue: GitHub.