{"record":{"id":"c072eeb58fdc23eb","repo":"apache/cassandra","slug":"could-not-remove-node-nodeiporid-from-cms","errorCode":null,"errorMessage":"Could not remove node ${nodeIpOrId} from CMS.","messagePattern":"Could not remove node (.+?) from CMS\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tools/CMSOfflineTool.java","lineNumber":402,"sourceCode":"            writeMetadata(output, updatedMetadata, outputFilePath);\n        }\n\n        ClusterMetadata maybeReconfigureCMS(ClusterMetadata metadata, NodeId nodeId)\n        {\n            InetAddressAndPort addressAndPort = metadata.directory.endpoint(nodeId);\n            if (!metadata.isCMSMember(addressAndPort))\n            {\n                return metadata;\n            }\n\n            // Ref: org.apache.cassandra.tcm.sequences.ReconfigureCMS.maybeReconfigureCMS\n            PrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, Set.of());\n            ClusterMetadata updatedMetadata = transformation.execute(metadata).success().metadata;\n            updatedMetadata = updatedMetadata.inProgressSequences.get(ReconfigureCMS.SequenceKey.instance)\n                                                                 .applyTo(updatedMetadata).success().metadata;\n            if (updatedMetadata.isCMSMember(addressAndPort))\n            {\n                throw new IllegalStateException(\"Could not remove node \" + nodeIdentifierOption.getNodeIpOrId() +\n                                                \" from CMS.\");\n            }\n            return updatedMetadata;\n        }\n    }\n\n    /**\n     * Replaces the entire CMS membership with a single node. All existing CMS replicas are\n     * removed and the specified node becomes the sole CMS member. Useful for recovering from\n     * a state where the CMS is unreachable.\n     */\n    @Command(name = \"resetcms\",\n    description = \"Replaces all CMS members with the specified node. WARNING: all existing CMS replicas are removed.\")\n    static class ResetCMS extends ClusterMetadataToolCmd\n    {\n        @CommandLine.ArgGroup(exclusive = true, multiplicity = \"1\")\n        NodeIdentifierOption nodeIdentifierOption;\n","sourceCodeStart":384,"sourceCodeEnd":420,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tools/CMSOfflineTool.java#L384-L420","documentation":"In maybeReconfigureCMS the tool applies a PrepareCMSReconfiguration.Simple transformation and the in-progress ReconfigureCMS sequence, then asserts the target node is no longer a CMS member. If it still is, the reconfiguration silently did not take effect, so it throws this IllegalStateException to signal the invariant was violated.","triggerScenarios":"Calling maybeReconfigureCMS from `execute` when, after applying the reconfiguration transformations to ClusterMetadata, updatedMetadata.isCMSMember(addressAndPort) still returns true.","commonSituations":"Metadata snapshot is stale or inconsistent so the applied transformation did not remove the member; a bug or unexpected sequence state in ReconfigureCMS.applyTo leaves the membership unchanged.","solutions":["Re-run the tool against a fresh ClusterMetadata snapshot.","Inspect the ReconfigureCMS in-progress sequence state and advance/cancel it before reconfiguring CMS membership.","Report/verify whether the transformation's members set actually excluded the node (Set.of() means remove all but empty placement); fix the parameters passed to PrepareCMSReconfiguration.Simple."],"exampleFix":"// before\nPrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, Set.of());\n// after\n// ensure the new CMS member set intentionally excludes the node and includes enough replicas\nPrepareCMSReconfiguration.Simple transformation = new PrepareCMSReconfiguration.Simple(nodeId, newCmsMembers);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { tool.reconfigureCms(nodeId); } catch (IllegalStateException e) { if (e.getMessage().startsWith(\"Could not remove node\")) { reloadFreshMetadataAndRetry(); } else throw e; }","preventionTips":["Verify the resulting CMS membership set after reconfiguration before proceeding.","Work from a freshly captured ClusterMetadata snapshot.","Ensure the transformation's target member set explicitly excludes the node."],"tags":["cms-membership","reconfiguration","internal-invariant","cluster-metadata"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}