apache/cassandra · error · IllegalStateException

Initial CMS node needs to be fully joined, not: %s

Error message

Initial CMS node needs to be fully joined, not: %s

What it means

During a gossip-to-TCM upgrade, the node attempting to become the initial CMS (Cluster Metadata Service) must itself be fully JOINED in the current ClusterMetadata. If its node state is anything else (e.g. BOOTSTRAPPING, LEAVING, CANDIDATE), upgradeFromGossip logs the message and throws IllegalStateException because a non-joined node cannot safely seed the metadata cluster.

Source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:439

        return ClusterMetadata.current().isCMSMember(peer);
    }

    public void upgradeFromGossip(List<String> ignoredEndpoints)
    {
        Set<InetAddressAndPort> ignored = ignoredEndpoints.stream().map(InetAddressAndPort::getByNameUnchecked).collect(toSet());
        if (ignored.contains(FBUtilities.getBroadcastAddressAndPort()))
        {
            String msg = String.format("Can't ignore local host %s when doing CMS migration", FBUtilities.getBroadcastAddressAndPort());
            logger.error(msg);
            throw new IllegalStateException(msg);
        }

        ClusterMetadata metadata = metadata();
        if (metadata.myNodeState() != NodeState.JOINED)
        {
            String msg = String.format("Initial CMS node needs to be fully joined, not: %s", metadata.myNodeState());
            logger.error(msg);
            throw new IllegalStateException(msg);
        }

        Set<InetAddressAndPort> existingMembers = metadata.fullCMSMembers();

        if (!metadata.directory.allAddresses().containsAll(ignored))
        {
            Set<InetAddressAndPort> allAddresses = Sets.newHashSet(metadata.directory.allAddresses());
            String msg = String.format("Ignored host(s) %s don't exist in the cluster", Sets.difference(ignored, allAddresses));
            logger.error(msg);
            throw new IllegalStateException(msg);
        }

        for (Map.Entry<NodeId, NodeVersion> entry : metadata.directory.versions.entrySet())
        {
            NodeVersion version = entry.getValue();
            InetAddressAndPort ep = metadata.directory.getNodeAddresses(entry.getKey()).broadcastAddress;
            if (ignored.contains(ep))
            {

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Pick a node whose NodeState is JOINED to run the upgrade from gossip (verify with nodetool status / ClusterMetadata.current().myNodeState()).
2. Complete or roll back any in-flight bootstrap/decommission operation on the initiating node before upgrading.
3. If node state is stuck, repair the node state (finish bootstrap with proper RF or remove/replace the node) and retry upgradeFromGossip.
4. Ensure the cluster upgrade procedure starts after ALL nodes have joined and the cluster is stable.

Example fix

// before
clusterMetadataService.upgradeFromGossip(ignoredSet); // called on a BOOTSTRAPPING node
// after
if (ClusterMetadata.current().myNodeState() == NodeState.JOINED)
    clusterMetadataService.upgradeFromGossip(ignoredSet);
else
    throw new IllegalStateException("Run upgradeFromGossip on a fully joined node");

if (ClusterMetadata.current().myNodeState() != NodeState.JOINED) throw new IllegalStateException("Node not JOINED; cannot upgrade from gossip");

boolean canUpgradeFromGossip = ClusterMetadata.current().myNodeState() == NodeState.JOINED;

When it happens

Trigger: Calling ClusterMetadataService.upgradeFromGossip() while metadata.myNodeState() != NodeState.JOINED — e.g. the initiating node is still bootstrapping, decommissioning/leaving, or has a partially completed state transition.

Common situations: Operator runs the CMS upgrade step on the wrong node (one still mid-bootstrap), a previous bootstrap/decommission was interrupted leaving stale node state, or the upgrade is triggered before the node has fully joined the ring during a rolling upgrade from pre-TCM versions.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can't upgrade from gossip since CMS is already initialized
• Ignored host(s) %s don't exist in the cluster
• All nodes are not yet upgraded - %s is running %s
• Tried to commit when in gossip mode
• Illegal state:

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/fd1ce8793eeffccd. Report an issue: GitHub.