apache/cassandra · error · IllegalStateException

All nodes are not yet upgraded - %s is running %s

Error message

All nodes are not yet upgraded - %s is running %s

What it means

During the gossip-to-TCM upgrade, every node in the directory must have registered a NodeVersion indicating it is upgraded (isUpgraded()). If any node is still running a pre-TCM (gossip-only) version, upgradeFromGossip aborts with IllegalStateException naming the offending endpoint and its version, since mixed-version clusters cannot safely switch to TCM.

Source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:475

            {
                // todo; what do we do if an endpoint has a mismatching gossip-clustermetadata?
                //       - we could add the node to --ignore and force this CM to it?
                //       - require operator to bounce/manually fix the CM on that node
                //       for now just requiring that any ignored host is also down
//                if (FailureDetector.instance.isAlive(ep))
//                    throw new IllegalStateException("Can't ignore " + ep + " during CMS migration - it is not down");
                logger.info("Endpoint {} running {} is ignored", ep, version);
                continue;
            }

            if (metadata.directory.peerState(entry.getKey()) == NodeState.LEFT)
                continue;

            if (!version.isUpgraded())
            {
                String msg = String.format("All nodes are not yet upgraded - %s is running %s", metadata.directory.endpoint(entry.getKey()), version);
                logger.error(msg);
                throw new IllegalStateException(msg);
            }
        }

        if (existingMembers.isEmpty())
        {
            logger.info("First CMS node");
            Set<InetAddressAndPort> candidates = metadata
                                                 .directory
                                                 .allJoinedEndpoints()
                                                 .stream()
                                                 .filter(ep -> !FBUtilities.getBroadcastAddressAndPort().equals(ep) &&
                                                               !ignored.contains(ep))
                                                 .collect(toImmutableSet());

            Election.instance.nominateSelf(candidates, ignored, metadata, true);
            ClusterMetadataService.instance().triggerSnapshot();
        }
        else

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Upgrade the named node to the TCM-capable version and let it re-register, then retry upgradeFromGossip.
2. Run nodetool/cluster version checks across ALL nodes before starting the gossip-to-TCM migration.
3. Remove or replace nodes that will never be upgraded (dead hosts) so their stale directory entries with old versions are cleared.
4. Re-run the upgrade procedure following the documented ordering: all nodes upgraded first, CMS upgrade last.

Example fix

// before
service.upgradeFromGossip(ignored); // node 10.0.0.5 still on 4.x
// after
if (ClusterMetadata.current().directory.versions.values().stream().allMatch(NodeVersion::isUpgraded))
    service.upgradeFromGossip(ignored);

boolean allUpgraded = ClusterMetadata.current().directory.versions.values().stream().allMatch(NodeVersion::isUpgraded);
if (!allUpgraded) throw new IllegalStateException("Finish rolling upgrade before gossip-to-TCM migration");

try { service.upgradeFromGossip(ignored); } catch (IllegalStateException e) { logger.error("Cluster not fully upgraded: {}", e.getMessage()); }

When it happens

Trigger: Calling upgradeFromGossip() while the rolling upgrade is incomplete: some node's directory entry (directory.versions) reports a NodeVersion with isUpgraded() == false, i.e. that node still runs an older Cassandra version using gossip.

Common situations: Operator starts the CMS upgrade step before finishing the binary rolling upgrade; a node that failed to upgrade is left on the old version; an intentionally-forgotten dead node still appears in the directory with an old version.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Initial CMS node needs to be fully joined, not: %s
• Ignored host(s) %s don't exist in the cluster
• Can't upgrade from gossip since CMS is already initialized
• HostId read from local system table does not match the one r
• Schema pull request from {} ignored - please upgrade

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/31408e0b042b97cd. Report an issue: GitHub.