{"record":{"id":"31408e0b042b97cd","repo":"apache/cassandra","slug":"all-nodes-are-not-yet-upgraded-s-is-running-s","errorCode":null,"errorMessage":"All nodes are not yet upgraded - %s is running %s","messagePattern":"All nodes are not yet upgraded - (.+?) is running (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/ClusterMetadataService.java","lineNumber":475,"sourceCode":"            {\n                // todo; what do we do if an endpoint has a mismatching gossip-clustermetadata?\n                //       - we could add the node to --ignore and force this CM to it?\n                //       - require operator to bounce/manually fix the CM on that node\n                //       for now just requiring that any ignored host is also down\n//                if (FailureDetector.instance.isAlive(ep))\n//                    throw new IllegalStateException(\"Can't ignore \" + ep + \" during CMS migration - it is not down\");\n                logger.info(\"Endpoint {} running {} is ignored\", ep, version);\n                continue;\n            }\n\n            if (metadata.directory.peerState(entry.getKey()) == NodeState.LEFT)\n                continue;\n\n            if (!version.isUpgraded())\n            {\n                String msg = String.format(\"All nodes are not yet upgraded - %s is running %s\", metadata.directory.endpoint(entry.getKey()), version);\n                logger.error(msg);\n                throw new IllegalStateException(msg);\n            }\n        }\n\n        if (existingMembers.isEmpty())\n        {\n            logger.info(\"First CMS node\");\n            Set<InetAddressAndPort> candidates = metadata\n                                                 .directory\n                                                 .allJoinedEndpoints()\n                                                 .stream()\n                                                 .filter(ep -> !FBUtilities.getBroadcastAddressAndPort().equals(ep) &&\n                                                               !ignored.contains(ep))\n                                                 .collect(toImmutableSet());\n\n            Election.instance.nominateSelf(candidates, ignored, metadata, true);\n            ClusterMetadataService.instance().triggerSnapshot();\n        }\n        else","sourceCodeStart":457,"sourceCodeEnd":493,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/ClusterMetadataService.java#L457-L493","documentation":"During the gossip-to-TCM upgrade, every node in the directory must have registered a NodeVersion indicating it is upgraded (isUpgraded()). If any node is still running a pre-TCM (gossip-only) version, upgradeFromGossip aborts with IllegalStateException naming the offending endpoint and its version, since mixed-version clusters cannot safely switch to TCM.","triggerScenarios":"Calling upgradeFromGossip() while the rolling upgrade is incomplete: some node's directory entry (directory.versions) reports a NodeVersion with isUpgraded() == false, i.e. that node still runs an older Cassandra version using gossip.","commonSituations":"Operator starts the CMS upgrade step before finishing the binary rolling upgrade; a node that failed to upgrade is left on the old version; an intentionally-forgotten dead node still appears in the directory with an old version.","solutions":["Upgrade the named node to the TCM-capable version and let it re-register, then retry upgradeFromGossip.","Run nodetool/cluster version checks across ALL nodes before starting the gossip-to-TCM migration.","Remove or replace nodes that will never be upgraded (dead hosts) so their stale directory entries with old versions are cleared.","Re-run the upgrade procedure following the documented ordering: all nodes upgraded first, CMS upgrade last."],"exampleFix":"// before\nservice.upgradeFromGossip(ignored); // node 10.0.0.5 still on 4.x\n// after\nif (ClusterMetadata.current().directory.versions.values().stream().allMatch(NodeVersion::isUpgraded))\n    service.upgradeFromGossip(ignored);","handlingStrategy":"validation","validationCode":"boolean allUpgraded = ClusterMetadata.current().directory.versions.values().stream().allMatch(NodeVersion::isUpgraded);\nif (!allUpgraded) throw new IllegalStateException(\"Finish rolling upgrade before gossip-to-TCM migration\");","typeGuard":null,"tryCatchPattern":"try { service.upgradeFromGossip(ignored); } catch (IllegalStateException e) { logger.error(\"Cluster not fully upgraded: {}\", e.getMessage()); }","preventionTips":["Verify every node's version before starting the CMS upgrade","Remove or replace permanently dead nodes first","Follow the documented upgrade order: binaries first, CMS migration last"],"tags":["upgrade","mixed-version","cluster-metadata","prerequisite"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}