{"record":{"id":"083c11652a4c8edd","repo":"apache/cassandra","slug":"got-mismatching-cluster-metadatas-check-logs-on-p","errorCode":null,"errorMessage":"Got mismatching cluster metadatas. Check logs on peers (%s) for details of mismatches. Aborting migration.","messagePattern":"Got mismatching cluster metadatas\\. Check logs on peers \\((.+?)\\) for details of mismatches\\. Aborting migration\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/migration/Election.java","lineNumber":145,"sourceCode":"    private void initiate(CMSInitializationRequest initializationRequest, Set<InetAddressAndPort> sendTo, ClusterMetadata metadata, boolean verifyAllPeersMetadata)\n    {\n        logger.info(\"No previous migration detected, initiating\");\n        Collection<Pair<InetAddressAndPort, CMSInitializationResponse>> metadatas = MessageDelivery.fanoutAndWait(messaging, sendTo, Verb.TCM_INIT_MIG_REQ, initializationRequest);\n        if (metadatas.size() != sendTo.size())\n        {\n            Set<InetAddressAndPort> responded = metadatas.stream().map(p -> p.left).collect(Collectors.toSet());\n            String msg = String.format(\"Did not get response from %s - not continuing with migration. Ignore down hosts with --ignore <host>\", Sets.difference(sendTo, responded));\n            logger.warn(msg);\n            throw new IllegalStateException(msg);\n        }\n\n        if (verifyAllPeersMetadata)\n        {\n            Set<InetAddressAndPort> mismatching = metadatas.stream().filter(p -> !p.right.metadataMatches).map(p -> p.left).collect(Collectors.toSet());\n            if (!mismatching.isEmpty())\n            {\n                String msg = String.format(\"Got mismatching cluster metadatas. Check logs on peers (%s) for details of mismatches. Aborting migration.\", mismatching);\n                throw new IllegalStateException(msg);\n            }\n        }\n    }\n\n    private void finish(Set<InetAddressAndPort> sendTo)\n    {\n        CMSInitializationRequest.Initiator currentInitiator = initiator.get();\n        if (currentInitiator != null &&\n            Objects.equals(currentInitiator.endpoint, FBUtilities.getBroadcastAddressAndPort()) &&\n            initiator.compareAndSet(currentInitiator, MIGRATING))\n        {\n            Startup.initializeAsFirstCMSNode();\n            updateInitiator(MIGRATING, MIGRATED);\n            MessageDelivery.fanoutAndWait(messaging, sendTo, Verb.TCM_NOTIFY_REQ, DistributedMetadataLogKeyspace.getLogState(Epoch.EMPTY, false));\n        }\n        else\n        {\n            throw new IllegalStateException(\"Can't finish migration, initiator=\"+currentInitiator);","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/migration/Election.java#L127-L163","documentation":"When verifyAllPeersMetadata is enabled, each peer responding to CMSInitializationRequest reports whether its ClusterMetadata matches the initiator's (metadataMatches). If any peer reports a mismatch, initiate() throws IllegalStateException listing the mismatching endpoints, aborting the migration to prevent electing a CMS with divergent cluster state.","triggerScenarios":"Running the CMS migration/initialize command with metadata verification on while peers have divergent ClusterMetadata (different log state, schema, tokens, or directory).","commonSituations":"A node that was down during earlier schema/token changes; partitioned nodes that missed metadata updates; restored-from-backup nodes with stale metadata; inconsistent gossip-era state across the ring.","solutions":["Inspect the logs on the mismatching peers (IPs are listed in the message) for the specific mismatch details","Bring lagging nodes back into sync (repair schema, restart to re-fetch log state) and retry migration","Compare cluster_metadata log state across nodes; catch up any node whose log is behind","If a peer is permanently unsalvageable, decommission/remove it from candidacy and retry"],"exampleFix":"// before\nnodetool cms initialize --verify-all-peers-metadata true  // aborts\n// after\n# fix node 10.0.0.5 metadata per its logs, then\nnodetool cms initialize --verify-all-peers-metadata true","handlingStrategy":"fallback","validationCode":"// compare log state epochs across nodes before migrating\nEpoch local = ClusterMetadata.current().epoch; // peers report metadataMatches in CMSInitializationResponse","typeGuard":null,"tryCatchPattern":"try { initiate(..., true); } catch (IllegalStateException e) { // read mismatching endpoints from the message, inspect their logs, resync, retry }","preventionTips":["Keep all nodes in the same schema/log state before migrating (catch up down nodes first)","Monitor for nodes that missed recent metadata changes","Avoid restoring nodes from stale backups into the candidate set"],"tags":["cassandra","tcm","consistency"],"backgroundTag":"checksum-mismatch","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}