apache/cassandra · error · IllegalStateException

Got mismatching cluster metadatas. Check logs on peers

Error message

Got mismatching cluster metadatas. Check logs on peers (%s) for details of mismatches. Aborting migration.

What it means

When verifyAllPeersMetadata is enabled, each peer responding to CMSInitializationRequest reports whether its ClusterMetadata matches the initiator's (metadataMatches). If any peer reports a mismatch, initiate() throws IllegalStateException listing the mismatching endpoints, aborting the migration to prevent electing a CMS with divergent cluster state.

Solutions

1. Inspect the logs on the mismatching peers (IPs are listed in the message) for the specific mismatch details
2. Bring lagging nodes back into sync (repair schema, restart to re-fetch log state) and retry migration
3. Compare cluster_metadata log state across nodes; catch up any node whose log is behind
4. If a peer is permanently unsalvageable, decommission/remove it from candidacy and retry

Example fix

// before
nodetool cms initialize --verify-all-peers-metadata true  // aborts
// after
# fix node 10.0.0.5 metadata per its logs, then
nodetool cms initialize --verify-all-peers-metadata true

// compare log state epochs across nodes before migrating
Epoch local = ClusterMetadata.current().epoch; // peers report metadataMatches in CMSInitializationResponse

try { initiate(..., true); } catch (IllegalStateException e) { // read mismatching endpoints from the message, inspect their logs, resync, retry }

When it happens

Trigger: Running the CMS migration/initialize command with metadata verification on while peers have divergent ClusterMetadata (different log state, schema, tokens, or directory).

Common situations: A node that was down during earlier schema/token changes; partitioned nodes that missed metadata updates; restored-from-backup nodes with stale metadata; inconsistent gossip-era state across the ring.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Addresses differ: !=
• Can only initialize cluster identifier during epoch
• Can't commit transformations when running in gossip mode…
• Can't finish migration, initiator=
• Can't ignore local host

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/083c11652a4c8edd. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/migration/Election.java:145

    private void initiate(CMSInitializationRequest initializationRequest, Set<InetAddressAndPort> sendTo, ClusterMetadata metadata, boolean verifyAllPeersMetadata)
    {
        logger.info("No previous migration detected, initiating");
        Collection<Pair<InetAddressAndPort, CMSInitializationResponse>> metadatas = MessageDelivery.fanoutAndWait(messaging, sendTo, Verb.TCM_INIT_MIG_REQ, initializationRequest);
        if (metadatas.size() != sendTo.size())
        {
            Set<InetAddressAndPort> responded = metadatas.stream().map(p -> p.left).collect(Collectors.toSet());
            String msg = String.format("Did not get response from %s - not continuing with migration. Ignore down hosts with --ignore <host>", Sets.difference(sendTo, responded));
            logger.warn(msg);
            throw new IllegalStateException(msg);
        }

        if (verifyAllPeersMetadata)
        {
            Set<InetAddressAndPort> mismatching = metadatas.stream().filter(p -> !p.right.metadataMatches).map(p -> p.left).collect(Collectors.toSet());
            if (!mismatching.isEmpty())
            {
                String msg = String.format("Got mismatching cluster metadatas. Check logs on peers (%s) for details of mismatches. Aborting migration.", mismatching);
                throw new IllegalStateException(msg);
            }
        }
    }

    private void finish(Set<InetAddressAndPort> sendTo)
    {
        CMSInitializationRequest.Initiator currentInitiator = initiator.get();
        if (currentInitiator != null &&
            Objects.equals(currentInitiator.endpoint, FBUtilities.getBroadcastAddressAndPort()) &&
            initiator.compareAndSet(currentInitiator, MIGRATING))
        {
            Startup.initializeAsFirstCMSNode();
            updateInitiator(MIGRATING, MIGRATED);
            MessageDelivery.fanoutAndWait(messaging, sendTo, Verb.TCM_NOTIFY_REQ, DistributedMetadataLogKeyspace.getLogState(Epoch.EMPTY, false));
        }
        else
        {
            throw new IllegalStateException("Can't finish migration, initiator="+currentInitiator);

View on GitHub (pinned to 88fd0f6a0e)