apache/cassandra · error · InvalidRequestException

Mismatching metadata id in survey request from

Error message

Mismatching metadata id in survey request from %s (%d)

What it means

The CMS discovery SurveyRequestHandler validates that an inbound SurveyRequest carries the same ClusterMetadata id the local node holds; a mismatch throws InvalidRequestException because responding to a survey from a different metadata generation could produce inconsistent discovery results.

Solutions

1. Restart the joining node so it reloads the current ClusterMetadata id and re-attempts discovery
2. Ensure only one node joins at a time during CMS establishment, or retry after the cluster is stable
3. Verify all nodes reference the same seed set so they converge on the same metadata id
4. If testing, note this is expected behavior (see testRequestWithMismatchingMetadataIdIsRejected) — refresh metadata before re-sending the survey

Example fix

// before: client reuses old cached metadataId
new SurveyRequest(staleMetadataId)
// after: refresh from latest fetched metadata before survey
ClusterMetadata cm = fetchLatestMetadata();
new SurveyRequest(cm.metadataId)

if (latestKnownMetadataId != responderReportedMetadataId)
    refreshMetadata(); // before sending SurveyRequest

try { messaging.send(surveyRequest); }
catch (InvalidRequestException e) {
    if (e.getMessage().startsWith("Mismatching metadata id")) {
        refreshMetadata();
        retryWithBackoff();
    } else throw e;
}

When it happens

Trigger: A joining node sends a Discovery SurveyRequest quoting a metadataId that differs from the responder's current metadataId.getAsInt() — e.g. the responder advanced metadata epochs (or booted with different metadata) between the joining node's snapshot and the survey.

Common situations: A node retrying join with stale metadata after cluster activity (writes/epoch bumps) during its bootstrap; re-joining a cluster that was rebuilt; racing surveys during simultaneous multi-node bootstrap.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Found no candidates during initialization. Check if the…
• Got mismatching cluster metadatas. Check logs on peers
• This node is not in the CMS, can't generate a consistent…
• Unable to identify a quorum of CMS members
• Addresses differ: !=

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/f516107d03686f52. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/discovery/SurveyRequestHandler.java:80

    private SurveyRequestHandler()
    {
        this(() -> ClusterMetadata.current().metadataIdentifier, MessagingService::instance);
    }

    @VisibleForTesting
    public SurveyRequestHandler(IntSupplier metadataId, Supplier<MessageDelivery> messaging)
    {
        this.metadataId = metadataId;
        this.messaging = messaging;
    }

    @Override
    public void doVerb(Message<SurveyRequest> message) throws IOException
    {
        logger.info("Responding to {} request from {}", message.verb(), message.from());
        int localMetadataId = metadataId.getAsInt();
        if (message.payload.metadataId != localMetadataId)
            throw new InvalidRequestException(String.format("Mismatching metadata id in survey request from %s (%d)",
                                                            message.from(),
                                                            message.payload.metadataId));

        Discovery.instance.discovered(message.from());
        // Respond with the node id from system.local and not ClusterMetadata.current().myNodeId() because if
        // this node is in the process of starting up with a new broadcast address, it will not yet recognise itself
        // as being in a REGISTERED state. This results in myNodeId() returning NodeId.UNREGISTERED.
        NodeId nodeId = NodeId.fromUUID(SystemKeyspace.getLocalHostId());
        InetAddressAndPort broadcastAddress = FBUtilities.getBroadcastAddressAndPort();
        SurveyResponse response = new SurveyResponse(localMetadataId, nodeId, broadcastAddress);
        messaging.get().respond(response, message);
    }
}

View on GitHub (pinned to 88fd0f6a0e)