{"record":{"id":"f516107d03686f52","repo":"apache/cassandra","slug":"mismatching-metadata-id-in-survey-request-from-s","errorCode":null,"errorMessage":"Mismatching metadata id in survey request from %s (%d)","messagePattern":"Mismatching metadata id in survey request from (.+?) \\((.+?)\\)","errorType":"exception","errorClass":"InvalidRequestException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/discovery/SurveyRequestHandler.java","lineNumber":80,"sourceCode":"    private SurveyRequestHandler()\n    {\n        this(() -> ClusterMetadata.current().metadataIdentifier, MessagingService::instance);\n    }\n\n    @VisibleForTesting\n    public SurveyRequestHandler(IntSupplier metadataId, Supplier<MessageDelivery> messaging)\n    {\n        this.metadataId = metadataId;\n        this.messaging = messaging;\n    }\n\n    @Override\n    public void doVerb(Message<SurveyRequest> message) throws IOException\n    {\n        logger.info(\"Responding to {} request from {}\", message.verb(), message.from());\n        int localMetadataId = metadataId.getAsInt();\n        if (message.payload.metadataId != localMetadataId)\n            throw new InvalidRequestException(String.format(\"Mismatching metadata id in survey request from %s (%d)\",\n                                                            message.from(),\n                                                            message.payload.metadataId));\n\n        Discovery.instance.discovered(message.from());\n        // Respond with the node id from system.local and not ClusterMetadata.current().myNodeId() because if\n        // this node is in the process of starting up with a new broadcast address, it will not yet recognise itself\n        // as being in a REGISTERED state. This results in myNodeId() returning NodeId.UNREGISTERED.\n        NodeId nodeId = NodeId.fromUUID(SystemKeyspace.getLocalHostId());\n        InetAddressAndPort broadcastAddress = FBUtilities.getBroadcastAddressAndPort();\n        SurveyResponse response = new SurveyResponse(localMetadataId, nodeId, broadcastAddress);\n        messaging.get().respond(response, message);\n    }\n}\n","sourceCodeStart":62,"sourceCodeEnd":94,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/discovery/SurveyRequestHandler.java#L62-L94","documentation":"The CMS discovery SurveyRequestHandler validates that an inbound SurveyRequest carries the same ClusterMetadata id the local node holds; a mismatch throws InvalidRequestException because responding to a survey from a different metadata generation could produce inconsistent discovery results.","triggerScenarios":"A joining node sends a Discovery SurveyRequest quoting a metadataId that differs from the responder's current metadataId.getAsInt() — e.g. the responder advanced metadata epochs (or booted with different metadata) between the joining node's snapshot and the survey.","commonSituations":"A node retrying join with stale metadata after cluster activity (writes/epoch bumps) during its bootstrap; re-joining a cluster that was rebuilt; racing surveys during simultaneous multi-node bootstrap.","solutions":["Restart the joining node so it reloads the current ClusterMetadata id and re-attempts discovery","Ensure only one node joins at a time during CMS establishment, or retry after the cluster is stable","Verify all nodes reference the same seed set so they converge on the same metadata id","If testing, note this is expected behavior (see testRequestWithMismatchingMetadataIdIsRejected) — refresh metadata before re-sending the survey"],"exampleFix":"// before: client reuses old cached metadataId\nnew SurveyRequest(staleMetadataId)\n// after: refresh from latest fetched metadata before survey\nClusterMetadata cm = fetchLatestMetadata();\nnew SurveyRequest(cm.metadataId)","handlingStrategy":"retry","validationCode":"if (latestKnownMetadataId != responderReportedMetadataId)\n    refreshMetadata(); // before sending SurveyRequest","typeGuard":null,"tryCatchPattern":"try { messaging.send(surveyRequest); }\ncatch (InvalidRequestException e) {\n    if (e.getMessage().startsWith(\"Mismatching metadata id\")) {\n        refreshMetadata();\n        retryWithBackoff();\n    } else throw e;\n}","preventionTips":["Refresh ClusterMetadata before each discovery attempt","Avoid concurrent bootstrap of many nodes during CMS establishment","Keep seed configuration identical across all nodes so metadata ids converge"],"tags":["tcm","discovery","consistency","internode"],"backgroundTag":"checksum-mismatch","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}