{"record":{"id":"ea655d97940e4359","repo":"apache/cassandra","slug":"received-unexpected-response-from-msg-from","errorCode":null,"errorMessage":"Received unexpected response from ${msg.from()}","messagePattern":"Received unexpected response from (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/paxos/cleanup/PaxosCleanupComplete.java","lineNumber":98,"sourceCode":"\n        for (InetAddressAndPort endpoint : waitingResponse)\n            ctx.messaging().sendWithCallback(message, endpoint, this);\n    }\n\n    @Override\n    public void onFailure(InetAddressAndPort from, RequestFailure reason)\n    {\n        tryFailure(new PaxosCleanupException(\"Timed out waiting on response from \" + from));\n    }\n\n    @Override\n    public synchronized void onResponse(Message<Void> msg)\n    {\n        if (isDone())\n            return;\n\n        if (!waitingResponse.remove(msg.from()))\n            throw new IllegalArgumentException(\"Received unexpected response from \" + msg.from());\n\n        if (waitingResponse.isEmpty())\n            trySuccess(null);\n    }\n\n    public static class Request\n    {\n        final TableId tableId;\n        final Ballot lowBound;\n        final Collection<Range<Token>> ranges;\n\n        Request(TableId tableId, Ballot lowBound, Collection<Range<Token>> ranges)\n        {\n            this.tableId = tableId;\n            this.ranges = ranges;\n            this.lowBound = lowBound;\n        }\n    }","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/paxos/cleanup/PaxosCleanupComplete.java#L80-L116","documentation":"During a Paxos cleanup round, PaxosCleanupComplete waits for an ack from each replica involved in the cleanup. onResponse throws IllegalArgumentException when a response arrives from a replica not in the current waitingResponse set, meaning the coordinator received a reply it never asked for (e.g. duplicate, late, or stale response).","triggerScenarios":"A replica sends a cleanup-complete ack twice, a response from a previous cleanup attempt arrives after a retry/timeout, or messaging-layer retries deliver extra responses after the coordinator has updated its waiting set.","commonSituations":"Network flaps combined with messaging retries, nodes restarted mid-cleanup causing request re-sends, or long GC pauses making a coordinator re-issue requests whose original responses arrive later.","solutions":["Check cluster logs for duplicate or retried PaxosCleanup messages and investigate the messaging service retry configuration","Restart the affected Paxos cleanup (nodetool repair / rerun the cleanup) so coordinator and replica state are in sync","Upgrade to a version where late/duplicate paxos cleanup responses are handled defensively rather than throwing","Verify no NTP/clock skew or node membership churn (decommission/replace) during cleanup"],"exampleFix":"// before\nif (!waitingResponse.remove(msg.from()))\n    throw new IllegalArgumentException(\"Received unexpected response from \" + msg.from());\n// after\nif (!waitingResponse.remove(msg.from()))\n{\n    logger.warn(\"Ignoring unexpected paxos cleanup response from {}\", msg.from());\n    return;\n}","handlingStrategy":"try-catch","validationCode":"// coordinator-side: ensure the request set matches replicas actually messaged\nSet<InetAddressAndPort> expected = endpoints.stream().collect(Collectors.toSet());\nif (!expected.equals(waitingResponse)) logger.warn(\"waiting set mismatch before sending cleanup requests\");","typeGuard":null,"tryCatchPattern":"try { session.onResponse(msg); } catch (IllegalArgumentException e) { logger.warn(\"Ignoring stray paxos cleanup response: {}\", e.getMessage()); }","preventionTips":["Avoid topology changes during paxos repair/cleanup","Keep messaging retry settings conservative to limit duplicate responses","Monitor GC pauses that inflate response latency and cause coordinator retries","Keep all nodes on the same patched Cassandra version"],"tags":["paxos","internal-invariant","distributed-coordination"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}