apache/cassandra · error · IllegalArgumentException

Received unexpected response from

Error message

Received unexpected response from ${msg.from()}

What it means

During a Paxos cleanup round, PaxosCleanupComplete waits for an ack from each replica involved in the cleanup. onResponse throws IllegalArgumentException when a response arrives from a replica not in the current waitingResponse set, meaning the coordinator received a reply it never asked for (e.g. duplicate, late, or stale response).

Solutions

1. Check cluster logs for duplicate or retried PaxosCleanup messages and investigate the messaging service retry configuration
2. Restart the affected Paxos cleanup (nodetool repair / rerun the cleanup) so coordinator and replica state are in sync
3. Upgrade to a version where late/duplicate paxos cleanup responses are handled defensively rather than throwing
4. Verify no NTP/clock skew or node membership churn (decommission/replace) during cleanup

Example fix

// before
if (!waitingResponse.remove(msg.from()))
    throw new IllegalArgumentException("Received unexpected response from " + msg.from());
// after
if (!waitingResponse.remove(msg.from()))
{
    logger.warn("Ignoring unexpected paxos cleanup response from {}", msg.from());
    return;
}

// coordinator-side: ensure the request set matches replicas actually messaged
Set<InetAddressAndPort> expected = endpoints.stream().collect(Collectors.toSet());
if (!expected.equals(waitingResponse)) logger.warn("waiting set mismatch before sending cleanup requests");

try { session.onResponse(msg); } catch (IllegalArgumentException e) { logger.warn("Ignoring stray paxos cleanup response: {}", e.getMessage()); }

When it happens

Trigger: A replica sends a cleanup-complete ack twice, a response from a previous cleanup attempt arrives after a retry/timeout, or messaging-layer retries deliver extra responses after the coordinator has updated its waiting set.

Common situations: Network flaps combined with messaging retries, nodes restarted mid-cleanup causing request re-sends, or long GC pauses making a coordinator re-issue requests whose original responses arrive later.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

• Attempting to compact pending repair sstables with sstables…
• Attempting to compact transient sstables with non transient…
• Can't do any consensus migrations to/from PaxosV1, switch…
• Cannot achieve consistency level
• Cannot begin paxos auto repair for

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ea655d97940e4359. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/paxos/cleanup/PaxosCleanupComplete.java:98

        for (InetAddressAndPort endpoint : waitingResponse)
            ctx.messaging().sendWithCallback(message, endpoint, this);
    }

    @Override
    public void onFailure(InetAddressAndPort from, RequestFailure reason)
    {
        tryFailure(new PaxosCleanupException("Timed out waiting on response from " + from));
    }

    @Override
    public synchronized void onResponse(Message<Void> msg)
    {
        if (isDone())
            return;

        if (!waitingResponse.remove(msg.from()))
            throw new IllegalArgumentException("Received unexpected response from " + msg.from());

        if (waitingResponse.isEmpty())
            trySuccess(null);
    }

    public static class Request
    {
        final TableId tableId;
        final Ballot lowBound;
        final Collection<Range<Token>> ranges;

        Request(TableId tableId, Ballot lowBound, Collection<Range<Token>> ranges)
        {
            this.tableId = tableId;
            this.ranges = ranges;
            this.lowBound = lowBound;
        }
    }

View on GitHub (pinned to 88fd0f6a0e)