apache/cassandra · error · IllegalStateException

Did not get response from

Error message

Did not get response from %s - not continuing with migration. Ignore down hosts with --ignore <host>

What it means

During CMS migration, Election.initiate fans out a TCM_INIT_MIG_REQ to all candidate peers (minus ignored ones) and waits for responses. If any candidate did not answer, migration aborts with an IllegalStateException listing the unresponsive hosts, because all peers must be reachable to safely migrate to TCM.

Solutions

Restart the down peers, then retry the migration command
Re-run with the unreachable hosts passed via the --ignore flag (they are removed from sendTo)
Verify messaging connectivity (incl. TLS settings) between the initiator and all candidates
Retry the command; transient GC pauses or slow nodes may respond on a second attempt

Example fix

// before
nodetool cms initiate   // fails because 10.0.0.5 is down
// after
nodetool cms initiate --ignore 10.0.0.5

Defensive patterns

Strategy: retry

Validate before calling

// pre-check peer reachability before initiating
sendTo.forEach(ep -> { if (!messaging.isConnected(ep)) throw new IllegalStateException("Peer down: " + ep); });

Try / catch

try { initiate(req, sendTo, metadata, verify); } catch (IllegalStateException e) { // retry, possibly with the unresponsive hosts added to --ignore }

Prevention

Verify all candidate nodes are up (nodetool status) before starting migration
Pre-decommission dead nodes or list them in --ignore
Check firewall/TLS connectivity on the TCM messaging port
Retry once for transient GC/network pauses

When it happens

Trigger: Calling nominateSelf when at least one non-ignored candidate node is down, partitioned, or too slow to answer Verb.TCM_INIT_MIG_REQ before the fanout timeout.

Common situations: Nodes that were decommissioned but remain in the candidate list; network/firewall issues blocking the TCM messaging port; a node overloaded or in GC making it miss the response window.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/f1a6fe964817914f. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/migration/Election.java:136

            Keyspaces.KeyspacesDiff diff = Keyspaces.diff(currentState.schema.getKeyspaces(), priorState.schema.getKeyspaces());
            Collection<Mutation> mutations = SchemaKeyspace.convertSchemaDiffToMutations(diff, FBUtilities.timestampMicros());
            SchemaKeyspace.applyChanges(mutations);

            ClusterMetadataService.instance().log().unsafeSetCommittedFromGossip(priorState);
            throw e;
        }
    }

    private void initiate(CMSInitializationRequest initializationRequest, Set<InetAddressAndPort> sendTo, ClusterMetadata metadata, boolean verifyAllPeersMetadata)
    {
        logger.info("No previous migration detected, initiating");
        Collection<Pair<InetAddressAndPort, CMSInitializationResponse>> metadatas = MessageDelivery.fanoutAndWait(messaging, sendTo, Verb.TCM_INIT_MIG_REQ, initializationRequest);
        if (metadatas.size() != sendTo.size())
        {
            Set<InetAddressAndPort> responded = metadatas.stream().map(p -> p.left).collect(Collectors.toSet());
            String msg = String.format("Did not get response from %s - not continuing with migration. Ignore down hosts with --ignore <host>", Sets.difference(sendTo, responded));
            logger.warn(msg);
            throw new IllegalStateException(msg);
        }

        if (verifyAllPeersMetadata)
        {
            Set<InetAddressAndPort> mismatching = metadatas.stream().filter(p -> !p.right.metadataMatches).map(p -> p.left).collect(Collectors.toSet());
            if (!mismatching.isEmpty())
            {
                String msg = String.format("Got mismatching cluster metadatas. Check logs on peers (%s) for details of mismatches. Aborting migration.", mismatching);
                throw new IllegalStateException(msg);
            }
        }
    }

    private void finish(Set<InetAddressAndPort> sendTo)
    {
        CMSInitializationRequest.Initiator currentInitiator = initiator.get();
        if (currentInitiator != null &&
            Objects.equals(currentInitiator.endpoint, FBUtilities.getBroadcastAddressAndPort()) &&

View on GitHub (pinned to 88fd0f6a0e)