apache/cassandra · error · IllegalStateException

Can not start range streaming as all candidates (%s) are dow

Error message

Can not start range streaming as all candidates (%s) are down

What it means

Thrown by ReconfigureCMS.streamRanges when the current node is the streaming target for a CMS reconfiguration range move and none of the streamCandidates nodes is alive per FailureDetector. Streaming of the metadata log keyspace table cannot start without one live source.

Source

Thrown at src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java:292

            responseTracker.await();
        }
        finally
        {
            DataMovements.instance.unregisterMovements(RESTORE_REPLICA_COUNT, operationId);
        }
    }

    public static void streamRanges(Replica replicaForStreaming, Set<InetAddressAndPort> streamCandidates) throws ExecutionException, InterruptedException
    {
        InetAddressAndPort endpoint = replicaForStreaming.endpoint();

        // Current node is the streaming target. We can pick any other live CMS node as a streaming source
        if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
        {
            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);
            Optional<InetAddressAndPort> streamingSource = streamCandidates.stream().filter(FailureDetector.instance::isAlive).findFirst();
            if (!streamingSource.isPresent())
                throw new IllegalStateException(String.format("Can not start range streaming as all candidates (%s) are down", streamCandidates));
            streamPlan.requestRanges(streamingSource.get(),
                                     SchemaConstants.METADATA_KEYSPACE_NAME,
                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).add(replicaForStreaming).build(),
                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).build(),
                                     DistributedMetadataLogKeyspace.TABLE_NAME);
            streamPlan.execute().get();
        }
        // Current node is a live CMS node, therefore the streaming source
        else if (streamCandidates.contains(FBUtilities.getBroadcastAddressAndPort()))
        {
            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);
            streamPlan.transferRanges(endpoint,
                                      SchemaConstants.METADATA_KEYSPACE_NAME,
                                      new RangesAtEndpoint.Builder(replicaForStreaming.endpoint()).add(replicaForStreaming).build(),
                                      DistributedMetadataLogKeyspace.TABLE_NAME);
            streamPlan.execute().get();
        }
        // We are neither a target, nor a source, so initiate streaming on the target

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Restore connectivity / restart at least one candidate node so failure detection marks it alive, then retry
2. Re-run the CMS reconfiguration with a fresh, currently-live candidate set
3. Tune FailureDetector phi threshold if nodes are falsely reported DOWN
4. Verify the candidates are valid, decommissioned-free members of the cluster

boolean anyAlive = streamCandidates.stream().anyMatch(FailureDetector.instance::isAlive);
if (!anyAlive) throw new PreconditionFailed("all CMS stream candidates down; defer reconfiguration");

try { streamRanges(...); }
catch (IllegalStateException e) { if (e.getMessage().contains("all candidates") && e.getMessage().contains("are down")) retryWithBackoff(); else throw e; }

When it happens

Trigger: Executing the streaming step of a CMS reconfiguration where every candidate endpoint listed in the sequence fails FailureDetector.instance.isAlive.

Common situations: Reconfiguring the CMS during a multi-node outage or network partition; stale streamCandidates referencing decommissioned/unreachable nodes; overly aggressive failure-detector settings marking live peers down.

Related errors

• The channel this output stream was writing to has been close
• This output stream is in an unsafe state after an asynchrono
• failed to connect to %s for streaming data
• Could not resolve any streamCandidates node ids to endpoints
• Streaming to the following hosts failed:

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/1e5a5bd1aa2e885e. Report an issue: GitHub.