apache/cassandra · error · IllegalStateException

Can not start range streaming as all candidates

Error message

Can not start range streaming as all candidates (%s) are down

What it means

Thrown by ReconfigureCMS.streamRanges when the current node is the streaming target for a CMS reconfiguration range move and none of the streamCandidates nodes is alive per FailureDetector. Streaming of the metadata log keyspace table cannot start without one live source.

Solutions

Restore connectivity / restart at least one candidate node so failure detection marks it alive, then retry
Re-run the CMS reconfiguration with a fresh, currently-live candidate set
Tune FailureDetector phi threshold if nodes are falsely reported DOWN
Verify the candidates are valid, decommissioned-free members of the cluster

Defensive patterns

Strategy: retry

Validate before calling

boolean anyAlive = streamCandidates.stream().anyMatch(FailureDetector.instance::isAlive);
if (!anyAlive) throw new PreconditionFailed("all CMS stream candidates down; defer reconfiguration");

Try / catch

try { streamRanges(...); }
catch (IllegalStateException e) { if (e.getMessage().contains("all candidates") && e.getMessage().contains("are down")) retryWithBackoff(); else throw e; }

Prevention

Run CMS reconfigurations only when a quorum of candidate nodes is alive
Check FailureDetector state for all candidates before starting
Keep candidate lists refreshed to exclude decommissioned nodes

When it happens

Trigger: Executing the streaming step of a CMS reconfiguration where every candidate endpoint listed in the sequence fails FailureDetector.instance.isAlive.

Common situations: Reconfiguring the CMS during a multi-node outage or network partition; stale streamCandidates referencing decommissioned/unreachable nodes; overly aggressive failure-detector settings marking live peers down.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/1e5a5bd1aa2e885e. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java:292

            responseTracker.await();
        }
        finally
        {
            DataMovements.instance.unregisterMovements(RESTORE_REPLICA_COUNT, operationId);
        }
    }

    public static void streamRanges(Replica replicaForStreaming, Set<InetAddressAndPort> streamCandidates) throws ExecutionException, InterruptedException
    {
        InetAddressAndPort endpoint = replicaForStreaming.endpoint();

        // Current node is the streaming target. We can pick any other live CMS node as a streaming source
        if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
        {
            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);
            Optional<InetAddressAndPort> streamingSource = streamCandidates.stream().filter(FailureDetector.instance::isAlive).findFirst();
            if (!streamingSource.isPresent())
                throw new IllegalStateException(String.format("Can not start range streaming as all candidates (%s) are down", streamCandidates));
            streamPlan.requestRanges(streamingSource.get(),
                                     SchemaConstants.METADATA_KEYSPACE_NAME,
                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).add(replicaForStreaming).build(),
                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).build(),
                                     DistributedMetadataLogKeyspace.TABLE_NAME);
            streamPlan.execute().get();
        }
        // Current node is a live CMS node, therefore the streaming source
        else if (streamCandidates.contains(FBUtilities.getBroadcastAddressAndPort()))
        {
            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);
            streamPlan.transferRanges(endpoint,
                                      SchemaConstants.METADATA_KEYSPACE_NAME,
                                      new RangesAtEndpoint.Builder(replicaForStreaming.endpoint()).add(replicaForStreaming).build(),
                                      DistributedMetadataLogKeyspace.TABLE_NAME);
            streamPlan.execute().get();
        }
        // We are neither a target, nor a source, so initiate streaming on the target

View on GitHub (pinned to 88fd0f6a0e)