{"record":{"id":"1e5a5bd1aa2e885e","repo":"apache/cassandra","slug":"can-not-start-range-streaming-as-all-candidates","errorCode":null,"errorMessage":"Can not start range streaming as all candidates (%s) are down","messagePattern":"Can not start range streaming as all candidates \\((.+?)\\) are down","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java","lineNumber":292,"sourceCode":"            responseTracker.await();\n        }\n        finally\n        {\n            DataMovements.instance.unregisterMovements(RESTORE_REPLICA_COUNT, operationId);\n        }\n    }\n\n    public static void streamRanges(Replica replicaForStreaming, Set<InetAddressAndPort> streamCandidates) throws ExecutionException, InterruptedException\n    {\n        InetAddressAndPort endpoint = replicaForStreaming.endpoint();\n\n        // Current node is the streaming target. We can pick any other live CMS node as a streaming source\n        if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))\n        {\n            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);\n            Optional<InetAddressAndPort> streamingSource = streamCandidates.stream().filter(FailureDetector.instance::isAlive).findFirst();\n            if (!streamingSource.isPresent())\n                throw new IllegalStateException(String.format(\"Can not start range streaming as all candidates (%s) are down\", streamCandidates));\n            streamPlan.requestRanges(streamingSource.get(),\n                                     SchemaConstants.METADATA_KEYSPACE_NAME,\n                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).add(replicaForStreaming).build(),\n                                     new RangesAtEndpoint.Builder(FBUtilities.getBroadcastAddressAndPort()).build(),\n                                     DistributedMetadataLogKeyspace.TABLE_NAME);\n            streamPlan.execute().get();\n        }\n        // Current node is a live CMS node, therefore the streaming source\n        else if (streamCandidates.contains(FBUtilities.getBroadcastAddressAndPort()))\n        {\n            StreamPlan streamPlan = new StreamPlan(StreamOperation.BOOTSTRAP, 1, true, null, PreviewKind.NONE);\n            streamPlan.transferRanges(endpoint,\n                                      SchemaConstants.METADATA_KEYSPACE_NAME,\n                                      new RangesAtEndpoint.Builder(replicaForStreaming.endpoint()).add(replicaForStreaming).build(),\n                                      DistributedMetadataLogKeyspace.TABLE_NAME);\n            streamPlan.execute().get();\n        }\n        // We are neither a target, nor a source, so initiate streaming on the target","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/sequences/ReconfigureCMS.java#L274-L310","documentation":"Thrown by ReconfigureCMS.streamRanges when the current node is the streaming target for a CMS reconfiguration range move and none of the streamCandidates nodes is alive per FailureDetector. Streaming of the metadata log keyspace table cannot start without one live source.","triggerScenarios":"Executing the streaming step of a CMS reconfiguration where every candidate endpoint listed in the sequence fails FailureDetector.instance.isAlive.","commonSituations":"Reconfiguring the CMS during a multi-node outage or network partition; stale streamCandidates referencing decommissioned/unreachable nodes; overly aggressive failure-detector settings marking live peers down.","solutions":["Restore connectivity / restart at least one candidate node so failure detection marks it alive, then retry","Re-run the CMS reconfiguration with a fresh, currently-live candidate set","Tune FailureDetector phi threshold if nodes are falsely reported DOWN","Verify the candidates are valid, decommissioned-free members of the cluster"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"boolean anyAlive = streamCandidates.stream().anyMatch(FailureDetector.instance::isAlive);\nif (!anyAlive) throw new PreconditionFailed(\"all CMS stream candidates down; defer reconfiguration\");","typeGuard":null,"tryCatchPattern":"try { streamRanges(...); }\ncatch (IllegalStateException e) { if (e.getMessage().contains(\"all candidates\") && e.getMessage().contains(\"are down\")) retryWithBackoff(); else throw e; }","preventionTips":["Run CMS reconfigurations only when a quorum of candidate nodes is alive","Check FailureDetector state for all candidates before starting","Keep candidate lists refreshed to exclude decommissioned nodes"],"tags":["network","streaming","cms","node-down"],"backgroundTag":"node-down","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}