apache/cassandra · error · IllegalStateException

Node should be either source or destination in the movement

Error message

Node should be either source or destination in the movement map

What it means

During Move.executeNext, when building the streaming plan, every destination endpoint in the movement map must appear either as a source or a destination for the streamed ranges. If a node is in the movement map but plays neither role for the given range, the code throws IllegalStateException, indicating the computed movement map is inconsistent. It is an internal invariant failure in range/endpoint calculation.

Source

Thrown at src/java/org/apache/cassandra/tcm/sequences/Move.java:357

                        String[] cfNames = StreamPlan.nonAccordTablesForKeyspace(ks);
                        for (Map.Entry<Replica, Replica> e : endpoints.flattenEntries())
                        {
                            Replica destination = e.getKey();
                            Replica source = e.getValue();
                            logger.info("Stream source: {} destination: {}", source, destination);
                            assert !source.endpoint().equals(destination.endpoint()) : String.format("Source %s should not be the same as destionation %s", source, destination);
                            if (source.isSelf())
                                streamPlan.transferRanges(destination.endpoint(), ks.name, RangesAtEndpoint.of(destination), cfNames);
                            else if (destination.isSelf())
                            {
                                if (destination.isFull())
                                    streamPlan.requestRanges(source.endpoint(), ks.name, RangesAtEndpoint.of(destination), RangesAtEndpoint.empty(destination.endpoint()), cfNames);
                                else
                                    streamPlan.requestRanges(source.endpoint(), ks.name, RangesAtEndpoint.empty(destination.endpoint()), RangesAtEndpoint.of(destination), cfNames);
                            }
                            else
                                throw new IllegalStateException("Node should be either source or destination in the movement map " + endpoints);
                        }
                    }

                    StreamResultFuture streamResult = streamPlan.execute();

                    Future<?> accordReady = AccordService.instance().epochReadyFor(metadata, EpochReady::reads);
                    Future<?> ready = FutureCombiner.allOf(streamResult, accordReady);
                    ready.get();
                    StorageService.instance.repairPaxosForTopologyChange("move");
                }
                catch (InterruptedException e)
                {
                    return continuable();
                }
                catch (ExecutionException e)
                {
                    StorageService.instance.markMoveFailed();
                    throw new RuntimeException("Unable to move", e);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Ensure no other topology operations are running and retry the move after the cluster is quiescent
Verify token metadata consistency (nodetool status, describecluster) before moving
Cancel/roll back the stuck move sequence via the TCM cancel path, then re-run
If reproducible on a quiescent cluster, capture the movement map and file a Cassandra bug

Example fix

// before
streamPlan.requestRanges(rangesFor(destination.endpoint()), rangesFor(destination.endpoint()));
// after
boolean isSource = rangesFor(destination.endpoint()).contains(range);
boolean isDest = destination.endpoint().equals(destination.endpoint());
if (!isSource && !isDest)
    throw new IllegalStateException("Node should be either source or destination in the movement map " + endpoints); // keep guard, but validate movementMap upstream

Defensive patterns

Strategy: validation

Validate before calling

if (ClusterMetadata.current().inProgressSequences.size() > 0)
    throw new IllegalStateException("Wait for in-progress topology operations to finish before nodetool move");

Try / catch

try { executeMove(); } catch (IllegalStateException e) { logger.error("Move plan inconsistent: {}", e.getMessage()); markMoveFailed(); alertOperator(); }

Prevention

Run only one topology operation at a time
Verify token metadata consistency before moving
Roll back and re-run a move after any failed attempt rather than layering operations

When it happens

Trigger: executeNext() streaming step encounters a movement-map entry whose endpoint is neither listed in sources nor as destination for a range; typically caused by inconsistent token metadata, a concurrent topology change mid-move, or a bug in movementMap computation.

Common situations: nodetool move on a node while another topology operation (bootstrap/decommission) is in flight; corrupted replication metadata after a failed earlier move; running move on a cluster with mismatched replication factor settings.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/7a8836b737c82313. Report an issue: GitHub.