apache/cassandra · error

A node required to move the data consistently is down

Error message

A node required to move the data consistently is down

What it means

In RangeStreamer.calculateRangesToFetchWithPreferredEndpoints, when no live source satisfies the required endpoints for a range in an RF=1 keyspace and strict consistency is requested, the code logs 'A node required to move the data consistently is down' and throws IllegalStateException: strict consistency demands the (single) replica be available to stream, and it is not.

Source

Thrown at src/java/org/apache/cassandra/dht/RangeStreamer.java:571

                 throw new IllegalStateException("Failed to find endpoints to fetch " + toFetch);

             /*
              * When we move forwards (shrink our bucket) we are the one losing a range and no one else loses
              * from that action (we also don't gain). When we move backwards there are two people losing a range. One is a full replica
              * and the other is a transient replica. So we must need fetch from two places in that case for the full range we gain.
              * For a transient range we only need to fetch from one.
              */
             if (useStrictConsistency && addressList.size() > 1 && (addressList.filter(Replica::isFull).size() > 1 || addressList.filter(Replica::isTransient).size() > 1))
                 throw new IllegalStateException(String.format("Multiple strict sources found for %s, sources: %s", toFetch, addressList));

             //We must have enough stuff to fetch from
             if (!any(addressList, isSufficient))
             {
                 if (strat.getReplicationFactor().allReplicas == 1)
                 {
                     if (useStrictConsistency)
                     {
                         logger.warn("A node required to move the data consistently is down");
                         throw new IllegalStateException("Unable to find sufficient sources for streaming range " + toFetch + " in keyspace " + keyspace + " with RF=1. " +
                                                         "Ensure this keyspace contains replicas in the source datacenter.");
                     }
                     else
                         logger.warn("Unable to find sufficient sources for streaming range {} in keyspace {} with RF=1. " +
                                     "Keyspace might be missing data.", toFetch, keyspace);
                 }
                 else
                 {
                     if (useStrictConsistency)
                         logger.warn("A node required to move the data consistently is down");
                     throw new IllegalStateException("Unable to find sufficient sources for streaming range " + toFetch + " in keyspace " + keyspace);
                 }
             }
         }
         return rangesToFetchWithPreferredEndpoints.build();
     }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

  1. Bring the down node required for streaming back up, then retry the bootstrap/operation.
  2. Temporarily raise the keyspace's replication factor and run repair so more sources exist.
  3. If data loss is acceptable for that keyspace, drop the RF=1 keyspace or proceed without strict consistency (disable consistent range movement) knowingly.
  4. Use a verified backup/snapshot to restore the missing replica before streaming.

Example fix

// ensure enough replicas before topology ops
cqlsh> ALTER KEYSPACE my_ks WITH replication = {'class':'NetworkTopologyStrategy','dc1':3};
$ nodetool repair -full my_ks
Defensive patterns

Strategy: validation

Validate before calling

// preflight: ensure all nodes are up and no RF=1 keyspaces exist before bootstrap
boolean nodesUp = session.execute("SELECT peer FROM system.peers").all()
    .stream().allMatch(r -> true); // combine with nodetool status
boolean hasRf1 = session.execute("SELECT keyspace_name, replication FROM system_schema.keyspaces").all()
    .stream().anyMatch(r -> r.getMap("replication", String.class, String.class).values().stream()
        .anyMatch(v -> v.equals("1")));

Prevention

When it happens

Trigger: nodetool bootstrap/move/decommission (fetchMap path) with -Dcassandra.consistent.rangemovement=true (strict) where the only replica of an RF=1 keyspace range is dead or not a valid streaming source.

Common situations: Bootstrap of a new node while the sole replica of a single-RF keyspace is down; RF=1 keyspaces left over from tests; hardware loss of the only replica combined with consistent range movement requirements.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors


AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/335ee5707299a5e1. Report an issue: GitHub.