apache/cassandra · critical · RuntimeException

Cannot begin paxos auto repair for %s in %s.%s, multiple pen

Error message

Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). Set -D%s=true to skip this check

What it means

When starting a paxos auto repair, Cassandra checks cluster metadata for pending range movements for the range's end token. If multiple pending endpoints exist for the range, a Paxos repair with EACH_QUORUM could be unsafe/incorrect, so a RuntimeException is thrown; it can be suppressed only via the unsafe system property PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.

Source

Thrown at src/java/org/apache/cassandra/service/ActiveRepairService.java:1238

                {
                    Set<InetAddressAndPort> downEndpoints = endpoints.filter(e -> !liveEndpoints.contains(e.endpoint())).endpoints();

                    throw new RuntimeException(String.format("Insufficient live nodes to repair paxos for %s in %s for %s.\n" +
                                                             "There must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\n" +
                                                             "This check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or " +
                                                             "the system property %s to false. The jmx property " +
                                                             "StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without " +
                                                             "restarting the node\n" +
                                                             "Individual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, the" +
                                                             "system property %s, or temporarily with the jmx" +
                                                             "property StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces\n" +
                                                             "Skipping this check can lead to paxos correctness issues",
                                                             range, ksName, reason, downEndpoints, SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE.getKey(), SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE_KEYSPACES.getKey()));
                }
                // todo: can probably be removed with TrM
                if (ClusterMetadata.current().hasPendingRangesFor(keyspace.getMetadata(), range.right) && PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getBoolean())
                {
                    throw new RuntimeException(String.format("Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). " +
                                                             "Set -D%s=true to skip this check",
                                                             range, table.keyspace, table.name, ClusterMetadata.current(), PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getKey()));

                }
                futures.add(() -> PaxosCleanup.cleanup(ctx, liveEndpoints, table, Collections.singleton(range), false, repairCommandExecutor()));
            }
        }

        return futures;
    }

    public int getPaxosRepairParallelism()
    {
        return DatabaseDescriptor.getPaxosRepairParallelism();
    }

    public void setPaxosRepairParallelism(int v)
    {

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Wait for pending ranges to drain (complete in-progress bootstrap/decommission) and re-run the paxos repair
Perform topology changes one at a time so only one pending endpoint exists per range
Set -Dcassandra.paxos_repair_allow_multiple_pending_unsafe=true to bypass (risks paxos correctness)

Example fix

// before (during concurrent bootstraps)
nodetool repair --paxos-only keyspace1
// after
nodetool netstats && nodetool status  # confirm no pending ops
nodetool repair --paxos-only keyspace1

Defensive patterns

Strategy: retry

Validate before calling

if (ClusterMetadata.current().hasPendingRangesFor(Keyspace.open(ks).getMetadata(), range.right)) {
    throw new IllegalStateException("pending ranges exist; defer paxos repair");
}

Try / catch

catch (RuntimeException e) {
    if (e.getMessage().startsWith("Cannot begin paxos auto repair")) {
        backoffAndRetryAfterTopologySettles();
    } else throw e;
}

Prevention

Serialize topology operations; never overlap bootstrap/decommission on the same range
Poll nodetool netstats until no pending bootstrap before paxos-only repair
Use the unsafe -Dcassandra.paxos_repair_allow_multiple_pending_unsafe=true only in controlled test environments

When it happens

Trigger: Initiating paxos-only repair (or topology-change-triggered paxos repair) while a multi-step topology change (e.g. decommission overlapping bootstrap) leaves more than one pending endpoint for the range.

Common situations: Running repairs concurrently with concurrent bootstrap operations; overlapping topology operations (replace + bootstrap) on the same range; scripted node churn without waiting for pending ranges to clear.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/da0588b1a70d65e8. Report an issue: GitHub.