apache/cassandra · critical · RuntimeException

Insufficient live nodes to repair paxos for %s in %s for %s.

Error message

Insufficient live nodes to repair paxos for %s in %s for %s.
There must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s
This check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or the system property %s to false. The jmx property StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without restarting the node
Individual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, thesystem property %s, or temporarily with the jmxproperty StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces
Skipping this check can lead to paxos correctness issues

What it means

Before starting an automatic Paxos repair (e.g. on topology change), Cassandra verifies enough replicas are live to satisfy EACH_QUORUM consistency for the Paxos repair over the given range. If any replica in the range is down, the paxos repair cannot be safely performed and a RuntimeException is thrown, unless the skip_paxos_repair_on_topology_change check is disabled.

Source

Thrown at src/java/org/apache/cassandra/service/ActiveRepairService.java:1223

        for (Range<Token> range: ranges)
        {
            for (TableMetadata table : tables)
            {

                ReplicationParams replication = keyspace.getMetadata().params.replication;
                // Special case meta keyspace as it uses a custom partitioner/tokens, but the paxos table and repairs
                // are based on the system partitioner
                EndpointsForRange endpoints = replication.isMeta()
                                              ? ClusterMetadata.current().fullCMSMembersAsReplicas()
                                              : ClusterMetadata.current().placement(replication).reads.forRange(range).get();

                Set<InetAddressAndPort> liveEndpoints = endpoints.filter(FailureDetector.isReplicaAlive).endpoints();
                if (!PaxosRepair.hasSufficientLiveNodesForTopologyChange(keyspace, range, liveEndpoints))
                {
                    Set<InetAddressAndPort> downEndpoints = endpoints.filter(e -> !liveEndpoints.contains(e.endpoint())).endpoints();

                    throw new RuntimeException(String.format("Insufficient live nodes to repair paxos for %s in %s for %s.\n" +
                                                             "There must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\n" +
                                                             "This check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or " +
                                                             "the system property %s to false. The jmx property " +
                                                             "StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without " +
                                                             "restarting the node\n" +
                                                             "Individual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, the" +
                                                             "system property %s, or temporarily with the jmx" +
                                                             "property StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces\n" +
                                                             "Skipping this check can lead to paxos correctness issues",
                                                             range, ksName, reason, downEndpoints, SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE.getKey(), SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE_KEYSPACES.getKey()));
                }
                // todo: can probably be removed with TrM
                if (ClusterMetadata.current().hasPendingRangesFor(keyspace.getMetadata(), range.right) && PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getBoolean())
                {
                    throw new RuntimeException(String.format("Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). " +
                                                             "Set -D%s=true to skip this check",
                                                             range, table.keyspace, table.name, ClusterMetadata.current(), PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getKey()));

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Bring the listed down nodes back up before performing the topology change
2. Set cassandra.yaml skip_paxos_repair_on_topology_change: false (accepting paxos correctness risk)
3. Start the node with -Dcassandra.skip_paxos_repair_on_topology_change=false, or toggle JMX property StorageService.SkipPaxosRepairOnTopologyChange to false
4. Exclude the specific keyspace via skip_paxos_repair_on_topology_change_keyspaces

Example fix

// cassandra.yaml before (default)
skip_paxos_repair_on_topology_change: true
// after (deliberate override)
skip_paxos_repair_on_topology_change: false

Set<InetAddressAndPort> down = endpoints.filter(e -> !FailureDetector.isReplicaAlive.test(e)).endpoints();
if (!down.isEmpty()) throw new IllegalStateException("nodes down; paxos repair would fail: " + down);

try { triggerTopologyChange(); }
catch (RuntimeException e) {
    if (e.getMessage().startsWith("Insufficient live nodes to repair paxos")) {
        bringNodesUpThenRetry();
    } else throw e;
}

When it happens

Trigger: Bootstrap/decommission/move triggering paxos auto-repair while one or more replicas of the affected range are down (FAILED or unreachable per FailureDetector).

Common situations: Replacing or decommissioning nodes with another node already down; running topology changes during an outage; misconfigured failure detector marking nodes down.

Related errors

• Cannot perform LWT operation as there is more than one (%d)
• Cannot begin paxos auto repair for %s in %s.%s, multiple pen
• Cannot achieve consistency level {consistencyForConsensus} {
• Insufficient read responses: ${readResponses}; need ${sizeOf
• Not running paxos repair for topology change because paxos r

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/91b9ad13a51177c2. Report an issue: GitHub.