{"record":{"id":"91b9ad13a51177c2","repo":"apache/cassandra","slug":"insufficient-live-nodes-to-repair-paxos-for-s-in","errorCode":null,"errorMessage":"Insufficient live nodes to repair paxos for %s in %s for %s.\nThere must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\nThis check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or the system property %s to false. The jmx property StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without restarting the node\nIndividual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, thesystem property %s, or temporarily with the jmxproperty StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces\nSkipping this check can lead to paxos correctness issues","messagePattern":"Insufficient live nodes to repair paxos for %s in %s for %s\\.\nThere must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\nThis check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or the system property %s to false\\. The jmx property StorageService\\.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without restarting the node\nIndividual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, thesystem property %s, or temporarily with the jmxproperty StorageService\\.SkipPaxosRepairOnTopologyChangeKeyspaces\nSkipping this check can lead to paxos correctness issues","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"src/java/org/apache/cassandra/service/ActiveRepairService.java","lineNumber":1223,"sourceCode":"\n        for (Range<Token> range: ranges)\n        {\n            for (TableMetadata table : tables)\n            {\n\n                ReplicationParams replication = keyspace.getMetadata().params.replication;\n                // Special case meta keyspace as it uses a custom partitioner/tokens, but the paxos table and repairs\n                // are based on the system partitioner\n                EndpointsForRange endpoints = replication.isMeta()\n                                              ? ClusterMetadata.current().fullCMSMembersAsReplicas()\n                                              : ClusterMetadata.current().placement(replication).reads.forRange(range).get();\n\n                Set<InetAddressAndPort> liveEndpoints = endpoints.filter(FailureDetector.isReplicaAlive).endpoints();\n                if (!PaxosRepair.hasSufficientLiveNodesForTopologyChange(keyspace, range, liveEndpoints))\n                {\n                    Set<InetAddressAndPort> downEndpoints = endpoints.filter(e -> !liveEndpoints.contains(e.endpoint())).endpoints();\n\n                    throw new RuntimeException(String.format(\"Insufficient live nodes to repair paxos for %s in %s for %s.\\n\" +\n                                                             \"There must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\\n\" +\n                                                             \"This check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or \" +\n                                                             \"the system property %s to false. The jmx property \" +\n                                                             \"StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without \" +\n                                                             \"restarting the node\\n\" +\n                                                             \"Individual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, the\" +\n                                                             \"system property %s, or temporarily with the jmx\" +\n                                                             \"property StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces\\n\" +\n                                                             \"Skipping this check can lead to paxos correctness issues\",\n                                                             range, ksName, reason, downEndpoints, SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE.getKey(), SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE_KEYSPACES.getKey()));\n                }\n                // todo: can probably be removed with TrM\n                if (ClusterMetadata.current().hasPendingRangesFor(keyspace.getMetadata(), range.right) && PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getBoolean())\n                {\n                    throw new RuntimeException(String.format(\"Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). \" +\n                                                             \"Set -D%s=true to skip this check\",\n                                                             range, table.keyspace, table.name, ClusterMetadata.current(), PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getKey()));\n","sourceCodeStart":1205,"sourceCodeEnd":1241,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/ActiveRepairService.java#L1205-L1241","documentation":"Before starting an automatic Paxos repair (e.g. on topology change), Cassandra verifies enough replicas are live to satisfy EACH_QUORUM consistency for the Paxos repair over the given range. If any replica in the range is down, the paxos repair cannot be safely performed and a RuntimeException is thrown, unless the skip_paxos_repair_on_topology_change check is disabled.","triggerScenarios":"Bootstrap/decommission/move triggering paxos auto-repair while one or more replicas of the affected range are down (FAILED or unreachable per FailureDetector).","commonSituations":"Replacing or decommissioning nodes with another node already down; running topology changes during an outage; misconfigured failure detector marking nodes down.","solutions":["Bring the listed down nodes back up before performing the topology change","Set cassandra.yaml skip_paxos_repair_on_topology_change: false (accepting paxos correctness risk)","Start the node with -Dcassandra.skip_paxos_repair_on_topology_change=false, or toggle JMX property StorageService.SkipPaxosRepairOnTopologyChange to false","Exclude the specific keyspace via skip_paxos_repair_on_topology_change_keyspaces"],"exampleFix":"// cassandra.yaml before (default)\nskip_paxos_repair_on_topology_change: true\n// after (deliberate override)\nskip_paxos_repair_on_topology_change: false","handlingStrategy":"validation","validationCode":"Set<InetAddressAndPort> down = endpoints.filter(e -> !FailureDetector.isReplicaAlive.test(e)).endpoints();\nif (!down.isEmpty()) throw new IllegalStateException(\"nodes down; paxos repair would fail: \" + down);","typeGuard":null,"tryCatchPattern":"try { triggerTopologyChange(); }\ncatch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Insufficient live nodes to repair paxos\")) {\n        bringNodesUpThenRetry();\n    } else throw e;\n}","preventionTips":["Check nodetool status for down nodes before bootstrap/decommission","Only disable skip_paxos_repair_on_topology_change knowingly; it risks paxos correctness","Schedule topology changes during full-cluster health windows"],"tags":["paxos","topology-change","quorum","runtime"],"backgroundTag":"insufficient-live-nodes-for-quorum","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}