{"record":{"id":"da0588b1a70d65e8","repo":"apache/cassandra","slug":"cannot-begin-paxos-auto-repair-for-s-in-s-s-mu","errorCode":null,"errorMessage":"Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). Set -D%s=true to skip this check","messagePattern":"Cannot begin paxos auto repair for (.+?) in (.+?)\\.(.+?), multiple pending endpoints exist for range \\(metadata = (.+?)\\)\\. Set -D(.+?)=true to skip this check","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"src/java/org/apache/cassandra/service/ActiveRepairService.java","lineNumber":1238,"sourceCode":"                {\n                    Set<InetAddressAndPort> downEndpoints = endpoints.filter(e -> !liveEndpoints.contains(e.endpoint())).endpoints();\n\n                    throw new RuntimeException(String.format(\"Insufficient live nodes to repair paxos for %s in %s for %s.\\n\" +\n                                                             \"There must be enough live nodes to satisfy EACH_QUORUM, but the following nodes are down: %s\\n\" +\n                                                             \"This check can be skipped by setting either the yaml property skip_paxos_repair_on_topology_change or \" +\n                                                             \"the system property %s to false. The jmx property \" +\n                                                             \"StorageService.SkipPaxosRepairOnTopologyChange can also be set to false to temporarily disable without \" +\n                                                             \"restarting the node\\n\" +\n                                                             \"Individual keyspaces can be skipped with the yaml property skip_paxos_repair_on_topology_change_keyspaces, the\" +\n                                                             \"system property %s, or temporarily with the jmx\" +\n                                                             \"property StorageService.SkipPaxosRepairOnTopologyChangeKeyspaces\\n\" +\n                                                             \"Skipping this check can lead to paxos correctness issues\",\n                                                             range, ksName, reason, downEndpoints, SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE.getKey(), SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE_KEYSPACES.getKey()));\n                }\n                // todo: can probably be removed with TrM\n                if (ClusterMetadata.current().hasPendingRangesFor(keyspace.getMetadata(), range.right) && PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getBoolean())\n                {\n                    throw new RuntimeException(String.format(\"Cannot begin paxos auto repair for %s in %s.%s, multiple pending endpoints exist for range (metadata = %s). \" +\n                                                             \"Set -D%s=true to skip this check\",\n                                                             range, table.keyspace, table.name, ClusterMetadata.current(), PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.getKey()));\n\n                }\n                futures.add(() -> PaxosCleanup.cleanup(ctx, liveEndpoints, table, Collections.singleton(range), false, repairCommandExecutor()));\n            }\n        }\n\n        return futures;\n    }\n\n    public int getPaxosRepairParallelism()\n    {\n        return DatabaseDescriptor.getPaxosRepairParallelism();\n    }\n\n    public void setPaxosRepairParallelism(int v)\n    {","sourceCodeStart":1220,"sourceCodeEnd":1256,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/ActiveRepairService.java#L1220-L1256","documentation":"When starting a paxos auto repair, Cassandra checks cluster metadata for pending range movements for the range's end token. If multiple pending endpoints exist for the range, a Paxos repair with EACH_QUORUM could be unsafe/incorrect, so a RuntimeException is thrown; it can be suppressed only via the unsafe system property PAXOS_REPAIR_ALLOW_MULTIPLE_PENDING_UNSAFE.","triggerScenarios":"Initiating paxos-only repair (or topology-change-triggered paxos repair) while a multi-step topology change (e.g. decommission overlapping bootstrap) leaves more than one pending endpoint for the range.","commonSituations":"Running repairs concurrently with concurrent bootstrap operations; overlapping topology operations (replace + bootstrap) on the same range; scripted node churn without waiting for pending ranges to clear.","solutions":["Wait for pending ranges to drain (complete in-progress bootstrap/decommission) and re-run the paxos repair","Perform topology changes one at a time so only one pending endpoint exists per range","Set -Dcassandra.paxos_repair_allow_multiple_pending_unsafe=true to bypass (risks paxos correctness)"],"exampleFix":"// before (during concurrent bootstraps)\nnodetool repair --paxos-only keyspace1\n// after\nnodetool netstats && nodetool status  # confirm no pending ops\nnodetool repair --paxos-only keyspace1","handlingStrategy":"retry","validationCode":"if (ClusterMetadata.current().hasPendingRangesFor(Keyspace.open(ks).getMetadata(), range.right)) {\n    throw new IllegalStateException(\"pending ranges exist; defer paxos repair\");\n}","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Cannot begin paxos auto repair\")) {\n        backoffAndRetryAfterTopologySettles();\n    } else throw e;\n}","preventionTips":["Serialize topology operations; never overlap bootstrap/decommission on the same range","Poll nodetool netstats until no pending bootstrap before paxos-only repair","Use the unsafe -Dcassandra.paxos_repair_allow_multiple_pending_unsafe=true only in controlled test environments"],"tags":["paxos","topology","consistency","runtime"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}