{"record":{"id":"b059e92b38877d2a","repo":"apache/cassandra","slug":"could-not-catchup-with-peers","errorCode":null,"errorMessage":"Could not catchup with peers","messagePattern":"Could not catchup with peers","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/accord/AccordService.java","lineNumber":752,"sourceCode":"                    long failAt = start + maxLatencyNanos;\n                    Future<Void> f;\n                    {\n                        AsyncChain<Void> submit;\n                        if (mode == HARD)\n                        {\n                            if (!node.durability().isStarted())\n                                node.durability().start();\n                            submit = CatchupHard.catchup(node);\n                        }\n                        else submit = Catchup.catchup(node);\n                        f = toFuture(submit);\n                    }\n                    if (!f.awaitUntilThrowUncheckedOnInterrupt(failAt))\n                    {\n                        if (spec.catchup_on_start_exit_on_failure)\n                        {\n                            logger.error(\"Catchup exceeded maximum latency of {}ns; shutting down\", maxLatencyNanos);\n                            throw new RuntimeException(\"Could not catchup with peers\");\n                        }\n                        logger.error(\"Catchup exceeded maximum latency of {}ns; continuing to startup\", maxLatencyNanos);\n                        break;\n                    }\n\n                    Throwable failed = f.cause();\n                    if (failed != null)\n                    {\n                        if (spec.catchup_on_start_exit_on_failure)\n                            throw new RuntimeException(\"Could not catchup with peers\", failed);\n\n                        logger.error(\"Could not catchup with peers; continuing to startup\");\n                        break;\n                    }\n\n                    long end = nanoTime();\n                    double seconds = NANOSECONDS.toMillis(end - start)/1000.0;\n                    logger.info(\"Finished catchup with all quorums. {}s elapsed.\", String.format(\"%.2f\", seconds));","sourceCodeStart":734,"sourceCodeEnd":770,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/accord/AccordService.java#L734-L770","documentation":"Thrown by AccordService.catchup() when the node fails to reach Accord catchup with its peers within the maximum allowed latency (failAt deadline) and accord.catchup_on_start_exit_on_failure is true. The node shuts down because remaining behind can compromise Accord correctness.","triggerScenarios":"Node startup catchup loop where awaitUntilThrowUncheckedOnInterrupt(failAt) returns false — i.e., the catchup future did not complete before maxLatencyNanos elapsed, with catchup_on_start_exit_on_failure=true.","commonSituations":"Restarting a node that has been down for a long time into a busy cluster; slow network between replicas; cluster under heavy transaction load so catchup cannot keep up within the deadline; too-low accord.catchup_on_start_max_latency config.","solutions":["Verify network connectivity and cluster health (nodetool status, gossip) and restart the node.","Increase the catchup latency budget (accord.catchup_on_start_max_latency) to allow slower catchup.","Set accord.catchup_on_start_exit_on_failure=false to continue startup despite slow catchup (a warning is logged) only if consistency risk is acceptable.","Reduce cluster transaction load or restart during low-traffic windows so catchup can complete faster."],"exampleFix":"// before\n-Dcassandra.accord.catchup_on_start_max_latency=30s\n// after\n-Dcassandra.accord.catchup_on_start_max_latency=120s","handlingStrategy":"retry","validationCode":"// before starting the node\nif (!peersReachable(cluster) || clusterUnderHeavyLoad())\n    waitUntil(peersHealthy(), thenStartNode);","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) {\n    if (e.getMessage().equals(\"Could not catchup with peers\")) {\n        verifyNetworkAndPeers();\n        scheduleRestartWithLargerCatchupBudget();\n    } else throw e;\n}","preventionTips":["Set catchup max latency generously for long-downtime restarts.","Verify network and peer health before restarting a stale node.","Restart lagging nodes during low-traffic windows."],"tags":["accord","catchup","startup","replication","timeout"],"backgroundTag":"peer-sync-failure","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}