apache/cassandra · error · RuntimeException

Could not catchup with peers

Error message

Could not catchup with peers

What it means

Thrown by AccordService.catchup() when the node fails to reach Accord catchup with its peers within the maximum allowed latency (failAt deadline) and accord.catchup_on_start_exit_on_failure is true. The node shuts down because remaining behind can compromise Accord correctness.

Solutions

1. Verify network connectivity and cluster health (nodetool status, gossip) and restart the node.
2. Increase the catchup latency budget (accord.catchup_on_start_max_latency) to allow slower catchup.
3. Set accord.catchup_on_start_exit_on_failure=false to continue startup despite slow catchup (a warning is logged) only if consistency risk is acceptable.
4. Reduce cluster transaction load or restart during low-traffic windows so catchup can complete faster.

Example fix

// before
-Dcassandra.accord.catchup_on_start_max_latency=30s
// after
-Dcassandra.accord.catchup_on_start_max_latency=120s

// before starting the node
if (!peersReachable(cluster) || clusterUnderHeavyLoad())
    waitUntil(peersHealthy(), thenStartNode);

catch (RuntimeException e) {
    if (e.getMessage().equals("Could not catchup with peers")) {
        verifyNetworkAndPeers();
        scheduleRestartWithLargerCatchupBudget();
    } else throw e;
}

When it happens

Trigger: Node startup catchup loop where awaitUntilThrowUncheckedOnInterrupt(failAt) returns false — i.e., the catchup future did not complete before maxLatencyNanos elapsed, with catchup_on_start_exit_on_failure=true.

Common situations: Restarting a node that has been down for a long time into a busy cluster; slow network between replicas; cluster under heavy transaction load so catchup cannot keep up within the deadline; too-low accord.catchup_on_start_max_latency config.

Related errors

• Gave up waiting on journal index to be ready
• Timeout waiting to exeute waiting transactions
• accord.journal_directory must not be the same as the…
• Cannot replace same address when accord transactions are…
• Cannot reset state and replay from a save point; must…

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/b059e92b38877d2a. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/accord/AccordService.java:752

                    long failAt = start + maxLatencyNanos;
                    Future<Void> f;
                    {
                        AsyncChain<Void> submit;
                        if (mode == HARD)
                        {
                            if (!node.durability().isStarted())
                                node.durability().start();
                            submit = CatchupHard.catchup(node);
                        }
                        else submit = Catchup.catchup(node);
                        f = toFuture(submit);
                    }
                    if (!f.awaitUntilThrowUncheckedOnInterrupt(failAt))
                    {
                        if (spec.catchup_on_start_exit_on_failure)
                        {
                            logger.error("Catchup exceeded maximum latency of {}ns; shutting down", maxLatencyNanos);
                            throw new RuntimeException("Could not catchup with peers");
                        }
                        logger.error("Catchup exceeded maximum latency of {}ns; continuing to startup", maxLatencyNanos);
                        break;
                    }

                    Throwable failed = f.cause();
                    if (failed != null)
                    {
                        if (spec.catchup_on_start_exit_on_failure)
                            throw new RuntimeException("Could not catchup with peers", failed);

                        logger.error("Could not catchup with peers; continuing to startup");
                        break;
                    }

                    long end = nanoTime();
                    double seconds = NANOSECONDS.toMillis(end - start)/1000.0;
                    logger.info("Finished catchup with all quorums. {}s elapsed.", String.format("%.2f", seconds));

View on GitHub (pinned to 88fd0f6a0e)