apache/cassandra · error · RuntimeException

Could not catchup with peers

Error message

Could not catchup with peers

What it means

Thrown by AccordService.catchup() when the node fails to reach Accord catchup with its peers within the maximum allowed latency (failAt deadline) and accord.catchup_on_start_exit_on_failure is true. The node shuts down because remaining behind can compromise Accord correctness.

Source

Thrown at src/java/org/apache/cassandra/service/accord/AccordService.java:752

                    long failAt = start + maxLatencyNanos;
                    Future<Void> f;
                    {
                        AsyncChain<Void> submit;
                        if (mode == HARD)
                        {
                            if (!node.durability().isStarted())
                                node.durability().start();
                            submit = CatchupHard.catchup(node);
                        }
                        else submit = Catchup.catchup(node);
                        f = toFuture(submit);
                    }
                    if (!f.awaitUntilThrowUncheckedOnInterrupt(failAt))
                    {
                        if (spec.catchup_on_start_exit_on_failure)
                        {
                            logger.error("Catchup exceeded maximum latency of {}ns; shutting down", maxLatencyNanos);
                            throw new RuntimeException("Could not catchup with peers");
                        }
                        logger.error("Catchup exceeded maximum latency of {}ns; continuing to startup", maxLatencyNanos);
                        break;
                    }

                    Throwable failed = f.cause();
                    if (failed != null)
                    {
                        if (spec.catchup_on_start_exit_on_failure)
                            throw new RuntimeException("Could not catchup with peers", failed);

                        logger.error("Could not catchup with peers; continuing to startup");
                        break;
                    }

                    long end = nanoTime();
                    double seconds = NANOSECONDS.toMillis(end - start)/1000.0;
                    logger.info("Finished catchup with all quorums. {}s elapsed.", String.format("%.2f", seconds));

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Verify network connectivity and cluster health (nodetool status, gossip) and restart the node.
2. Increase the catchup latency budget (accord.catchup_on_start_max_latency) to allow slower catchup.
3. Set accord.catchup_on_start_exit_on_failure=false to continue startup despite slow catchup (a warning is logged) only if consistency risk is acceptable.
4. Reduce cluster transaction load or restart during low-traffic windows so catchup can complete faster.

Example fix

// before
-Dcassandra.accord.catchup_on_start_max_latency=30s
// after
-Dcassandra.accord.catchup_on_start_max_latency=120s

// before starting the node
if (!peersReachable(cluster) || clusterUnderHeavyLoad())
    waitUntil(peersHealthy(), thenStartNode);

catch (RuntimeException e) {
    if (e.getMessage().equals("Could not catchup with peers")) {
        verifyNetworkAndPeers();
        scheduleRestartWithLargerCatchupBudget();
    } else throw e;
}

When it happens

Trigger: Node startup catchup loop where awaitUntilThrowUncheckedOnInterrupt(failAt) returns false — i.e., the catchup future did not complete before maxLatencyNanos elapsed, with catchup_on_start_exit_on_failure=true.

Common situations: Restarting a node that has been down for a long time into a busy cluster; slow network between replicas; cluster under heavy transaction load so catchup cannot keep up within the deadline; too-low accord.catchup_on_start_max_latency config.

Related errors

• Timeout waiting to exeute waiting transactions
• Gave up waiting on journal index to be ready
• accord.journal_directory must not be the same as the commitl
• hints_directory must not be the same as the accord.journal_d
• saved_caches_directory must not be the same as the accord.jo

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/b059e92b38877d2a. Report an issue: GitHub.