apache/cassandra · error · RuntimeException

Timeout waiting to exeute waiting transactions

Error message

Timeout waiting to exeute waiting transactions

What it means

Thrown by AccordService.localStartup() when transactions queued for execution at node start (waiting transactions) do not finish within accord.journal.execute_waiting_on_start_timeout and accord.journal.execute_waiting_on_start_fail_on_timeout is true. This blocks node startup rather than continuing in a degraded state.

Source

Thrown at src/java/org/apache/cassandra/service/accord/AccordService.java:647

            }
            else
            {
                replayJournal(minSegments);

                if (getAccord().execute_waiting_on_start)
                {
                    logger.info("Execute waiting transactions...");
                    List<AsyncResult<Void>> results = new ArrayList<>();
                    node.commandStores().forAllUnsafe(commandStore -> results.add(commandStore.tryToExecuteListeningTxns(false)));
                    if (!results.isEmpty())
                    {
                        Future<?> future = toFuture(AsyncResults.reduce(results, Reduce.toNull()));
                        long timeoutSeconds = getAccord().execute_waiting_on_start_timeout.toSeconds();
                        if (timeoutSeconds <= 0) future.awaitUninterruptibly().rethrowIfFailed();
                        else if (!future.awaitUninterruptibly(timeoutSeconds, SECONDS))
                        {
                            if (getAccord().execute_waiting_on_start_fail_on_timeout)
                                throw new RuntimeException("Timeout waiting to exeute waiting transactions");
                            logger.warn("Timeout waiting to exeute waiting transactions");
                        }
                        else future.rethrowIfFailed();
                    }
                }
            }
        }
        finally
        {
            node.unsafeSetReplaying(false);
        }
        node.commandStores().forAllUnsafe(commandStore -> ((AccordCommandStore)commandStore).ensureDurable());
    }

    private Long2LongHashMap restoreFromSavePoints(CommandStores commandStores)
    {
        Long2LongHashMap result = new Long2LongHashMap(0);
        Future<List<Map.Entry<Integer, Long>>> future = toFuture(((AccordCommandStores)commandStores).restoreState());

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Increase accord.journal.execute_waiting_on_start_timeout to allow the backlog to drain.
2. Set accord.journal.execute_waiting_on_start_fail_on_timeout=false to continue startup with a warning instead of failing (a warning is logged either way).
3. Fix root-cause slowness: disk I/O, GC settings, or reduce the transaction backlog before restarting.
4. Retry startup during lower-load windows after the backlog shrinks.

Example fix

// before
-Dcassandra.accord.execute_waiting_on_start_timeout=30s
// after
-Dcassandra.accord.execute_waiting_on_start_timeout=300s

// before restart with backlog
long backlog = estimateWaitingAccordTxns(node);
if (backlog * avgExecMillis > executeWaitingOnStartTimeoutMs)
    raiseTimeoutConfig(backlog);

catch (RuntimeException e) {
    if (e.getMessage().contains("exeute waiting transactions")) {
        increaseExecuteWaitingOnStartTimeout();
        restartNode();
    } else throw e;
}

When it happens

Trigger: Node startup with pending Accord transactions to execute, where the execution future does not complete within the configured timeout seconds and execute_waiting_on_start_fail_on_timeout=true.

Common situations: Node starting under heavy load with a large backlog of waiting Accord transactions, slow storage delaying replay/execution, or a too-short execute_waiting_on_start_timeout value.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Could not catchup with peers
• Gave up waiting on journal index to be ready
• accord.journal_directory must not be the same as the commitl
• hints_directory must not be the same as the accord.journal_d
• saved_caches_directory must not be the same as the accord.jo

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/5852bfdab7179ec6. Report an issue: GitHub.