apache/cassandra · warning

{} timeout awaiting durability: {}

Error message

{} timeout awaiting durability: {}

What it means

awaitStoreTermination waited for an AccordCommandStore to reach durability (bounded by deadlineNanos) and timed out. It logs the redundant-before summary and returns false, indicating shutdown/durability could not be confirmed within the deadline.

Source

Thrown at src/java/org/apache/cassandra/service/accord/AccordCommandStores.java:263

        for (ShardHolder shard : current())
        {
            AccordCommandStore commandStore = (AccordCommandStore) shard.store;
            tableIds.add(commandStore.tableId());
            async.add(commandStore.shutdownAsync());
        }
        if (!async.isEmpty())
            AccordService.getBlocking(AsyncResults.reduce(async, Reduce.toNull()));
        return tableIds;
    }

    public boolean awaitStoreTermination(long deadlineNanos)
    {
        for (ShardHolder shard : current())
        {
            AccordCommandStore commandStore = (AccordCommandStore) shard.store;
            if (!commandStore.awaitTerminationUntil(deadlineNanos))
            {
                logger.warn("{} timeout awaiting durability: {}", commandStore, DurablyAppliedTo.summarise(commandStore.unsafeGetRedundantBefore()));
                return false;
            }
        }
        return true;
    }

    public void shutdownExecutors()
    {
        for (AccordExecutor executor : executors)
            executor.shutdown();
    }

    @Override
    public void shutdown()
    {
        shutdownStores();
        shutdownExecutors();
    }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Increase the shutdown durability deadline / grace period
2. Check disk latency and journal fsync times; resolve I/O saturation
3. Retry shutdown after load subsides; verify journal state on next startup

catch (TimeoutException e) { /* extend deadline and retry safeShutdown */ safeShutdown(deadlineNanos * 2); }

When it happens

Trigger: safeShutdown calls awaitStoreTermination with a deadline; a command store's async durability (flush/journal fsync) does not complete before deadlineNanos elapses.

Common situations: Slow or saturated disks preventing fsync completion under deadline; heavy Accord transaction load during shutdown; very short shutdown grace period.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• ${executor.name} not terminated
• Timeout waiting to exeute waiting transactions
• Could not catchup with peers
• Scheduler has shut down.
• Gave up waiting on journal index to be ready

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/479e53d4c984f817. Report an issue: GitHub.