{"record":{"id":"479e53d4c984f817","repo":"apache/cassandra","slug":"timeout-awaiting-durability","errorCode":null,"errorMessage":"{} timeout awaiting durability: {}","messagePattern":"(.+?) timeout awaiting durability: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/service/accord/AccordCommandStores.java","lineNumber":263,"sourceCode":"        for (ShardHolder shard : current())\n        {\n            AccordCommandStore commandStore = (AccordCommandStore) shard.store;\n            tableIds.add(commandStore.tableId());\n            async.add(commandStore.shutdownAsync());\n        }\n        if (!async.isEmpty())\n            AccordService.getBlocking(AsyncResults.reduce(async, Reduce.toNull()));\n        return tableIds;\n    }\n\n    public boolean awaitStoreTermination(long deadlineNanos)\n    {\n        for (ShardHolder shard : current())\n        {\n            AccordCommandStore commandStore = (AccordCommandStore) shard.store;\n            if (!commandStore.awaitTerminationUntil(deadlineNanos))\n            {\n                logger.warn(\"{} timeout awaiting durability: {}\", commandStore, DurablyAppliedTo.summarise(commandStore.unsafeGetRedundantBefore()));\n                return false;\n            }\n        }\n        return true;\n    }\n\n    public void shutdownExecutors()\n    {\n        for (AccordExecutor executor : executors)\n            executor.shutdown();\n    }\n\n    @Override\n    public void shutdown()\n    {\n        shutdownStores();\n        shutdownExecutors();\n    }","sourceCodeStart":245,"sourceCodeEnd":281,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/accord/AccordCommandStores.java#L245-L281","documentation":"awaitStoreTermination waited for an AccordCommandStore to reach durability (bounded by deadlineNanos) and timed out. It logs the redundant-before summary and returns false, indicating shutdown/durability could not be confirmed within the deadline.","triggerScenarios":"safeShutdown calls awaitStoreTermination with a deadline; a command store's async durability (flush/journal fsync) does not complete before deadlineNanos elapses.","commonSituations":"Slow or saturated disks preventing fsync completion under deadline; heavy Accord transaction load during shutdown; very short shutdown grace period.","solutions":["Increase the shutdown durability deadline / grace period","Check disk latency and journal fsync times; resolve I/O saturation","Retry shutdown after load subsides; verify journal state on next startup"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (TimeoutException e) { /* extend deadline and retry safeShutdown */ safeShutdown(deadlineNanos * 2); }","preventionTips":["Set generous shutdown grace periods for Accord durability","Monitor disk latency; avoid shutdown under heavy write load","Reduce Accord traffic before planned shutdowns"],"tags":["accord","durability","timeout","shutdown"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}