{"record":{"id":"f7400edefe39afd6","repo":"apache/cassandra","slug":"caught-exception-while-waiting-for-memtable-flushe","errorCode":null,"errorMessage":"Caught exception while waiting for memtable flushes during shutdown hook","messagePattern":"Caught exception while waiting for memtable flushes during shutdown hook","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/service/StorageService.java","lineNumber":4013,"sourceCode":"            for (Keyspace keyspace : Keyspace.nonLocalStrategy())\n            {\n                for (ColumnFamilyStore cfs : keyspace.getColumnFamilyStores())\n                    flushes.add(cfs.forceFlush(ColumnFamilyStore.FlushReason.DRAIN));\n            }\n            // wait for the flushes.\n            // TODO this is a godawful way to track progress, since they flush in parallel.  a long one could\n            // thus make several short ones \"instant\" if we wait for them later.\n            for (Future f : flushes)\n            {\n                try\n                {\n                    FBUtilities.waitOnFuture(f);\n                }\n                catch (Throwable t)\n                {\n                    JVMStabilityInspector.inspectThrowable(t);\n                    // don't let this stop us from shutting down the commitlog and other thread pools\n                    logger.warn(\"Caught exception while waiting for memtable flushes during shutdown hook\", t);\n                }\n\n                remainingCFs--;\n            }\n\n            // Interrupt ongoing compactions and shutdown CM to prevent further compactions.\n            CompactionManager.instance.forceShutdown();\n            // Flush the system tables after all other tables are flushed, just in case flushing modifies any system state\n            // like CASSANDRA-5151. Don't bother with progress tracking since system data is tiny.\n            // Flush system tables after stopping compactions since they modify\n            // system tables (for example compactions can obsolete sstables and the tidiers in SSTableReader update\n            // system tables, see SSTableReader.GlobalTidy)\n            flushes.clear();\n            for (Keyspace keyspace : Keyspace.system())\n            {\n                for (ColumnFamilyStore cfs : keyspace.getColumnFamilyStores())\n                    flushes.add(cfs.forceFlush(ColumnFamilyStore.FlushReason.DRAIN));\n            }","sourceCodeStart":3995,"sourceCodeEnd":4031,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/StorageService.java#L3995-L4031","documentation":"During the shutdown hook, waiting on a memtable flush future threw; the exception is inspected for JVM-stability and then logged as a warning so shutdown can proceed to commitlog shutdown. It means a flush failed during shutdown, risking data in memtables not being persisted.","triggerScenarios":"Waiting for remaining memtable flushes during JVM shutdown when the flush throws - disk full, I/O error, or executor already shut down mid-flush.","commonSituations":"Disk-full nodes shutting down; failing disk/RAID issues; commit log or flush executor already terminated causing flush futures to complete exceptionally.","solutions":["Inspect the logged cause; fix disk space/I-O issues before restarting","Check system.log for the flush failure and repair the affected SSTables","If data loss in the affected memtable is acceptable, restart normally"],"exampleFix":"// before\n# shutdown warning, disk at 100%\ndf -h  # /var/lib/cassandra 100%\n// after\n# free space, then restart so commitlog replays safely\ndf -h  # free space available","handlingStrategy":"try-catch","validationCode":"// pre-shutdown check\nif (Flushers are pending && diskUsage > 0.95) log.warn(\"flush may fail during shutdown\");","typeGuard":null,"tryCatchPattern":"catch (Throwable t) { JVMStabilityInspector.inspectThrowable(t); /* proceed with shutdown; verify data on restart */ }","preventionTips":["Monitor disk space to keep headroom for flushes","Drain gracefully (nodetool drain) before stopping the process","Alert on flush failures in system.log"],"tags":["memtable","flush","shutdown","disk-io"],"backgroundTag":"database-write-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}