Caught exception while waiting for memtable flushes during s

Error message

Caught exception while waiting for memtable flushes during shutdown hook

What it means

During the shutdown hook, waiting on a memtable flush future threw; the exception is inspected for JVM-stability and then logged as a warning so shutdown can proceed to commitlog shutdown. It means a flush failed during shutdown, risking data in memtables not being persisted.

Source

Thrown at src/java/org/apache/cassandra/service/StorageService.java:4013

            for (Keyspace keyspace : Keyspace.nonLocalStrategy())
            {
                for (ColumnFamilyStore cfs : keyspace.getColumnFamilyStores())
                    flushes.add(cfs.forceFlush(ColumnFamilyStore.FlushReason.DRAIN));
            }
            // wait for the flushes.
            // TODO this is a godawful way to track progress, since they flush in parallel.  a long one could
            // thus make several short ones "instant" if we wait for them later.
            for (Future f : flushes)
            {
                try
                {
                    FBUtilities.waitOnFuture(f);
                }
                catch (Throwable t)
                {
                    JVMStabilityInspector.inspectThrowable(t);
                    // don't let this stop us from shutting down the commitlog and other thread pools
                    logger.warn("Caught exception while waiting for memtable flushes during shutdown hook", t);
                }

                remainingCFs--;
            }

            // Interrupt ongoing compactions and shutdown CM to prevent further compactions.
            CompactionManager.instance.forceShutdown();
            // Flush the system tables after all other tables are flushed, just in case flushing modifies any system state
            // like CASSANDRA-5151. Don't bother with progress tracking since system data is tiny.
            // Flush system tables after stopping compactions since they modify
            // system tables (for example compactions can obsolete sstables and the tidiers in SSTableReader update
            // system tables, see SSTableReader.GlobalTidy)
            flushes.clear();
            for (Keyspace keyspace : Keyspace.system())
            {
                for (ColumnFamilyStore cfs : keyspace.getColumnFamilyStores())
                    flushes.add(cfs.forceFlush(ColumnFamilyStore.FlushReason.DRAIN));
            }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Inspect the logged cause; fix disk space/I-O issues before restarting
Check system.log for the flush failure and repair the affected SSTables
If data loss in the affected memtable is acceptable, restart normally

Example fix

// before
# shutdown warning, disk at 100%
df -h  # /var/lib/cassandra 100%
// after
# free space, then restart so commitlog replays safely
df -h  # free space available

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-shutdown check
if (Flushers are pending && diskUsage > 0.95) log.warn("flush may fail during shutdown");

Try / catch

catch (Throwable t) { JVMStabilityInspector.inspectThrowable(t); /* proceed with shutdown; verify data on restart */ }

Prevention

Monitor disk space to keep headroom for flushes
Drain gracefully (nodetool drain) before stopping the process
Alert on flush failures in system.log

When it happens

Trigger: Waiting for remaining memtable flushes during JVM shutdown when the flush throws - disk full, I/O error, or executor already shut down mid-flush.

Common situations: Disk-full nodes shutting down; failing disk/RAID issues; commit log or flush executor already terminated causing flush futures to complete exceptionally.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/f7400edefe39afd6. Report an issue: GitHub.