What causes "Unable to terminate fast tasks within 1 minute." in apache/cassandra?

Inspect the attached stack trace (t) to identify the stuck task class.

Unable to terminate fast tasks within 1 minute.

Error message

Unable to terminate fast tasks within 1 minute.

What it means

WARN logged in StorageService.runMayThrow's shutdown hook when ExecutorUtils.shutdownNowAndWait(1, MINUTES, ScheduledExecutors.scheduledFastTasks) does not finish within its 1-minute timeout. Some fast scheduled tasks refused to terminate promptly at JVM shutdown, so shutdown proceeds with those tasks still running or interrupted; the original Throwable is attached.

Source

Thrown at src/java/org/apache/cassandra/service/StorageService.java:766

        {
            throw new AssertionError(e);
        }

        // daemon threads, like our executors', continue to run while shutdown hooks are invoked
        drainOnShutdown = NamedThreadFactory.createThread(new WrappedRunnable()
        {
            @Override
            public void runMayThrow() throws InterruptedException, ExecutionException, IOException
            {
                drain(true);
                try
                {
                    ExecutorUtils.shutdownNowAndWait(1, MINUTES, ScheduledExecutors.scheduledFastTasks);
                    logger.info("Cassandra shutdown complete");
                }
                catch (Throwable t)
                {
                    logger.warn("Unable to terminate fast tasks within 1 minute.", t);
                }
                finally
                {
                    LoggingSupportFactory.getLoggingSupport().onShutdown();
                }
            }
        }, "StorageServiceShutdownHook");
        Runtime.getRuntime().addShutdownHook(drainOnShutdown);

        Schema.instance.saveSystemKeyspace();
        DatabaseDescriptor.getInternodeAuthenticator().setupInternode();

        if (ClusterMetadataService.state() == ClusterMetadataService.State.GOSSIP)
        {
            // register listener before starting gossiper to avoid missing messages
            Gossiper.instance.register(new GossipCMSListener());
        }
        sstablesTracker.register((notification, o) -> {

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Inspect the attached stack trace (t) to identify the stuck task class.
Increase shutdown grace period (e.g. container stop timeout) and retry shutdown.
Take a thread dump during shutdown (jstack) to find blocking code.
If a specific task consistently hangs, report/investigate that executor's task; workaround by stopping dependent subsystems first ('nodetool drain').

Defensive patterns

Strategy: try-catch

Try / catch

// shutdown hook already catches Throwable; treat WARN as signal to inspect hung executors
try {
    ExecutorUtils.shutdownNowAndWait(1, MINUTES, scheduledFastTasks);
} catch (Throwable t) {
    logger.warn("Unable to terminate fast tasks within 1 minute.", t);
}

Prevention

Take thread dumps during slow shutdowns to find hung tasks.
Increase container/process stop timeout beyond task worst case.
Run 'nodetool drain' before shutdown to quiesce work.

When it happens

Trigger: JVM shutdown hook runs (SIGTERM / daemon.decommission) and scheduledFastTasks executor fails to terminate within the 1-minute wait — typically tasks blocked on slow I/O, long compaction-adjacent work, or deadlocks.

Common situations: Graceful shutdown under heavy load (draining, large memtable flushes), container stop timeouts, hung worker threads blocking the scheduled executor.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/5caaa26a1787461c. Report an issue: GitHub.