apache/cassandra · error · RejectedExecutionException

ScheduledThreadPoolExecutor has shut down.

Error message

ScheduledThreadPoolExecutor has shut down.

What it means

When a task is submitted to a ScheduledThreadPoolExecutorPlus that has already been shut down, the rejected-execution handler throws RejectedExecutionException — but only if the whole Cassandra node (StorageService) is NOT shutting down. If the node itself is going down, the rejection is treated as expected: the task is cancelled and only logged. This distinguishes an application bug (submitting after executor shutdown) from normal node shutdown.

Source

Thrown at src/java/org/apache/cassandra/concurrent/ScheduledThreadPoolExecutorPlus.java:63

 *
 * Catches exceptions during Task execution so that they don't suppress subsequent invocations of the task.
 *
 * Finally, there is a special rejected execution handler for tasks rejected during the shutdown hook.
 *  - For fire and forget tasks (like ref tidy) we can safely ignore the exceptions.
 *  - For any callers that care to know their task was rejected we cancel passed task.
 */
public class ScheduledThreadPoolExecutorPlus extends ScheduledThreadPoolExecutor implements ScheduledExecutorPlus
{
    private static final Logger logger = LoggerFactory.getLogger(ScheduledThreadPoolExecutorPlus.class);
    private static final TaskFactory taskFactory = TaskFactory.standard();

    public static final RejectedExecutionHandler rejectedExecutionHandler = (task, executor) ->
    {
        if (executor.isShutdown())
        {
            // TODO: this sequence of events seems poorly thought out
            if (!StorageService.instance.isShutdown())
                throw new RejectedExecutionException("ScheduledThreadPoolExecutor has shut down.");

            //Give some notification to the caller the task isn't going to run
            if (task instanceof java.util.concurrent.Future)
                ((java.util.concurrent.Future<?>) task).cancel(false);

            logger.debug("ScheduledThreadPoolExecutor has shut down as part of C* shutdown");
        }
        else
        {
            throw new AssertionError("Unknown rejection of ScheduledThreadPoolExecutor task");
        }
    };

    ScheduledThreadPoolExecutorPlus(NamedThreadFactory threadFactory)
    {
        super(1, threadFactory);
        setRejectedExecutionHandler(rejectedExecutionHandler);
    }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Ensure the executor is only shut down when the owning component stops submitting to it; coordinate lifecycle so shutdown happens last
2. Recreate the executor (or never shut it down) if submissions continue — ScheduledThreadPoolExecutorPlus instances are typically global and node-lifecycle-scoped
3. Catch RejectedExecutionException at the submit site and treat it as 'executor stopped' for the caller
4. If seen during node shutdown, verify StorageService.isShutdown() returns true; if the node is mid-shutdown this message is expected noise

Example fix

// before
scheduledExecutor.shutdown();
scheduledExecutor.schedule(task, 1, SECONDS); // RejectedExecutionException
// after
if (!scheduledExecutor.isShutdown())
    scheduledExecutor.schedule(task, 1, SECONDS);

if (executor.isShutdown())
    throw new IllegalStateException("cannot schedule on shut-down executor");

static boolean canSchedule(ScheduledThreadPoolExecutorPlus e)
{
    return !e.isShutdown();
}

try
{
    executor.schedule(task, delay, unit);
}
catch (RejectedExecutionException e)
{
    // executor shut down; drop the task or route it elsewhere
    logger.info("Task dropped: executor shut down");
}

When it happens

Trigger: Submitting a scheduled task after shutdown() was called on the executor while StorageService.isShutdown() is false; lifecycle races where a component keeps posting periodic tasks to an executor that another component already shut down.

Common situations: Component ordering bugs during partial shutdown (one subsystem stops its executor while others still submit); restarting a service in-process after shutdown without recreating the executor; tests that shut down executors but leave scheduled jobs registered.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Scheduler has shut down.
• SASL negotiation not complete
• MBeanServer already initialized
• UnsupportedOperationException
• Cannot resize core pool size of SEPExecutor

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ac6f97bcd129f9b6. Report an issue: GitHub.