apache/cassandra · error · RejectedExecutionException

ScheduledThreadPoolExecutor has shut down.

Error message

ScheduledThreadPoolExecutor has shut down.

What it means

When a task is submitted to a ScheduledThreadPoolExecutorPlus that has already been shut down, the rejected-execution handler throws RejectedExecutionException — but only if the whole Cassandra node (StorageService) is NOT shutting down. If the node itself is going down, the rejection is treated as expected: the task is cancelled and only logged. This distinguishes an application bug (submitting after executor shutdown) from normal node shutdown.

Solutions

1. Ensure the executor is only shut down when the owning component stops submitting to it; coordinate lifecycle so shutdown happens last
2. Recreate the executor (or never shut it down) if submissions continue — ScheduledThreadPoolExecutorPlus instances are typically global and node-lifecycle-scoped
3. Catch RejectedExecutionException at the submit site and treat it as 'executor stopped' for the caller
4. If seen during node shutdown, verify StorageService.isShutdown() returns true; if the node is mid-shutdown this message is expected noise

Example fix

// before
scheduledExecutor.shutdown();
scheduledExecutor.schedule(task, 1, SECONDS); // RejectedExecutionException
// after
if (!scheduledExecutor.isShutdown())
    scheduledExecutor.schedule(task, 1, SECONDS);

if (executor.isShutdown())
    throw new IllegalStateException("cannot schedule on shut-down executor");

static boolean canSchedule(ScheduledThreadPoolExecutorPlus e)
{
    return !e.isShutdown();
}

try
{
    executor.schedule(task, delay, unit);
}
catch (RejectedExecutionException e)
{
    // executor shut down; drop the task or route it elsewhere
    logger.info("Task dropped: executor shut down");
}

When it happens

Trigger: Submitting a scheduled task after shutdown() was called on the executor while StorageService.isShutdown() is false; lifecycle races where a component keeps posting periodic tasks to an executor that another component already shut down.

Common situations: Component ordering bugs during partial shutdown (one subsystem stops its executor while others still submit); restarting a service in-process after shutdown without recreating the executor; tests that shut down executors but leave scheduled jobs registered.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Scheduler has shut down.
• Attempted to release storage-attached index segment builder…
• Bootstrap can be started exactly once, but seems to have…
• Cannot resize core pool size of SEPExecutor
• commit log header has not been written

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ac6f97bcd129f9b6. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/concurrent/ScheduledThreadPoolExecutorPlus.java:63

 *
 * Catches exceptions during Task execution so that they don't suppress subsequent invocations of the task.
 *
 * Finally, there is a special rejected execution handler for tasks rejected during the shutdown hook.
 *  - For fire and forget tasks (like ref tidy) we can safely ignore the exceptions.
 *  - For any callers that care to know their task was rejected we cancel passed task.
 */
public class ScheduledThreadPoolExecutorPlus extends ScheduledThreadPoolExecutor implements ScheduledExecutorPlus
{
    private static final Logger logger = LoggerFactory.getLogger(ScheduledThreadPoolExecutorPlus.class);
    private static final TaskFactory taskFactory = TaskFactory.standard();

    public static final RejectedExecutionHandler rejectedExecutionHandler = (task, executor) ->
    {
        if (executor.isShutdown())
        {
            // TODO: this sequence of events seems poorly thought out
            if (!StorageService.instance.isShutdown())
                throw new RejectedExecutionException("ScheduledThreadPoolExecutor has shut down.");

            //Give some notification to the caller the task isn't going to run
            if (task instanceof java.util.concurrent.Future)
                ((java.util.concurrent.Future<?>) task).cancel(false);

            logger.debug("ScheduledThreadPoolExecutor has shut down as part of C* shutdown");
        }
        else
        {
            throw new AssertionError("Unknown rejection of ScheduledThreadPoolExecutor task");
        }
    };

    ScheduledThreadPoolExecutorPlus(NamedThreadFactory threadFactory)
    {
        super(1, threadFactory);
        setRejectedExecutionHandler(rejectedExecutionHandler);
    }

View on GitHub (pinned to 88fd0f6a0e)