apache/cassandra · error · TimeoutException

${executor.name} not terminated

Error message

${executor.name} not terminated

What it means

SharedExecutorPool.shutdownAndWait waits up to the given timeout for each SEPExecutor to terminate after shutdown; if an executor still has live workers when the deadline expires, it throws TimeoutException naming the executor. Called from test cleanup (e.g. testLocalStatePropagation), it means workers did not finish draining within the allotted time.

Source

Thrown at src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java:163

        SEPExecutor executor = new SEPExecutor(this, maxConcurrency, maximumPoolSizeListener, jmxPath, name);
        executors.add(executor);
        return executor;
    }

    public synchronized void shutdownAndWait(long timeout, TimeUnit unit) throws InterruptedException, TimeoutException
    {
        shuttingDown = true;
        for (SEPExecutor executor : executors)
            executor.shutdownNow();

        terminateWorkers();

        long until = nanoTime() + unit.toNanos(timeout);
        for (SEPExecutor executor : executors)
        {
            executor.shutdown.await(until - nanoTime(), TimeUnit.NANOSECONDS);
            if (!executor.isTerminated())
                throw new TimeoutException(executor.name + " not terminated");
        }
    }

    void terminateWorkers()
    {
        assert shuttingDown;

        // To terminate our workers, we only need to unpark thread to make it runnable again,
        // so that the pool.shuttingDown boolean is checked. If work was already in the process
        // of being scheduled, worker will terminate upon running the task.
        Map.Entry<Long, SEPWorker> e;
        while (null != (e = descheduled.pollFirstEntry()))
            e.getValue().assign(Work.SPINNING, false);

        while (null != (e = spinning.pollFirstEntry()))
            LockSupport.unpark(e.getValue().thread);
    }
}

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Ensure all tasks submitted to the executor complete or are cancelled before shutdownAndWait; drain pending work in test teardown
2. Check for workers blocked on external resources (network, locks, incomplete futures) and unblock them before shutdown
3. Increase the shutdown timeout to cover realistically slow teardown
4. If a task is genuinely stuck, this indicates a bug — take a thread dump to find the blocked worker and fix the underlying wait

Example fix

// before
sharedPool.shutdownAndWait(1, TimeUnit.SECONDS); // TimeoutException: ... not terminated
// after
sharedPool.shutdownAndWait(30, TimeUnit.SECONDS); // allow workers to drain

if (!executor.isTerminated())
    logger.warn("executor still running tasks before shutdownAndWait");

try
{
    sharedPool.shutdownAndWait(30, TimeUnit.SECONDS);
}
catch (TimeoutException e)
{
    logger.error("Executor did not terminate: " + e.getMessage());
    // take thread dump / force cleanup
}

When it happens

Trigger: Calling shutdownAndWait(timeout) (directly or via test-cluster teardown) while SEP executors still have queued tasks or workers blocked on something (I/O, a missing signal, a leaked task refusing to finish) past the timeout.

Common situations: dtest/JVM-test teardown hanging because a test left long-running or blocked tasks on an executor; timeouts set too short for the pending work; a worker stuck waiting on a promise/future never completed by the test.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Unable to terminate fast tasks within 1 minute.
• Failed to submit background compaction task
• {} timeout awaiting durability: {}
• UnsupportedOperationException
• Cannot resize core pool size of SEPExecutor

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/aa3855f5157c0339. Report an issue: GitHub.