apache/pulsar · error · IllegalStateException

Executor %s should have been shutdown before entering the te

Error message

Executor %s should have been shutdown before entering the termination handler.

What it means

GracefulExecutorServicesTerminationHandler is constructed (via GracefulExecutorServicesShutdown) to await termination of executors that were already shut down. Its constructor validates every ExecutorService with isShutdown() and throws IllegalStateException if any is still running, because awaiting termination of a non-shutdown executor would be a caller bug.

Source

Thrown at pulsar-common/src/main/java/org/apache/pulsar/common/util/GracefulExecutorServicesTerminationHandler.java:56

@CustomLog
class GracefulExecutorServicesTerminationHandler {
    private static final long SHUTDOWN_THREAD_COMPLETION_TIMEOUT_NANOS = Duration.ofMillis(100L).toNanos();
    private final List<ExecutorService> executors;
    private final CompletableFuture<Void> future;
    private final Duration shutdownTimeout;
    private final Duration terminationTimeout;
    private final CountDownLatch shutdownThreadCompletedLatch = new CountDownLatch(1);

    GracefulExecutorServicesTerminationHandler(Duration shutdownTimeout, Duration terminationTimeout,
                                               List<ExecutorService> executorServices) {
        this.shutdownTimeout = shutdownTimeout;
        this.terminationTimeout = terminationTimeout;
        this.executors = Collections.unmodifiableList(new ArrayList<>(executorServices));
        this.future = new CompletableFuture<>();
        log.info().attr("executorCount", executors.size()).log("Starting termination handler");
        for (ExecutorService executor : executors) {
            if (!executor.isShutdown()) {
                throw new IllegalStateException(
                        String.format("Executor %s should have been shutdown before entering the termination handler.",
                                executor));
            }
        }
        if (haveExecutorsBeenTerminated()) {
            markShutdownCompleted();
        } else {
            if (shutdownTimeout.isZero() || shutdownTimeout.isNegative()) {
                terminateExecutors();
                markShutdownCompleted();
            } else {
                Thread shutdownWaitingThread = new Thread(this::awaitShutdown, getClass().getSimpleName());
                shutdownWaitingThread.setDaemon(false);
                shutdownWaitingThread.setUncaughtExceptionHandler((thread, exception) -> {
                  log.error().attr("thread", thread).exception(exception)
                          .log("Uncaught exception in shutdown thread");
                });
                shutdownWaitingThread.start();

View on GitHub (pinned to 820761864e)

Solutions

1. Call executor.shutdown() (or shutdownNow()) on every executor before passing it to the graceful shutdown helper.
2. Use GracefulExecutorServicesShutdown.shutdown(...) as the single entry point so shutdown and termination tracking happen together.
3. Audit the list of executors being passed and assert each reports isShutdown() before constructing the handler.

Example fix

// before
GracefulExecutorServicesShutdown.shutdown(Duration.ofSeconds(30), List.of(ioExecutor)); // throws: not shutdown
// after
ioExecutor.shutdown();
GracefulExecutorServicesShutdown.shutdown(Duration.ofSeconds(30), List.of(ioExecutor));

static void requireAllShutdown(Collection<ExecutorService> executors) {
    List<ExecutorService> running = executors.stream()
            .filter(e -> !e.isShutdown())
            .collect(Collectors.toList());
    if (!running.isEmpty()) {
        running.forEach(ExecutorService::shutdown); // or throw, per policy
    }
}

try {
    GracefulExecutorServicesShutdown.shutdown(timeout, executors).get();
} catch (IllegalStateException e) {
    if (e.getMessage().contains("should have been shutdown")) {
        executors.forEach(ExecutorService::shutdownNow);
    }
    throw e;
}

When it happens

Trigger: Calling GracefulExecutorServicesShutdown.shutdown(...) (which builds this handler) with an ExecutorService on which shutdown()/shutdownNow() was never called, e.g. mixing graceful shutdown helpers with one manually managed executor.

Common situations: Refactoring shutdown code to use GracefulExecutorServicesShutdown while forgetting to call shutdown() on one of the pooled executors; passing executors in the wrong order (handler built before shutdown); Pulsar broker/proxy shutdown code paths during development.

Related errors

• ${e}
• ProducerCache is already closed
• Timeout during managed ledger close
• ManagedLedgerFactory is already closed.
• ManagedLedger ${name} has already been closed

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/5e27ccb36e8ae49f. Report an issue: GitHub.