What causes "Listener executor did not terminate in time" in apache/druid?

Make listener callbacks fast/non-blocking; offload heavy work to your own executor.

Listener executor did not terminate in time

Error message

Listener executor did not terminate in time

What it means

Emitted in ConsulDruidNodeDiscoveryProvider.stop when the listener executor does not terminate within 10 seconds after shutdown(). The code then calls shutdownNow() to force interruption. This indicates listener callbacks were still running during provider shutdown.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java:169

  @LifecycleStop
  public void stop()
  {
    if (!lifecycleLock.canStop()) {
      throw new ISE("can't stop.");
    }

    LOGGER.info("Stopping ConsulDruidNodeDiscoveryProvider");

    for (NodeRoleWatcher watcher : nodeRoleWatchers.values()) {
      watcher.stop();
    }
    nodeRoleWatchers.clear();

    // Watcher threads must finish before shutting down listener executor to avoid RejectedExecutionException
    try {
      listenerExecutor.shutdown();
      if (!listenerExecutor.awaitTermination(10, TimeUnit.SECONDS)) {
        LOGGER.warn("Listener executor did not terminate in time");
        listenerExecutor.shutdownNow();
      }
    }
    catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      LOGGER.warn("Interrupted while waiting for listener executor termination");
      listenerExecutor.shutdownNow();
    }

    LOGGER.info("Stopped ConsulDruidNodeDiscoveryProvider");
    lifecycleLock.exitStopAndReset();
  }

  static class NodeRoleWatcher implements DruidNodeDiscovery
  {
    private static final Logger LOGGER = new Logger(NodeRoleWatcher.class);

    private final ConsulApiClient consulApiClient;

View on GitHub (pinned to 9b90983fd2)

Solutions

Make listener callbacks fast/non-blocking; offload heavy work to your own executor.
Check for deadlocks or long waits inside registered DruidNodeDiscoverySubscriber callbacks.
If callbacks legitimately take longer, extend the 10-second timeout.
Ensure watchers are stopped before provider stop() as the code expects (normal lifecycle does this).

Example fix

// before: slow work inside listener callback
public void nodeAdded(DiscoveryDruidNode node) { queryConsulLongOp(node); }
// after
public void nodeAdded(DiscoveryDruidNode node) { workerExecutor.submit(() -> queryConsulLongOp(node)); }

Defensive patterns

Strategy: try-catch

Try / catch

try {
  provider.stop();
} finally {
  // confirm listeners were delivered or document missed callbacks
}

Prevention

Keep listener callbacks fast and non-blocking
Offload heavy processing to a separate executor
Avoid locks inside subscriber callbacks

When it happens

Trigger: stop() invoked while a discovery listener callback is long-running or blocked (e.g. slow handler, lock contention), exceeding the 10-second awaitTermination window.

Common situations: A subscriber callback doing slow I/O or waiting on a lock; JVM under heavy load during shutdown; a watcher thread not fully finished before listener executor shutdown (the code orders these deliberately to avoid RejectedExecutionException).

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/58328e347efc2d93. Report an issue: GitHub.