apache/druid · warning

Leader selector executor did not terminate in time

Error message

Leader selector executor did not terminate in time

What it means

Warning logged during unregisterListener() when the leader-election executor does not finish shutting down within the 5-second awaitTermination window after shutdownNow(). It means the election loop was likely blocked in a slow Consul RPC or a long sleep, so the JVM may retain a lingering thread after service stop.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:197

        leader.set(false);
      }

      // Destroying session releases the Consul lock, allowing another node to become leader
      if (sessionId != null) {
        try {
          consulClient.sessionDestroy(sessionId, buildQueryParams(), config.getAuth().getAclToken());
        }
        catch (Exception e) {
          LOGGER.error(e, "Failed to destroy Consul session");
        }
        sessionId = null;
      }

      if (executorService != null) {
        executorService.shutdownNow();
        try {
          if (!executorService.awaitTermination(5, TimeUnit.SECONDS)) {
            LOGGER.warn("Leader selector executor did not terminate in time");
          }
        }
        catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      }

      if (sessionKeeperService != null) {
        sessionKeeperService.shutdownNow();
        try {
          if (!sessionKeeperService.awaitTermination(5, TimeUnit.SECONDS)) {
            LOGGER.warn("Session keeper service did not terminate in time");
          }
        }
        catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Ensure Consul is reachable and responding before shutdown; fix network/agent issues that stall RPCs.
2. Reduce leaderRetryBackoffMax and healthCheckInterval so loop sleeps stay under the 5s termination window.
3. Accept the warning if benign — shutdownNow() already interrupted the thread and it will exit at the next interruptible point.

Example fix

// before (long backoff stalls shutdown)
druid.discovery.consul.leader.retryBackoffMax=PT5M
// after
druid.discovery.consul.leader.retryBackoffMax=PT10S

try {
  selector.unregisterListener();
} finally {
  // executor threads exit asynchronously after shutdownNow(); do not assume immediate termination
}

When it happens

Trigger: Calling unregisterListener() (service shutdown) while leaderElectionLoop is blocked in a Consul HTTP call (long network timeout or unreachable Consul) or in a Thread.sleep backoff longer than 5 seconds.

Common situations: Decommissioning a node with an unreachable/slow Consul agent, very large leaderRetryBackoffMax or healthCheckInterval causing long sleeps, or network partitions during rolling restarts.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Session keeper service did not terminate in time
• Health check executor did not terminate in time
• Listener executor did not terminate in time
• Failed to stop watchExecutor for role[%s]
• can't stop.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c92dc1625480a1dc. Report an issue: GitHub.