What causes "Session keeper service did not terminate in time" in apache/druid?

Verify Consul connectivity/latency at shutdown time; fix agent or network stalls.

Session keeper service did not terminate in time

Error message

Session keeper service did not terminate in time

What it means

Warning logged during unregisterListener() when the session-keeper executor fails to terminate within the 5-second awaitTermination window after shutdownNow(). The keeper loop renews the Consul session on a schedule; if it is mid-renewal in a slow RPC or sleeping, it may outlive the wait, leaving a lingering thread that exits on its next interruptible point.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:209

      }

      if (executorService != null) {
        executorService.shutdownNow();
        try {
          if (!executorService.awaitTermination(5, TimeUnit.SECONDS)) {
            LOGGER.warn("Leader selector executor did not terminate in time");
          }
        }
        catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      }

      if (sessionKeeperService != null) {
        sessionKeeperService.shutdownNow();
        try {
          if (!sessionKeeperService.awaitTermination(5, TimeUnit.SECONDS)) {
            LOGGER.warn("Session keeper service did not terminate in time");
          }
        }
        catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      }
    }
    finally {
      lifecycleLock.exitStop();
    }
  }

  private void startLeaderElection()
  {
    executorService.submit(this::leaderElectionLoop);
    sessionKeeperService.submit(this::sessionKeeperLoop);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify Consul connectivity/latency at shutdown time; fix agent or network stalls.
Lower healthCheckInterval so scheduled renewal tasks are short and frequent.
Accept the warning if benign: shutdownNow() interrupted the task and it will terminate at the next interruptible call.

Example fix

// before
druid.discovery.consul.service.healthCheckInterval=PT2M
// after
druid.discovery.consul.service.healthCheckInterval=PT15S

Defensive patterns

Strategy: try-catch

Try / catch

try {
  selector.unregisterListener();
} catch (RuntimeException e) {
  LOGGER.warn(e, "Leader selector stop reported issues; keeper thread will exit at next interrupt point");
}

Prevention

Keep healthCheckInterval short relative to TTL
Avoid shutting down while Consul is known to be unreachable
Gracefully destroy sessions before executor shutdown where possible

When it happens

Trigger: Calling unregisterListener() while sessionKeeperLoop is blocked in consulClient.renewSession() against a slow/unreachable Consul, or in the scheduled-delay before its next renewal run.

Common situations: Network partitions during service shutdown, Consul agent stalls, or healthCheckInterval large enough that the scheduled renewal task sits queued beyond the 5s wait.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4a5d4e173924be6c. Report an issue: GitHub.