{"record":{"id":"c92dc1625480a1dc","repo":"apache/druid","slug":"leader-selector-executor-did-not-terminate-in-time","errorCode":null,"errorMessage":"Leader selector executor did not terminate in time","messagePattern":"Leader selector executor did not terminate in time","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":197,"sourceCode":"        leader.set(false);\n      }\n\n      // Destroying session releases the Consul lock, allowing another node to become leader\n      if (sessionId != null) {\n        try {\n          consulClient.sessionDestroy(sessionId, buildQueryParams(), config.getAuth().getAclToken());\n        }\n        catch (Exception e) {\n          LOGGER.error(e, \"Failed to destroy Consul session\");\n        }\n        sessionId = null;\n      }\n\n      if (executorService != null) {\n        executorService.shutdownNow();\n        try {\n          if (!executorService.awaitTermination(5, TimeUnit.SECONDS)) {\n            LOGGER.warn(\"Leader selector executor did not terminate in time\");\n          }\n        }\n        catch (InterruptedException e) {\n          Thread.currentThread().interrupt();\n        }\n      }\n\n      if (sessionKeeperService != null) {\n        sessionKeeperService.shutdownNow();\n        try {\n          if (!sessionKeeperService.awaitTermination(5, TimeUnit.SECONDS)) {\n            LOGGER.warn(\"Session keeper service did not terminate in time\");\n          }\n        }\n        catch (InterruptedException e) {\n          Thread.currentThread().interrupt();\n        }\n      }","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L179-L215","documentation":"Warning logged during unregisterListener() when the leader-election executor does not finish shutting down within the 5-second awaitTermination window after shutdownNow(). It means the election loop was likely blocked in a slow Consul RPC or a long sleep, so the JVM may retain a lingering thread after service stop.","triggerScenarios":"Calling unregisterListener() (service shutdown) while leaderElectionLoop is blocked in a Consul HTTP call (long network timeout or unreachable Consul) or in a Thread.sleep backoff longer than 5 seconds.","commonSituations":"Decommissioning a node with an unreachable/slow Consul agent, very large leaderRetryBackoffMax or healthCheckInterval causing long sleeps, or network partitions during rolling restarts.","solutions":["Ensure Consul is reachable and responding before shutdown; fix network/agent issues that stall RPCs.","Reduce leaderRetryBackoffMax and healthCheckInterval so loop sleeps stay under the 5s termination window.","Accept the warning if benign — shutdownNow() already interrupted the thread and it will exit at the next interruptible point."],"exampleFix":"// before (long backoff stalls shutdown)\ndruid.discovery.consul.leader.retryBackoffMax=PT5M\n// after\ndruid.discovery.consul.leader.retryBackoffMax=PT10S","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  selector.unregisterListener();\n} finally {\n  // executor threads exit asynchronously after shutdownNow(); do not assume immediate termination\n}","preventionTips":["Keep retry backoff caps well under 5s if fast shutdown matters","Monitor Consul agent health before rolling restarts","Call unregisterListener() only from lifecycle-stop threads, never time-sensitive threads"],"tags":["shutdown","executor","timeout","consul"],"backgroundTag":"request-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}