{"record":{"id":"58328e347efc2d93","repo":"apache/druid","slug":"listener-executor-did-not-terminate-in-time","errorCode":null,"errorMessage":"Listener executor did not terminate in time","messagePattern":"Listener executor did not terminate in time","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java","lineNumber":169,"sourceCode":"  @LifecycleStop\n  public void stop()\n  {\n    if (!lifecycleLock.canStop()) {\n      throw new ISE(\"can't stop.\");\n    }\n\n    LOGGER.info(\"Stopping ConsulDruidNodeDiscoveryProvider\");\n\n    for (NodeRoleWatcher watcher : nodeRoleWatchers.values()) {\n      watcher.stop();\n    }\n    nodeRoleWatchers.clear();\n\n    // Watcher threads must finish before shutting down listener executor to avoid RejectedExecutionException\n    try {\n      listenerExecutor.shutdown();\n      if (!listenerExecutor.awaitTermination(10, TimeUnit.SECONDS)) {\n        LOGGER.warn(\"Listener executor did not terminate in time\");\n        listenerExecutor.shutdownNow();\n      }\n    }\n    catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      LOGGER.warn(\"Interrupted while waiting for listener executor termination\");\n      listenerExecutor.shutdownNow();\n    }\n\n    LOGGER.info(\"Stopped ConsulDruidNodeDiscoveryProvider\");\n    lifecycleLock.exitStopAndReset();\n  }\n\n  static class NodeRoleWatcher implements DruidNodeDiscovery\n  {\n    private static final Logger LOGGER = new Logger(NodeRoleWatcher.class);\n\n    private final ConsulApiClient consulApiClient;","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java#L151-L187","documentation":"Emitted in ConsulDruidNodeDiscoveryProvider.stop when the listener executor does not terminate within 10 seconds after shutdown(). The code then calls shutdownNow() to force interruption. This indicates listener callbacks were still running during provider shutdown.","triggerScenarios":"stop() invoked while a discovery listener callback is long-running or blocked (e.g. slow handler, lock contention), exceeding the 10-second awaitTermination window.","commonSituations":"A subscriber callback doing slow I/O or waiting on a lock; JVM under heavy load during shutdown; a watcher thread not fully finished before listener executor shutdown (the code orders these deliberately to avoid RejectedExecutionException).","solutions":["Make listener callbacks fast/non-blocking; offload heavy work to your own executor.","Check for deadlocks or long waits inside registered DruidNodeDiscoverySubscriber callbacks.","If callbacks legitimately take longer, extend the 10-second timeout.","Ensure watchers are stopped before provider stop() as the code expects (normal lifecycle does this)."],"exampleFix":"// before: slow work inside listener callback\npublic void nodeAdded(DiscoveryDruidNode node) { queryConsulLongOp(node); }\n// after\npublic void nodeAdded(DiscoveryDruidNode node) { workerExecutor.submit(() -> queryConsulLongOp(node)); }","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  provider.stop();\n} finally {\n  // confirm listeners were delivered or document missed callbacks\n}","preventionTips":["Keep listener callbacks fast and non-blocking","Offload heavy processing to a separate executor","Avoid locks inside subscriber callbacks"],"tags":["shutdown","executor","consul","listener"],"backgroundTag":"executor-termination-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}