{"record":{"id":"cbb7494705079748","repo":"apache/druid","slug":"failed-to-stop-watchexecutor-for-role-s","errorCode":null,"errorMessage":"Failed to stop watchExecutor for role[%s]","messagePattern":"Failed to stop watchExecutor for role\\[(.+?)\\]","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java","lineNumber":411,"sourceCode":"        LOGGER.info(\"Started NodeRoleWatcher for role[%s].\", nodeRole);\n      }\n      finally {\n        lifecycleLock.exitStart();\n      }\n    }\n\n    public void stop()\n    {\n      if (!lifecycleLock.canStop()) {\n        throw new ISE(\"can't stop.\");\n      }\n\n      try {\n        LOGGER.info(\"Stopping NodeRoleWatcher for role[%s]...\", nodeRole);\n        watchExecutor.shutdownNow();\n\n        if (!watchExecutor.awaitTermination(15, TimeUnit.SECONDS)) {\n          LOGGER.warn(\"Failed to stop watchExecutor for role[%s]\", nodeRole);\n        }\n        ConsulMetrics.emitCount(\n            emitter,\n            \"consul/watch/lifecycle\",\n            \"role\",\n            nodeRole.getJsonName(),\n            \"state\",\n            \"stop\"\n        );\n        LOGGER.info(\"Stopped NodeRoleWatcher for role[%s].\", nodeRole);\n      }\n      catch (Exception ex) {\n        LOGGER.error(ex, \"Failed to stop NodeRoleWatcher for role[%s].\", nodeRole);\n      }\n      finally {\n        // Allow restart and leave lock in a clean state\n        lifecycleLock.exitStopAndReset();\n      }","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java#L393-L429","documentation":"Emitted in ConsulDruidNodeDiscoveryProvider.stop when shutdownNow() on a role's watchExecutor does not terminate it within 15 seconds. The watcher thread is likely stuck in a blocking Consul long-poll or slow cleanup, forcing shutdown Now was already attempted; the log marks it as not gracefully stopped.","triggerScenarios":"stop() called while a NodeRoleWatcher thread is blocked in a long-running Consul blocking HTTP query or an uninterruptible operation, exceeding the 15-second awaitTermination window.","commonSituations":"Long blocking-query durations exceeding the shutdown wait; socket reads without timeouts keeping the thread blocked despite interruption; JVM under load at shutdown.","solutions":["Set the Consul client's connect/socket timeouts below the 15-second termination window so blocking queries can end quickly.","Reduce configured blocking-query duration so watches cycle and observe interrupts sooner.","Verify the underlying network to Consul is healthy; stuck sockets are the usual cause.","If legitimately slow, extend the 15-second awaitTermination timeout."],"exampleFix":"// before\nRequestConfig rc = RequestConfig.custom().build(); // no socket timeout\n// after\nRequestConfig rc = RequestConfig.custom().setConnectTimeout(2000).setSocketTimeout(5000).build();","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  provider.stop();\n} finally {\n  // assume watcher threads may linger; avoid resource reuse\n}","preventionTips":["Configure HTTP connect/socket timeouts below the 15s shutdown wait","Use moderate blocking-query durations so watches interrupt promptly","Ensure healthy networking to Consul to avoid stuck sockets"],"tags":["shutdown","executor","consul","watch"],"backgroundTag":"executor-termination-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}