{"record":{"id":"ec6f0bdc9de034ff","repo":"apache/druid","slug":"health-check-executor-did-not-terminate-in-time","errorCode":null,"errorMessage":"Health check executor did not terminate in time","messagePattern":"Health check executor did not terminate in time","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java","lineNumber":120,"sourceCode":"      lifecycleLock.exitStart();\n    }\n  }\n\n  @LifecycleStop\n  public void stop()\n  {\n    if (!lifecycleLock.canStop()) {\n      throw new ISE(\"can't stop\");\n    }\n\n    LOGGER.info(\"Stopping ConsulDruidNodeAnnouncer\");\n\n    healthCheckExecutor.shutdownNow();\n\n    // Wait for health check to finish so we don't deregister while health check is in progress\n    try {\n      if (!healthCheckExecutor.awaitTermination(EXECUTOR_TERMINATION_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {\n        LOGGER.warn(\"Health check executor did not terminate in time\");\n      }\n    }\n    catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      LOGGER.warn(\"Interrupted while waiting for health check termination\");\n    }\n\n    for (String serviceId : announcedNodes.keySet()) {\n      try {\n        consulApiClient.deregisterService(serviceId);\n      }\n      catch (Exception e) {\n        LOGGER.error(e, \"Failed to deregister service [%s] during shutdown\", serviceId);\n      }\n    }\n\n    announcedNodes.clear();\n    lifecycleLock.exitStop();","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java#L102-L138","documentation":"Emitted in ConsulDruidNodeAnnouncer.stop when the health-check executor does not finish terminating within EXECUTOR_TERMINATION_TIMEOUT_SECONDS after shutdownNow(). The announcer waits so a service is never deregistered while a health check is mid-flight; this warning means the wait timed out and deregistration may race an in-progress check.","triggerScenarios":"Calling stop() on ConsulDruidNodeAnnouncer while the scheduled health-check task is blocked (e.g. Consul request hanging) and does not respond to interruption within the timeout.","commonSituations":"Slow or partitioned Consul agent causing health-check HTTP calls to hang; JVM under heavy load starving the executor threads during shutdown.","solutions":["Check Consul agent responsiveness/network from the node; a hanging agent is the usual cause.","Increase EXECUTOR_TERMINATION_TIMEOUT_SECONDS if checks legitimately take long.","Ensure health-check HTTP client calls use connect/read timeouts shorter than the termination timeout so tasks are interruptible.","If interruptions during shutdown recur, verify no non-interruptible blocking calls (e.g. socket reads without timeouts) in the check task."],"exampleFix":"// health check request without timeout\nHttpResponse resp = client.execute(get);\n// after\nRequestConfig rc = RequestConfig.custom().setConnectTimeout(2000).setSocketTimeout(5000).build();\nHttpResponse resp = client.execute(get, rc);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  announcer.stop();\n} finally {\n  // verify service deregistration completed in Consul\n}","preventionTips":["Give health-check HTTP calls short connect/read timeouts","Avoid long-running work in health-check tasks","Allow adequate shutdown time in lifecycle orchestration"],"tags":["shutdown","executor","consul","health-check"],"backgroundTag":"executor-termination-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}