apache/druid · warning

Health check executor did not terminate in time

Error message

Health check executor did not terminate in time

What it means

Emitted in ConsulDruidNodeAnnouncer.stop when the health-check executor does not finish terminating within EXECUTOR_TERMINATION_TIMEOUT_SECONDS after shutdownNow(). The announcer waits so a service is never deregistered while a health check is mid-flight; this warning means the wait timed out and deregistration may race an in-progress check.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java:120

      lifecycleLock.exitStart();
    }
  }

  @LifecycleStop
  public void stop()
  {
    if (!lifecycleLock.canStop()) {
      throw new ISE("can't stop");
    }

    LOGGER.info("Stopping ConsulDruidNodeAnnouncer");

    healthCheckExecutor.shutdownNow();

    // Wait for health check to finish so we don't deregister while health check is in progress
    try {
      if (!healthCheckExecutor.awaitTermination(EXECUTOR_TERMINATION_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {
        LOGGER.warn("Health check executor did not terminate in time");
      }
    }
    catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      LOGGER.warn("Interrupted while waiting for health check termination");
    }

    for (String serviceId : announcedNodes.keySet()) {
      try {
        consulApiClient.deregisterService(serviceId);
      }
      catch (Exception e) {
        LOGGER.error(e, "Failed to deregister service [%s] during shutdown", serviceId);
      }
    }

    announcedNodes.clear();
    lifecycleLock.exitStop();

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check Consul agent responsiveness/network from the node; a hanging agent is the usual cause.
2. Increase EXECUTOR_TERMINATION_TIMEOUT_SECONDS if checks legitimately take long.
3. Ensure health-check HTTP client calls use connect/read timeouts shorter than the termination timeout so tasks are interruptible.
4. If interruptions during shutdown recur, verify no non-interruptible blocking calls (e.g. socket reads without timeouts) in the check task.

Example fix

// health check request without timeout
HttpResponse resp = client.execute(get);
// after
RequestConfig rc = RequestConfig.custom().setConnectTimeout(2000).setSocketTimeout(5000).build();
HttpResponse resp = client.execute(get, rc);

try {
  announcer.stop();
} finally {
  // verify service deregistration completed in Consul
}

When it happens

Trigger: Calling stop() on ConsulDruidNodeAnnouncer while the scheduled health-check task is blocked (e.g. Consul request hanging) and does not respond to interruption within the timeout.

Common situations: Slow or partitioned Consul agent causing health-check HTTP calls to hang; JVM under heavy load starving the executor threads during shutdown.

Related errors

• Listener executor did not terminate in time
• Failed to stop watchExecutor for role[%s]
• Leader selector executor did not terminate in time
• Session keeper service did not terminate in time
• Health check failed [%d/%d] for [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ec6f0bdc9de034ff. Report an issue: GitHub.