apache/druid · warning

Failed to stop watchExecutor for role[%s]

Error message

Failed to stop watchExecutor for role[%s]

What it means

Emitted in ConsulDruidNodeDiscoveryProvider.stop when shutdownNow() on a role's watchExecutor does not terminate it within 15 seconds. The watcher thread is likely stuck in a blocking Consul long-poll or slow cleanup, forcing shutdown Now was already attempted; the log marks it as not gracefully stopped.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java:411

        LOGGER.info("Started NodeRoleWatcher for role[%s].", nodeRole);
      }
      finally {
        lifecycleLock.exitStart();
      }
    }

    public void stop()
    {
      if (!lifecycleLock.canStop()) {
        throw new ISE("can't stop.");
      }

      try {
        LOGGER.info("Stopping NodeRoleWatcher for role[%s]...", nodeRole);
        watchExecutor.shutdownNow();

        if (!watchExecutor.awaitTermination(15, TimeUnit.SECONDS)) {
          LOGGER.warn("Failed to stop watchExecutor for role[%s]", nodeRole);
        }
        ConsulMetrics.emitCount(
            emitter,
            "consul/watch/lifecycle",
            "role",
            nodeRole.getJsonName(),
            "state",
            "stop"
        );
        LOGGER.info("Stopped NodeRoleWatcher for role[%s].", nodeRole);
      }
      catch (Exception ex) {
        LOGGER.error(ex, "Failed to stop NodeRoleWatcher for role[%s].", nodeRole);
      }
      finally {
        // Allow restart and leave lock in a clean state
        lifecycleLock.exitStopAndReset();
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Set the Consul client's connect/socket timeouts below the 15-second termination window so blocking queries can end quickly.
2. Reduce configured blocking-query duration so watches cycle and observe interrupts sooner.
3. Verify the underlying network to Consul is healthy; stuck sockets are the usual cause.
4. If legitimately slow, extend the 15-second awaitTermination timeout.

Example fix

// before
RequestConfig rc = RequestConfig.custom().build(); // no socket timeout
// after
RequestConfig rc = RequestConfig.custom().setConnectTimeout(2000).setSocketTimeout(5000).build();

try {
  provider.stop();
} finally {
  // assume watcher threads may linger; avoid resource reuse
}

When it happens

Trigger: stop() called while a NodeRoleWatcher thread is blocked in a long-running Consul blocking HTTP query or an uninterruptible operation, exceeding the 15-second awaitTermination window.

Common situations: Long blocking-query durations exceeding the shutdown wait; socket reads without timeouts keeping the thread blocked despite interruption; JVM under load at shutdown.

Related errors

• Health check executor did not terminate in time
• Listener executor did not terminate in time
• Leader selector executor did not terminate in time
• Session keeper service did not terminate in time
• Exception while watching for role[%s], will retry.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cbb7494705079748. Report an issue: GitHub.