apache/druid · warning

Exception while watching for role[%s], will retry.

Error message

Exception while watching for role[%s], will retry.

What it means

WARN logged by a NodeRoleWatcher's watch loop in ConsulDruidNodeDiscoveryProvider when the long-poll blocking query to Consul for a node role throws a non-timeout exception. The loop logs, emits a consul/watch/error metric, increments a retry counter, and retries; after exceeding maxWatchRetries it engages a circuit-breaker sleep.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java:312

                }
              }
            }
          }

          retryCount.set(0);
        }
        catch (Exception ex) {
          if (Thread.currentThread().isInterrupted()) {
            LOGGER.info("Watch interrupted during shutdown for role[%s]", nodeRole);
            break;
          }

          if (isSocketTimeout(ex)) {
            LOGGER.debug("Watch timeout for role[%s], re-issuing blocking query.", nodeRole);
            continue;
          }

          LOGGER.warn(ex, "Exception while watching for role[%s], will retry.", nodeRole);

          ConsulMetrics.emitCount(emitter, "consul/watch/error", "role", nodeRole.getJsonName());

          long count = retryCount.incrementAndGet();
          if (config.getWatch().getMaxWatchRetries() != Long.MAX_VALUE && count > config.getWatch().getMaxWatchRetries()) {
            long circuitBreakerSleepMs = config.getWatch().getCircuitBreakerSleep().getMillis();
            LOGGER.error(
                "Max watch retries [%d] exceeded for role[%s]; circuit breaker OPEN, sleeping %s then retry",
                config.getWatch().getMaxWatchRetries(),
                nodeRole,
                config.getWatch().getCircuitBreakerSleep()
            );
            ConsulMetrics.emitCount(emitter, "consul/watch/circuit_breaker",
                "state", "open", "role", nodeRole.getJsonName());

            sleep(circuitBreakerSleepMs);

            retryCount.set(0);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the logged stack trace to identify the root exception class.
2. Verify Consul agent availability and network connectivity; fix the underlying outage — the watcher retries automatically.
3. If retries are exhausted, tune watch.maxWatchRetries and watch.circuitBreakerSleep for your environment.
4. Check Consul ACL token validity if errors are permission-related.

Example fix

// no code change; environment check
// curl http://localhost:8500/v1/catalog/nodes  (verify agent reachable)// confirm ACL token has catalog:read

// reachability pre-check before starting watchers
HttpGet check = new HttpGet("http://localhost:8500/v1/status/leader");

try {
  watch(role);
} catch (Exception e) {
  // watcher auto-retries; check maxWatchRetries and circuitBreakerSleep config
}

When it happens

Trigger: Any exception from the Consul blocking query that is not a socket timeout: connection refused/reset, DNS failure, HTTP 5xx, deserialization error, or interrupted/other IO errors during watch().

Common situations: Consul agent restart or crash on the watched node; network flaps between Druid and Consul; Consul returning 5xx under load; ACL token revoked mid-watch.

Related errors

• Health check failed [%d/%d] for [%s]
• Expection in watching pods, code[%d] and error[%s].
• start method returned false for lookup [%s]:[%s]
• Failed to stop watchExecutor for role[%s]
• Session creation failed for [%s]; backing off before retry

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/350313b283af6ae6. Report an issue: GitHub.