{"record":{"id":"350313b283af6ae6","repo":"apache/druid","slug":"exception-while-watching-for-role-s-will-retry","errorCode":null,"errorMessage":"Exception while watching for role[%s], will retry.","messagePattern":"Exception while watching for role\\[(.+?)\\], will retry\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java","lineNumber":312,"sourceCode":"                }\n              }\n            }\n          }\n\n          retryCount.set(0);\n        }\n        catch (Exception ex) {\n          if (Thread.currentThread().isInterrupted()) {\n            LOGGER.info(\"Watch interrupted during shutdown for role[%s]\", nodeRole);\n            break;\n          }\n\n          if (isSocketTimeout(ex)) {\n            LOGGER.debug(\"Watch timeout for role[%s], re-issuing blocking query.\", nodeRole);\n            continue;\n          }\n\n          LOGGER.warn(ex, \"Exception while watching for role[%s], will retry.\", nodeRole);\n\n          ConsulMetrics.emitCount(emitter, \"consul/watch/error\", \"role\", nodeRole.getJsonName());\n\n          long count = retryCount.incrementAndGet();\n          if (config.getWatch().getMaxWatchRetries() != Long.MAX_VALUE && count > config.getWatch().getMaxWatchRetries()) {\n            long circuitBreakerSleepMs = config.getWatch().getCircuitBreakerSleep().getMillis();\n            LOGGER.error(\n                \"Max watch retries [%d] exceeded for role[%s]; circuit breaker OPEN, sleeping %s then retry\",\n                config.getWatch().getMaxWatchRetries(),\n                nodeRole,\n                config.getWatch().getCircuitBreakerSleep()\n            );\n            ConsulMetrics.emitCount(emitter, \"consul/watch/circuit_breaker\",\n                \"state\", \"open\", \"role\", nodeRole.getJsonName());\n\n            sleep(circuitBreakerSleepMs);\n\n            retryCount.set(0);","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeDiscoveryProvider.java#L294-L330","documentation":"WARN logged by a NodeRoleWatcher's watch loop in ConsulDruidNodeDiscoveryProvider when the long-poll blocking query to Consul for a node role throws a non-timeout exception. The loop logs, emits a consul/watch/error metric, increments a retry counter, and retries; after exceeding maxWatchRetries it engages a circuit-breaker sleep.","triggerScenarios":"Any exception from the Consul blocking query that is not a socket timeout: connection refused/reset, DNS failure, HTTP 5xx, deserialization error, or interrupted/other IO errors during watch().","commonSituations":"Consul agent restart or crash on the watched node; network flaps between Druid and Consul; Consul returning 5xx under load; ACL token revoked mid-watch.","solutions":["Inspect the logged stack trace to identify the root exception class.","Verify Consul agent availability and network connectivity; fix the underlying outage — the watcher retries automatically.","If retries are exhausted, tune watch.maxWatchRetries and watch.circuitBreakerSleep for your environment.","Check Consul ACL token validity if errors are permission-related."],"exampleFix":"// no code change; environment check\n// curl http://localhost:8500/v1/catalog/nodes  (verify agent reachable)// confirm ACL token has catalog:read","handlingStrategy":"retry","validationCode":"// reachability pre-check before starting watchers\nHttpGet check = new HttpGet(\"http://localhost:8500/v1/status/leader\");","typeGuard":null,"tryCatchPattern":"try {\n  watch(role);\n} catch (Exception e) {\n  // watcher auto-retries; check maxWatchRetries and circuitBreakerSleep config\n}","preventionTips":["Set maxWatchRetries and circuitBreakerSleep appropriately for your network","Monitor the consul/watch/error metric by role","Keep Consul ACL tokens valid for catalog reads"],"tags":["consul","watch","network","retry"],"backgroundTag":"network-request-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}