{"record":{"id":"98b5fee3382074dc","repo":"apache/druid","slug":"re-registering-s-after-d-failures","errorCode":null,"errorMessage":"Re-registering [%s] after %d failures","messagePattern":"Re-registering \\[(.+?)\\] after (.+?) failures","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java","lineNumber":302,"sourceCode":"\n        consecutiveFailures.remove(serviceId);\n      }\n      catch (Exception e) {\n        failureCount++;\n\n        int failures = consecutiveFailures\n            .computeIfAbsent(serviceId, k -> new AtomicInteger(0))\n            .incrementAndGet();\n\n        // Keep WARN for failures - these matter\n        LOGGER.warn(e, \"Health check failed [%d/%d] for [%s]\",\n                    failures, MAX_FAILURES_BEFORE_REREGISTER, serviceId);\n        ConsulMetrics.emitCount(emitter, \"consul/healthcheck/failure\",\n            \"serviceId\", serviceId, \"consecutiveFailures\", String.valueOf(failures));\n\n        if (failures >= MAX_FAILURES_BEFORE_REREGISTER) {\n          // Keep WARN for recovery actions - these are important state changes\n          LOGGER.warn(\"Re-registering [%s] after %d failures\", serviceId, failures);\n          try {\n            // Re-fetch from map; node may have been concurrently removed during shutdown\n            DiscoveryDruidNode node = announcedNodes.get(serviceId);\n            if (node == null) {\n              // Node was unannounced (e.g., during shutdown) - skip re-registration\n              LOGGER.info(\"Skipping re-registration for [%s] - node no longer announced\", serviceId);\n              consecutiveFailures.remove(serviceId);\n              continue;\n            }\n            consulApiClient.registerService(node);\n            consulApiClient.passTtlCheck(serviceId, \"Re-registered\");\n            consecutiveFailures.remove(serviceId);\n            ConsulMetrics.emitCount(emitter, \"consul/healthcheck/reregister\",\n                \"serviceId\", serviceId, \"totalFailures\", String.valueOf(failures));\n            LOGGER.info(\"Successfully re-registered [%s]\", serviceId);\n          }\n          catch (Exception reregEx) {\n            LOGGER.error(reregEx, \"Re-registration failed for [%s]\", serviceId);","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java#L284-L320","documentation":"Recovery-action WARN in ConsulDruidNodeAnnouncer.updateHealthChecks: after consecutive health-check failures reach MAX_FAILURES_BEFORE_REREGISTER, the announcer proactively re-registers the service in Consul to heal possibly-stale/lost registrations.","triggerScenarios":"The failure counter for a serviceId reaches MAX_FAILURES_BEFORE_REREGISTER during updateHealthChecks; the node must still be present in announcedNodes for re-registration to proceed (otherwise it is skipped).","commonSituations":"Sustained Consul connectivity problems that later recover, leaving a stale or deregistered entry; Consul agent restart that dropped the service registration.","solutions":["Let the automatic re-registration complete; verify the service reappears in the Consul catalog.","If re-registration also fails, fix underlying Consul connectivity/ACL issues first.","If this fires often, tune health-check interval and MAX_FAILURES_BEFORE_REREGISTER, and stabilize the Consul agent."],"exampleFix":"// no code change; environment hardening\n// run Consul agent as a local sidecar with retry_join and proper ACLs\n// monitor consul/healthcheck/failure metric","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  updateHealthChecks();\n} catch (Exception e) {\n  // built-in flow re-registers after MAX_FAILURES_BEFORE_REREGISTER; just log\n}","preventionTips":["Stabilize Consul agent availability to avoid reaching the failure threshold","Tune MAX_FAILURES_BEFORE_REREGISTER for your network reliability","Verify service re-registration in the Consul catalog after episodes"],"tags":["consul","health-check","recovery","re-registration"],"backgroundTag":"health-check-failure","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}