{"record":{"id":"559a3ec718d25088","repo":"apache/druid","slug":"health-check-failed-d-d-for-s","errorCode":null,"errorMessage":"Health check failed [%d/%d] for [%s]","messagePattern":"Health check failed \\[(.+?)/(.+?)\\] for \\[(.+?)\\]","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java","lineNumber":295,"sourceCode":"        consulApiClient.passTtlCheck(serviceId, \"Druid node is healthy\");\n\n        long healthCheckLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - healthCheckStart);\n        ConsulMetrics.emitTimer(emitter, \"consul/healthcheck/latency\", healthCheckLatency,\n            \"serviceId\", serviceId);\n\n        successCount++;\n\n        consecutiveFailures.remove(serviceId);\n      }\n      catch (Exception e) {\n        failureCount++;\n\n        int failures = consecutiveFailures\n            .computeIfAbsent(serviceId, k -> new AtomicInteger(0))\n            .incrementAndGet();\n\n        // Keep WARN for failures - these matter\n        LOGGER.warn(e, \"Health check failed [%d/%d] for [%s]\",\n                    failures, MAX_FAILURES_BEFORE_REREGISTER, serviceId);\n        ConsulMetrics.emitCount(emitter, \"consul/healthcheck/failure\",\n            \"serviceId\", serviceId, \"consecutiveFailures\", String.valueOf(failures));\n\n        if (failures >= MAX_FAILURES_BEFORE_REREGISTER) {\n          // Keep WARN for recovery actions - these are important state changes\n          LOGGER.warn(\"Re-registering [%s] after %d failures\", serviceId, failures);\n          try {\n            // Re-fetch from map; node may have been concurrently removed during shutdown\n            DiscoveryDruidNode node = announcedNodes.get(serviceId);\n            if (node == null) {\n              // Node was unannounced (e.g., during shutdown) - skip re-registration\n              LOGGER.info(\"Skipping re-registration for [%s] - node no longer announced\", serviceId);\n              consecutiveFailures.remove(serviceId);\n              continue;\n            }\n            consulApiClient.registerService(node);\n            consulApiClient.passTtlCheck(serviceId, \"Re-registered\");","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java#L277-L313","documentation":"A WARN logged by the periodic health-check loop in ConsulDruidNodeAnnouncer.updateHealthChecks when querying Consul health checks for a registered service throws. A per-serviceId consecutive-failure counter is incremented and the message shows the current streak out of MAX_FAILURES_BEFORE_REREGISTER. A ConsulMetrics counter is also emitted.","triggerScenarios":"Consul API health-check queries repeatedly failing for a serviceId — network errors, Consul agent down/restarting, HTTP 5xx, or timeouts — until the exception path is hit MAX_FAILURES_BEFORE_REREGISTER times.","commonSituations":"Consul agent restart or upgrade on the local node; network partition between Druid and Consul; Consul overloaded and returning errors.","solutions":["Check Consul agent health and connectivity from the Druid node (curl the /v1/agent/healthChecks endpoint).","Inspect the logged exception (first argument) for the root cause — socket timeout vs connection refused vs HTTP error.","Watch the consul/healthcheck/failure metric; if it reaches MAX_FAILURES_BEFORE_REREGISTER, re-registration follows automatically.","If failures are transient and infrequent, no action needed; the counter resets on the next successful check."],"exampleFix":"// no code fix; verify environment\n// curl http://localhost:8500/v1/agent/healthChecks\n// ensure network/ACLs allow Druid->Consul queries","handlingStrategy":"retry","validationCode":"// pre-check before relying on health checks\nHttpGet agentHealth = new HttpGet(\"http://localhost:8500/v1/agent/healthChecks\");","typeGuard":null,"tryCatchPattern":"try {\n  updateHealthChecks();\n} catch (ConsulException | IOException e) {\n  // rely on built-in consecutive-failure counter and re-registration\n}","preventionTips":["Monitor the consul/healthcheck/failure metric","Keep the Consul agent co-located/healthy","Set alerts before MAX_FAILURES_BEFORE_REREGISTER is reached"],"tags":["consul","health-check","network","retry"],"backgroundTag":"health-check-failure","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}