apache/druid · warning

Health check failed [%d/%d] for [%s]

Error message

Health check failed [%d/%d] for [%s]

What it means

A WARN logged by the periodic health-check loop in ConsulDruidNodeAnnouncer.updateHealthChecks when querying Consul health checks for a registered service throws. A per-serviceId consecutive-failure counter is incremented and the message shows the current streak out of MAX_FAILURES_BEFORE_REREGISTER. A ConsulMetrics counter is also emitted.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java:295

        consulApiClient.passTtlCheck(serviceId, "Druid node is healthy");

        long healthCheckLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - healthCheckStart);
        ConsulMetrics.emitTimer(emitter, "consul/healthcheck/latency", healthCheckLatency,
            "serviceId", serviceId);

        successCount++;

        consecutiveFailures.remove(serviceId);
      }
      catch (Exception e) {
        failureCount++;

        int failures = consecutiveFailures
            .computeIfAbsent(serviceId, k -> new AtomicInteger(0))
            .incrementAndGet();

        // Keep WARN for failures - these matter
        LOGGER.warn(e, "Health check failed [%d/%d] for [%s]",
                    failures, MAX_FAILURES_BEFORE_REREGISTER, serviceId);
        ConsulMetrics.emitCount(emitter, "consul/healthcheck/failure",
            "serviceId", serviceId, "consecutiveFailures", String.valueOf(failures));

        if (failures >= MAX_FAILURES_BEFORE_REREGISTER) {
          // Keep WARN for recovery actions - these are important state changes
          LOGGER.warn("Re-registering [%s] after %d failures", serviceId, failures);
          try {
            // Re-fetch from map; node may have been concurrently removed during shutdown
            DiscoveryDruidNode node = announcedNodes.get(serviceId);
            if (node == null) {
              // Node was unannounced (e.g., during shutdown) - skip re-registration
              LOGGER.info("Skipping re-registration for [%s] - node no longer announced", serviceId);
              consecutiveFailures.remove(serviceId);
              continue;
            }
            consulApiClient.registerService(node);
            consulApiClient.passTtlCheck(serviceId, "Re-registered");

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check Consul agent health and connectivity from the Druid node (curl the /v1/agent/healthChecks endpoint).
2. Inspect the logged exception (first argument) for the root cause — socket timeout vs connection refused vs HTTP error.
3. Watch the consul/healthcheck/failure metric; if it reaches MAX_FAILURES_BEFORE_REREGISTER, re-registration follows automatically.
4. If failures are transient and infrequent, no action needed; the counter resets on the next successful check.

Example fix

// no code fix; verify environment
// curl http://localhost:8500/v1/agent/healthChecks
// ensure network/ACLs allow Druid->Consul queries

// pre-check before relying on health checks
HttpGet agentHealth = new HttpGet("http://localhost:8500/v1/agent/healthChecks");

try {
  updateHealthChecks();
} catch (ConsulException | IOException e) {
  // rely on built-in consecutive-failure counter and re-registration
}

When it happens

Trigger: Consul API health-check queries repeatedly failing for a serviceId — network errors, Consul agent down/restarting, HTTP 5xx, or timeouts — until the exception path is hit MAX_FAILURES_BEFORE_REREGISTER times.

Common situations: Consul agent restart or upgrade on the local node; network partition between Druid and Consul; Consul overloaded and returning errors.

Related errors

• Exception while watching for role[%s], will retry.
• start method returned false for lookup [%s]:[%s]
• Health check executor did not terminate in time
• Re-registering [%s] after %d failures
• Session creation failed for [%s]; backing off before retry

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/559a3ec718d25088. Report an issue: GitHub.