apache/druid · warning

Re-registering [%s] after %d failures

Error message

Re-registering [%s] after %d failures

What it means

Recovery-action WARN in ConsulDruidNodeAnnouncer.updateHealthChecks: after consecutive health-check failures reach MAX_FAILURES_BEFORE_REREGISTER, the announcer proactively re-registers the service in Consul to heal possibly-stale/lost registrations.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.java:302

        consecutiveFailures.remove(serviceId);
      }
      catch (Exception e) {
        failureCount++;

        int failures = consecutiveFailures
            .computeIfAbsent(serviceId, k -> new AtomicInteger(0))
            .incrementAndGet();

        // Keep WARN for failures - these matter
        LOGGER.warn(e, "Health check failed [%d/%d] for [%s]",
                    failures, MAX_FAILURES_BEFORE_REREGISTER, serviceId);
        ConsulMetrics.emitCount(emitter, "consul/healthcheck/failure",
            "serviceId", serviceId, "consecutiveFailures", String.valueOf(failures));

        if (failures >= MAX_FAILURES_BEFORE_REREGISTER) {
          // Keep WARN for recovery actions - these are important state changes
          LOGGER.warn("Re-registering [%s] after %d failures", serviceId, failures);
          try {
            // Re-fetch from map; node may have been concurrently removed during shutdown
            DiscoveryDruidNode node = announcedNodes.get(serviceId);
            if (node == null) {
              // Node was unannounced (e.g., during shutdown) - skip re-registration
              LOGGER.info("Skipping re-registration for [%s] - node no longer announced", serviceId);
              consecutiveFailures.remove(serviceId);
              continue;
            }
            consulApiClient.registerService(node);
            consulApiClient.passTtlCheck(serviceId, "Re-registered");
            consecutiveFailures.remove(serviceId);
            ConsulMetrics.emitCount(emitter, "consul/healthcheck/reregister",
                "serviceId", serviceId, "totalFailures", String.valueOf(failures));
            LOGGER.info("Successfully re-registered [%s]", serviceId);
          }
          catch (Exception reregEx) {
            LOGGER.error(reregEx, "Re-registration failed for [%s]", serviceId);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Let the automatic re-registration complete; verify the service reappears in the Consul catalog.
2. If re-registration also fails, fix underlying Consul connectivity/ACL issues first.
3. If this fires often, tune health-check interval and MAX_FAILURES_BEFORE_REREGISTER, and stabilize the Consul agent.

Example fix

// no code change; environment hardening
// run Consul agent as a local sidecar with retry_join and proper ACLs
// monitor consul/healthcheck/failure metric

try {
  updateHealthChecks();
} catch (Exception e) {
  // built-in flow re-registers after MAX_FAILURES_BEFORE_REREGISTER; just log
}

When it happens

Trigger: The failure counter for a serviceId reaches MAX_FAILURES_BEFORE_REREGISTER during updateHealthChecks; the node must still be present in announcedNodes for re-registration to proceed (otherwise it is skipped).

Common situations: Sustained Consul connectivity problems that later recover, leaving a stale or deregistered entry; Consul agent restart that dropped the service registration.

Related errors

• Health check executor did not terminate in time
• Health check failed [%d/%d] for [%s]
• Failed to immediately mark service [%s] as healthy, will ret
• can't stop.
• can't start

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/98b5fee3382074dc. Report an issue: GitHub.