Failed to immediately mark service [%s] as healthy, will ret

Error message

Failed to immediately mark service [%s] as healthy, will retry via periodic health check

What it means

DefaultConsulApiClient.registerService attempts an immediate agentCheckPass("service:" + serviceId) so a newly registered service shows as healthy right away. When that call fails, this warning is logged and registration is not considered failed — the periodic health check is expected to mark the check as passing later. The service may therefore appear unhealthy (critical) in Consul for one check interval after registration.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java:147

    NewService.Check check = new NewService.Check();
    long intervalSeconds = Math.max(MIN_HEALTH_CHECK_INTERVAL_SECONDS, config.getService().getHealthCheckInterval().getStandardSeconds());
    long ttlSeconds = Math.max(MIN_SESSION_TTL_SECONDS, intervalSeconds * 3);
    check.setTtl(StringUtils.format("%ds", ttlSeconds));
    check.setDeregisterCriticalServiceAfter(
        StringUtils.format("%ds", config.getService().getDeregisterAfter().getStandardSeconds())
    );
    service.setCheck(check);

    consulClient.agentServiceRegister(service, config.getAuth().getAclToken());
    LOGGER.info("Registered service [%s] with Consul", serviceId);

    try {
      consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", config.getAuth().getAclToken());
    }
    catch (Exception e) {
      // Log but don't fail - the periodic health check will eventually mark it as passing
      LOGGER.warn(e, "Failed to immediately mark service [%s] as healthy, will retry via periodic health check", serviceId);
    }
  }

  @Override
  @SuppressWarnings("RedundantThrows")
  public void deregisterService(String serviceId) throws Exception
  {
    try {
      String kvKey = ConsulServiceIds.nodeKvKey(config, serviceId);
      consulClient.deleteKVValue(kvKey, config.getAuth().getAclToken());
    }
    catch (Exception e) {
      LOGGER.debug(e, "Failed to delete KV entry for service [%s] during deregistration", serviceId);
    }

    consulClient.agentServiceDeregister(serviceId, config.getAuth().getAclToken());
    LOGGER.info("Deregistered service [%s] from Consul", serviceId);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the Consul ACL token used by config.getAuth().getAclToken() includes service:check-write (and service:write) for the service prefix — this is the most common cause after enabling ACLs
Check connectivity to the local Consul agent (curl http://localhost:8500/v1/agent/checks) and retry; if transient, no action is needed since the periodic check self-heals
If the warning repeats every cycle, inspect druid console logs for the underlying exception (the LOGGER.warn includes `e`) and confirm the check id naming convention 'service:<serviceId>' matches what registerService created
Increase health-check interval tolerance temporarily, or pass the check state at registration time instead of a separate agentCheckPass call

Example fix

// before: check written separately after registration, can fail transiently
consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", config.getAuth().getAclToken());
// after: align ACL policy so the token can update checks
// service "druid" { policy = "write" }
// (or) pass note/output during check registration:
service { id = "<serviceId>"; check { ttl = "10s"; status = "passing" } }

Defensive patterns

Strategy: fallback

Validate before calling

// preflight: confirm the agent accepts check writes with this token
curl -s -H "X-Consul-Token: $ACL_TOKEN" http://localhost:8500/v1/agent/checks | jq 'keys'
// and validate the ACL policy locally before starting the node:
consul acl token read -id <token> | grep -i 'service:check-write'

Try / catch

try {
  consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", aclToken);
} catch (Exception e) {
  LOGGER.warn(e, "Failed to immediately mark service [%s] as healthy, will retry via periodic health check", serviceId);
  // non-fatal: rely on periodic TTL check pass
}

Prevention

Grant the Consul ACL token service:check-write (and service:write) for the service prefix
Ensure the local Consul agent is reachable and responsive at node startup
Set the TTL check interval comfortably larger than the immediate-pass retry so brief failures self-heal
Alert on repeated occurrences — repeated warnings indicate a persistent ACL or connectivity problem, not a transient one

When it happens

Trigger: registerService(serviceId) invokes consulClient.agentCheckPass(...) and any exception is thrown — e.g. transient Consul agent HTTP failure, the TTL check 'service:<id>' not existing yet (registration race), or an ACL token lacking the 'service:check-write' permission.

Common situations: ACL token misconfiguration after enabling Consul ACLs (missing check-write rule); slow Consul agent responding to the registration write so the check isn't yet registered when agentCheckPass fires; network blip between the Druid node and the local Consul agent; auth token not propagated in config.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4c766b51c698833f. Report an issue: GitHub.