apache/druid · warning

Failed to immediately mark service

Error message

Failed to immediately mark service [%s] as healthy, will retry via periodic health check

What it means

DefaultConsulApiClient.registerService attempts an immediate agentCheckPass("service:" + serviceId) so a newly registered service shows as healthy right away. When that call fails, this warning is logged and registration is not considered failed — the periodic health check is expected to mark the check as passing later. The service may therefore appear unhealthy (critical) in Consul for one check interval after registration.

Solutions

1. Verify the Consul ACL token used by config.getAuth().getAclToken() includes service:check-write (and service:write) for the service prefix — this is the most common cause after enabling ACLs
2. Check connectivity to the local Consul agent (curl http://localhost:8500/v1/agent/checks) and retry; if transient, no action is needed since the periodic check self-heals
3. If the warning repeats every cycle, inspect druid console logs for the underlying exception (the LOGGER.warn includes `e`) and confirm the check id naming convention 'service:<serviceId>' matches what registerService created
4. Increase health-check interval tolerance temporarily, or pass the check state at registration time instead of a separate agentCheckPass call

Example fix

// before: check written separately after registration, can fail transiently
consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", config.getAuth().getAclToken());
// after: align ACL policy so the token can update checks
// service "druid" { policy = "write" }
// (or) pass note/output during check registration:
service { id = "<serviceId>"; check { ttl = "10s"; status = "passing" } }

// preflight: confirm the agent accepts check writes with this token
curl -s -H "X-Consul-Token: $ACL_TOKEN" http://localhost:8500/v1/agent/checks | jq 'keys'
// and validate the ACL policy locally before starting the node:
consul acl token read -id <token> | grep -i 'service:check-write'

try {
  consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", aclToken);
} catch (Exception e) {
  LOGGER.warn(e, "Failed to immediately mark service [%s] as healthy, will retry via periodic health check", serviceId);
  // non-fatal: rely on periodic TTL check pass
}

When it happens

Trigger: registerService(serviceId) invokes consulClient.agentCheckPass(...) and any exception is thrown — e.g. transient Consul agent HTTP failure, the TTL check 'service:<id>' not existing yet (registration race), or an ACL token lacking the 'service:check-write' permission.

Common situations: ACL token misconfiguration after enabling Consul ACLs (missing check-write rule); slow Consul agent responding to the registration write so the check isn't yet registered when agentCheckPass fires; network blip between the Druid node and the local Consul agent; auth token not propagated in config.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Health check executor did not terminate in time
• Health check failed [ / ] for [ ]
• Re-registering [ ] after failures
• Session creation failed for
• can't start

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4c766b51c698833f. Report an issue: GitHub.

Appendix: source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java:147

    NewService.Check check = new NewService.Check();
    long intervalSeconds = Math.max(MIN_HEALTH_CHECK_INTERVAL_SECONDS, config.getService().getHealthCheckInterval().getStandardSeconds());
    long ttlSeconds = Math.max(MIN_SESSION_TTL_SECONDS, intervalSeconds * 3);
    check.setTtl(StringUtils.format("%ds", ttlSeconds));
    check.setDeregisterCriticalServiceAfter(
        StringUtils.format("%ds", config.getService().getDeregisterAfter().getStandardSeconds())
    );
    service.setCheck(check);

    consulClient.agentServiceRegister(service, config.getAuth().getAclToken());
    LOGGER.info("Registered service [%s] with Consul", serviceId);

    try {
      consulClient.agentCheckPass("service:" + serviceId, "Druid node is healthy", config.getAuth().getAclToken());
    }
    catch (Exception e) {
      // Log but don't fail - the periodic health check will eventually mark it as passing
      LOGGER.warn(e, "Failed to immediately mark service [%s] as healthy, will retry via periodic health check", serviceId);
    }
  }

  @Override
  @SuppressWarnings("RedundantThrows")
  public void deregisterService(String serviceId) throws Exception
  {
    try {
      String kvKey = ConsulServiceIds.nodeKvKey(config, serviceId);
      consulClient.deleteKVValue(kvKey, config.getAuth().getAclToken());
    }
    catch (Exception e) {
      LOGGER.debug(e, "Failed to delete KV entry for service [%s] during deregistration", serviceId);
    }

    consulClient.agentServiceDeregister(serviceId, config.getAuth().getAclToken());
    LOGGER.info("Deregistered service [%s] from Consul", serviceId);
  }

View on GitHub (pinned to 9b90983fd2)