{"record":{"id":"4c766b51c698833f","repo":"apache/druid","slug":"failed-to-immediately-mark-service-s-as-healthy","errorCode":null,"errorMessage":"Failed to immediately mark service [%s] as healthy, will retry via periodic health check","messagePattern":"Failed to immediately mark service \\[(.+?)\\] as healthy, will retry via periodic health check","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java","lineNumber":147,"sourceCode":"\n    NewService.Check check = new NewService.Check();\n    long intervalSeconds = Math.max(MIN_HEALTH_CHECK_INTERVAL_SECONDS, config.getService().getHealthCheckInterval().getStandardSeconds());\n    long ttlSeconds = Math.max(MIN_SESSION_TTL_SECONDS, intervalSeconds * 3);\n    check.setTtl(StringUtils.format(\"%ds\", ttlSeconds));\n    check.setDeregisterCriticalServiceAfter(\n        StringUtils.format(\"%ds\", config.getService().getDeregisterAfter().getStandardSeconds())\n    );\n    service.setCheck(check);\n\n    consulClient.agentServiceRegister(service, config.getAuth().getAclToken());\n    LOGGER.info(\"Registered service [%s] with Consul\", serviceId);\n\n    try {\n      consulClient.agentCheckPass(\"service:\" + serviceId, \"Druid node is healthy\", config.getAuth().getAclToken());\n    }\n    catch (Exception e) {\n      // Log but don't fail - the periodic health check will eventually mark it as passing\n      LOGGER.warn(e, \"Failed to immediately mark service [%s] as healthy, will retry via periodic health check\", serviceId);\n    }\n  }\n\n  @Override\n  @SuppressWarnings(\"RedundantThrows\")\n  public void deregisterService(String serviceId) throws Exception\n  {\n    try {\n      String kvKey = ConsulServiceIds.nodeKvKey(config, serviceId);\n      consulClient.deleteKVValue(kvKey, config.getAuth().getAclToken());\n    }\n    catch (Exception e) {\n      LOGGER.debug(e, \"Failed to delete KV entry for service [%s] during deregistration\", serviceId);\n    }\n\n    consulClient.agentServiceDeregister(serviceId, config.getAuth().getAclToken());\n    LOGGER.info(\"Deregistered service [%s] from Consul\", serviceId);\n  }","sourceCodeStart":129,"sourceCodeEnd":165,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java#L129-L165","documentation":"DefaultConsulApiClient.registerService attempts an immediate agentCheckPass(\"service:\" + serviceId) so a newly registered service shows as healthy right away. When that call fails, this warning is logged and registration is not considered failed — the periodic health check is expected to mark the check as passing later. The service may therefore appear unhealthy (critical) in Consul for one check interval after registration.","triggerScenarios":"registerService(serviceId) invokes consulClient.agentCheckPass(...) and any exception is thrown — e.g. transient Consul agent HTTP failure, the TTL check 'service:<id>' not existing yet (registration race), or an ACL token lacking the 'service:check-write' permission.","commonSituations":"ACL token misconfiguration after enabling Consul ACLs (missing check-write rule); slow Consul agent responding to the registration write so the check isn't yet registered when agentCheckPass fires; network blip between the Druid node and the local Consul agent; auth token not propagated in config.","solutions":["Verify the Consul ACL token used by config.getAuth().getAclToken() includes service:check-write (and service:write) for the service prefix — this is the most common cause after enabling ACLs","Check connectivity to the local Consul agent (curl http://localhost:8500/v1/agent/checks) and retry; if transient, no action is needed since the periodic check self-heals","If the warning repeats every cycle, inspect druid console logs for the underlying exception (the LOGGER.warn includes `e`) and confirm the check id naming convention 'service:<serviceId>' matches what registerService created","Increase health-check interval tolerance temporarily, or pass the check state at registration time instead of a separate agentCheckPass call"],"exampleFix":"// before: check written separately after registration, can fail transiently\nconsulClient.agentCheckPass(\"service:\" + serviceId, \"Druid node is healthy\", config.getAuth().getAclToken());\n// after: align ACL policy so the token can update checks\n// service \"druid\" { policy = \"write\" }\n// (or) pass note/output during check registration:\nservice { id = \"<serviceId>\"; check { ttl = \"10s\"; status = \"passing\" } }","handlingStrategy":"fallback","validationCode":"// preflight: confirm the agent accepts check writes with this token\ncurl -s -H \"X-Consul-Token: $ACL_TOKEN\" http://localhost:8500/v1/agent/checks | jq 'keys'\n// and validate the ACL policy locally before starting the node:\nconsul acl token read -id <token> | grep -i 'service:check-write'","typeGuard":null,"tryCatchPattern":"try {\n  consulClient.agentCheckPass(\"service:\" + serviceId, \"Druid node is healthy\", aclToken);\n} catch (Exception e) {\n  LOGGER.warn(e, \"Failed to immediately mark service [%s] as healthy, will retry via periodic health check\", serviceId);\n  // non-fatal: rely on periodic TTL check pass\n}","preventionTips":["Grant the Consul ACL token service:check-write (and service:write) for the service prefix","Ensure the local Consul agent is reachable and responsive at node startup","Set the TTL check interval comfortably larger than the immediate-pass retry so brief failures self-heal","Alert on repeated occurrences — repeated warnings indicate a persistent ACL or connectivity problem, not a transient one"],"tags":["consul","service-registration","health-check","acl"],"backgroundTag":"http-request-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}