{"record":{"id":"526c03b8696b5362","repo":"apache/druid","slug":"session-creation-failed-for-s-backing-off-befo","errorCode":null,"errorMessage":"Session creation failed for [%s]; backing off before retry","messagePattern":"Session creation failed for \\[(.+?)\\]; backing off before retry","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":239,"sourceCode":"\n  private void startLeaderElection()\n  {\n    executorService.submit(this::leaderElectionLoop);\n    sessionKeeperService.submit(this::sessionKeeperLoop);\n  }\n\n  private void leaderElectionLoop()\n  {\n    LOGGER.info(\"Starting leader election loop for [%s]\", lockKey);\n    ConsulMetrics.emitCount(emitter, \"consul/leader/loop\", \"lock\", lockKey, \"state\", \"start\");\n\n    while (!stopping && !Thread.currentThread().isInterrupted()) {\n      try {\n        if (sessionId == null) {\n          sessionId = createSession();\n          if (sessionId == null) {\n            // Session creation failed; back off before retrying to avoid tight loop\n            LOGGER.warn(\"Session creation failed for [%s]; backing off before retry\", lockKey);\n            Thread.sleep(config.getService().getHealthCheckInterval().getMillis());\n            continue;\n          }\n        }\n\n        if (!leader.get()) {\n          if (!isSessionValid(sessionId)) {\n            LOGGER.info(\"Follower session [%s] expired or invalid, recreating\", shortSessionId(sessionId));\n            sessionId = null;\n            continue;\n          }\n        }\n\n        boolean acquired = tryAcquireLock(sessionId);\n\n        if (acquired && !leader.get()) {\n          boolean interrupted = Thread.currentThread().isInterrupted();\n          if (stopping || interrupted) {","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L221-L257","documentation":"Warning logged by the leader-election loop when consulClient.sessionCreate() yields no session id (null), meaning Consul refused or failed to create the lock session. The loop backs off for healthCheckInterval before retrying to avoid a hot spin against Consul.","triggerScenarios":"createSession() returns null or a response with no value — typically because the Consul agent is down, the ACL token lacks session write permissions, the configured TTL is rejected (e.g. below the agent's min TTL), or the datacenter is wrong.","commonSituations":"Consul agent restarted/unreachable during a deploy, ACL token rotation dropping 'session' write rights, setting leaderSessionTtl below the Consul agent's min_session_ttl, or misconfigured druid.discovery.consul.service.datacenter.","solutions":["Check Consul agent health and connectivity from the Druid node (consul members / /v1/agent/health).","Verify the ACL token has session:create permission for the datacenter.","Set leaderSessionTtl within the agent's [min_session_ttl, max_session_ttl] bounds.","Fix the datacenter config if druid.discovery.consul.service.datacenter points to a non-existent DC."],"exampleFix":"// before\ndruid.discovery.consul.leader.sessionTtl=PT2S // below agent min_session_ttl\n// after\ndruid.discovery.consul.leader.sessionTtl=PT30S","handlingStrategy":"retry","validationCode":"// before enabling the selector: verify token session write access and TTL bounds\nlong ttl = config.getLeader().getLeaderSessionTtl().getStandardSeconds();\nif (ttl < 10 || ttl > 86400) { throw new IllegalStateException(\"TTL outside Consul agent bounds\"); }","typeGuard":null,"tryCatchPattern":"// the loop already retries with backoff; alert on sustained session-creation failures","preventionTips":["Pre-flight check ACL token session permissions","Keep TTL between agent min_session_ttl and max_session_ttl","Monitor Consul agent health in deployment checks"],"tags":["consul","session","leader-election","acl","retry"],"backgroundTag":"upstream-api-error","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}