Session creation failed for [%s]; backing off before retry

Error message

Session creation failed for [%s]; backing off before retry

What it means

Warning logged by the leader-election loop when consulClient.sessionCreate() yields no session id (null), meaning Consul refused or failed to create the lock session. The loop backs off for healthCheckInterval before retrying to avoid a hot spin against Consul.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:239

  private void startLeaderElection()
  {
    executorService.submit(this::leaderElectionLoop);
    sessionKeeperService.submit(this::sessionKeeperLoop);
  }

  private void leaderElectionLoop()
  {
    LOGGER.info("Starting leader election loop for [%s]", lockKey);
    ConsulMetrics.emitCount(emitter, "consul/leader/loop", "lock", lockKey, "state", "start");

    while (!stopping && !Thread.currentThread().isInterrupted()) {
      try {
        if (sessionId == null) {
          sessionId = createSession();
          if (sessionId == null) {
            // Session creation failed; back off before retrying to avoid tight loop
            LOGGER.warn("Session creation failed for [%s]; backing off before retry", lockKey);
            Thread.sleep(config.getService().getHealthCheckInterval().getMillis());
            continue;
          }
        }

        if (!leader.get()) {
          if (!isSessionValid(sessionId)) {
            LOGGER.info("Follower session [%s] expired or invalid, recreating", shortSessionId(sessionId));
            sessionId = null;
            continue;
          }
        }

        boolean acquired = tryAcquireLock(sessionId);

        if (acquired && !leader.get()) {
          boolean interrupted = Thread.currentThread().isInterrupted();
          if (stopping || interrupted) {

View on GitHub (pinned to 9b90983fd2)

Solutions

Check Consul agent health and connectivity from the Druid node (consul members / /v1/agent/health).
Verify the ACL token has session:create permission for the datacenter.
Set leaderSessionTtl within the agent's [min_session_ttl, max_session_ttl] bounds.
Fix the datacenter config if druid.discovery.consul.service.datacenter points to a non-existent DC.

Example fix

// before
druid.discovery.consul.leader.sessionTtl=PT2S // below agent min_session_ttl
// after
druid.discovery.consul.leader.sessionTtl=PT30S

Defensive patterns

Strategy: retry

Validate before calling

// before enabling the selector: verify token session write access and TTL bounds
long ttl = config.getLeader().getLeaderSessionTtl().getStandardSeconds();
if (ttl < 10 || ttl > 86400) { throw new IllegalStateException("TTL outside Consul agent bounds"); }

Try / catch

// the loop already retries with backoff; alert on sustained session-creation failures

Prevention

Pre-flight check ACL token session permissions
Keep TTL between agent min_session_ttl and max_session_ttl
Monitor Consul agent health in deployment checks

When it happens

Trigger: createSession() returns null or a response with no value — typically because the Consul agent is down, the ACL token lacks session write permissions, the configured TTL is rejected (e.g. below the agent's min TTL), or the datacenter is wrong.

Common situations: Consul agent restarted/unreachable during a deploy, ACL token rotation dropping 'session' write rights, setting leaderSessionTtl below the Consul agent's min_session_ttl, or misconfigured druid.discovery.consul.service.datacenter.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/526c03b8696b5362. Report an issue: GitHub.