apache/druid · warning

Session Keeper: Failed to renew session [%s], it may have ex

Error message

Session Keeper: Failed to renew session [%s], it may have expired

What it means

Warning from the sessionKeeperLoop when consulClient.renewSession() returns null or a response without a value, meaning Consul no longer knows the session — it has expired (TTL lapsed without renewal) or was destroyed. If the node believed it was leader, an error is logged and the main loop's ownership check will trigger step down.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:350

  private void sessionKeeperLoop()
  {
    if (stopping) {
      return;
    }

    String currentSessionId = this.sessionId;
    if (currentSessionId != null) {
      final String truncatedId = shortSessionId(currentSessionId);
      try {
        Response<Session> response = consulClient.renewSession(
            currentSessionId,
            buildQueryParams(),
            config.getAuth().getAclToken()
        );

        if (response == null || response.getValue() == null) {
          LOGGER.warn("Session Keeper: Failed to renew session [%s], it may have expired", truncatedId);
          // Don't null it out here, main loop handles recreating if it fails to use it
          // But we can signal leadership loss if we thought we were leader
          if (leader.get()) {
            LOGGER.error("Session Keeper: Leader lost session [%s], triggering step down", truncatedId);
            // Trigger immediate check in main loop or let main loop fail on next check
            // Ideally, we could interrupt main loop, but for safety we let main loop handle state
          }
          ConsulMetrics.emitCount(emitter, "consul/leader/renew/fail", "lock", lockKey);
        } else {
          LOGGER.debug("Session Keeper: Successfully renewed session [%s]", truncatedId);
        }
      }
      catch (Exception e) {
        LOGGER.error(e, "Session Keeper: Exception renewing session [%s]", truncatedId);
        ConsulMetrics.emitCount(emitter, "consul/leader/renew/fail", "lock", lockKey);
      }
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Reduce healthCheckInterval so renewals (interval/3) are comfortably more frequent than leaderSessionTtl.
2. Increase leaderSessionTtl (up to 120s) to absorb renewal hiccups.
3. Fix GC pauses or network latency between Druid and the Consul agent.
4. No manual recovery needed: the main loop detects lost ownership and re-creates a session to re-elect.

Example fix

// before
druid.discovery.consul.service.healthCheckInterval=PT1M
druid.discovery.consul.leader.sessionTtl=PT10S
// after
druid.discovery.consul.service.healthCheckInterval=PT10S
druid.discovery.consul.leader.sessionTtl=PT30S

long ttl = config.getLeader().getLeaderSessionTtl().getStandardSeconds();
long renewEvery = config.getService().getHealthCheckInterval().getStandardSeconds() / 3;
if (renewEvery * 3 >= ttl) {
  throw new IllegalStateException("healthCheckInterval too large: renewal cadence must be well under TTL");
}

When it happens

Trigger: The keeper's scheduled renewal (every healthCheckInterval/3) hits an expired session — renewal RPCs were delayed by network issues, GC pauses, or Consul agent restarts, letting the TTL lapse; or the session was destroyed externally.

Common situations: Long GC pauses on the Druid process, Consul agent outage longer than the TTL, healthCheckInterval so large that renewal cadence approaches the TTL, or clock/network latency spikes in busy clusters.

Related errors

• leaderSessionTtl is %s; leader failover may take up to %s
• Main Loop: Lost lock ownership check for [%s], stepping down
• can't start
• can't stop
• watchSeconds (%ds) is much larger than leaderSessionTtl (%ds

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/61885cd444f802f2. Report an issue: GitHub.