{"record":{"id":"61885cd444f802f2","repo":"apache/druid","slug":"session-keeper-failed-to-renew-session-s-it-m","errorCode":null,"errorMessage":"Session Keeper: Failed to renew session [%s], it may have expired","messagePattern":"Session Keeper: Failed to renew session \\[(.+?)\\], it may have expired","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":350,"sourceCode":"\n  private void sessionKeeperLoop()\n  {\n    if (stopping) {\n      return;\n    }\n\n    String currentSessionId = this.sessionId;\n    if (currentSessionId != null) {\n      final String truncatedId = shortSessionId(currentSessionId);\n      try {\n        Response<Session> response = consulClient.renewSession(\n            currentSessionId,\n            buildQueryParams(),\n            config.getAuth().getAclToken()\n        );\n\n        if (response == null || response.getValue() == null) {\n          LOGGER.warn(\"Session Keeper: Failed to renew session [%s], it may have expired\", truncatedId);\n          // Don't null it out here, main loop handles recreating if it fails to use it\n          // But we can signal leadership loss if we thought we were leader\n          if (leader.get()) {\n            LOGGER.error(\"Session Keeper: Leader lost session [%s], triggering step down\", truncatedId);\n            // Trigger immediate check in main loop or let main loop fail on next check\n            // Ideally, we could interrupt main loop, but for safety we let main loop handle state\n          }\n          ConsulMetrics.emitCount(emitter, \"consul/leader/renew/fail\", \"lock\", lockKey);\n        } else {\n          LOGGER.debug(\"Session Keeper: Successfully renewed session [%s]\", truncatedId);\n        }\n      }\n      catch (Exception e) {\n        LOGGER.error(e, \"Session Keeper: Exception renewing session [%s]\", truncatedId);\n        ConsulMetrics.emitCount(emitter, \"consul/leader/renew/fail\", \"lock\", lockKey);\n      }\n    }\n","sourceCodeStart":332,"sourceCodeEnd":368,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L332-L368","documentation":"Warning from the sessionKeeperLoop when consulClient.renewSession() returns null or a response without a value, meaning Consul no longer knows the session — it has expired (TTL lapsed without renewal) or was destroyed. If the node believed it was leader, an error is logged and the main loop's ownership check will trigger step down.","triggerScenarios":"The keeper's scheduled renewal (every healthCheckInterval/3) hits an expired session — renewal RPCs were delayed by network issues, GC pauses, or Consul agent restarts, letting the TTL lapse; or the session was destroyed externally.","commonSituations":"Long GC pauses on the Druid process, Consul agent outage longer than the TTL, healthCheckInterval so large that renewal cadence approaches the TTL, or clock/network latency spikes in busy clusters.","solutions":["Reduce healthCheckInterval so renewals (interval/3) are comfortably more frequent than leaderSessionTtl.","Increase leaderSessionTtl (up to 120s) to absorb renewal hiccups.","Fix GC pauses or network latency between Druid and the Consul agent.","No manual recovery needed: the main loop detects lost ownership and re-creates a session to re-elect."],"exampleFix":"// before\ndruid.discovery.consul.service.healthCheckInterval=PT1M\ndruid.discovery.consul.leader.sessionTtl=PT10S\n// after\ndruid.discovery.consul.service.healthCheckInterval=PT10S\ndruid.discovery.consul.leader.sessionTtl=PT30S","handlingStrategy":"retry","validationCode":"long ttl = config.getLeader().getLeaderSessionTtl().getStandardSeconds();\nlong renewEvery = config.getService().getHealthCheckInterval().getStandardSeconds() / 3;\nif (renewEvery * 3 >= ttl) {\n  throw new IllegalStateException(\"healthCheckInterval too large: renewal cadence must be well under TTL\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep renewal cadence at least 3x faster than the TTL","Alert on consul/leader/renew/fail metric","Keep JVM GC pauses short on nodes running leader election"],"tags":["consul","session-renewal","ttl","session-expiry","leader-election"],"backgroundTag":"session-expired","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}