{"record":{"id":"4a4d34a2bdc01ce5","repo":"apache/druid","slug":"main-loop-lost-lock-ownership-check-for-s-ste","errorCode":null,"errorMessage":"Main Loop: Lost lock ownership check for [%s], stepping down","messagePattern":"Main Loop: Lost lock ownership check for \\[(.+?)\\], stepping down","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":283,"sourceCode":"          } else if (validateLockOwnership(sessionId)) {\n            long electionStart = System.nanoTime();\n            becomeLeader();\n            long electionLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - electionStart);\n            ConsulMetrics.emitTimer(emitter, \"consul/leader/election_latency\", electionLatency,\n                \"lock\", lockKey);\n          } else {\n            LOGGER.warn(\"Lock ownership validation failed for [%s]; will retry\", lockKey);\n            emitOwnershipMismatchMetric();\n          }\n        } else if (!acquired && leader.get()) {\n          loseLeadership();\n        }\n\n        if (leader.get()) {\n          // Session renewal handled by sessionKeeperLoop; here we just verify lock ownership\n          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());\n          if (sessionId != null && !validateLockOwnership(sessionId)) {\n            LOGGER.warn(\"Main Loop: Lost lock ownership check for [%s], stepping down\", lockKey);\n            loseLeadership();\n          }\n        } else {\n          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());\n        }\n        errorRetryCount = 0;\n      }\n      catch (InterruptedException e) {\n        Thread.currentThread().interrupt();\n        break;\n      }\n      catch (Exception e) {\n        LOGGER.error(e, \"Error in leader election loop\");\n\n        if (leader.get()) {\n          loseLeadership();\n        }\n        sessionId = null;","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L265-L301","documentation":"Warning logged by the leader-election loop when a periodic re-check shows the Consul KV lock key is no longer owned by this node's session. The selector steps down (loseLeadership) so the listener's stopBeingLeader() runs and another node can take over. This is the primary fail-safe against silently holding leadership after losing the underlying lock.","triggerScenarios":"While leader, the scheduled validateLockOwnership() finds the key missing, owned by another session, or the KV read throws — e.g. the session expired (renewal failures), another node force-acquired the key, or the KV entry was deleted.","commonSituations":"Consul agent restart or network partition causing session expiry, leaderSessionTtl too short for the renewal cadence, an operator deleting the KV key, or a competing node overwriting the key.","solutions":["Fix session renewal reliability: ensure healthCheckInterval/3 renewals succeed; check 'Failed to renew session' warnings.","Increase leaderSessionTtl (up to 120s) if transient RPC latency causes expiry between renewals.","Verify Consul agent stability and network between the Druid node and Consul.","Confirm no other process writes to the lock key; after step-down, the loop will re-elect automatically."],"exampleFix":"// before: TTL 10s with RPC hiccups expiring it between renewals\ndruid.discovery.consul.leader.sessionTtl=PT10S\n// after\ndruid.discovery.consul.leader.sessionTtl=PT30S","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// listeners must tolerate stopBeingLeader() at any time\npublic void stopBeingLeader() {\n  if (!currentlyLeading) return; // idempotent step-down\n  // release resources\n}","preventionTips":["Make listener.becomeLeader/stopBeingLeader idempotent and fast","Renew session well within TTL (keeper renews at healthCheckInterval/3)","Keep TTL >= 3x expected Consul RPC latency","Monitor consul/leader/stop and renew/fail metrics"],"tags":["consul","leader-election","failover","session-expiry","step-down"],"backgroundTag":"leader-lock-lost","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}