{"record":{"id":"7798e43151864e46","repo":"apache/druid","slug":"lock-ownership-validation-failed-for-s-will-re","errorCode":null,"errorMessage":"Lock ownership validation failed for [%s]; will retry","messagePattern":"Lock ownership validation failed for \\[(.+?)\\]; will retry","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":272,"sourceCode":"\n        if (acquired && !leader.get()) {\n          boolean interrupted = Thread.currentThread().isInterrupted();\n          if (stopping || interrupted) {\n            LOGGER.info(\n                \"Skipping leadership for [%s] because selector is stopping (interrupted=%s)\",\n                lockKey,\n                interrupted\n            );\n          } else if (sessionId == null) {\n            LOGGER.warn(\"Skipping leadership for [%s] because session is null\", lockKey);\n          } else if (validateLockOwnership(sessionId)) {\n            long electionStart = System.nanoTime();\n            becomeLeader();\n            long electionLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - electionStart);\n            ConsulMetrics.emitTimer(emitter, \"consul/leader/election_latency\", electionLatency,\n                \"lock\", lockKey);\n          } else {\n            LOGGER.warn(\"Lock ownership validation failed for [%s]; will retry\", lockKey);\n            emitOwnershipMismatchMetric();\n          }\n        } else if (!acquired && leader.get()) {\n          loseLeadership();\n        }\n\n        if (leader.get()) {\n          // Session renewal handled by sessionKeeperLoop; here we just verify lock ownership\n          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());\n          if (sessionId != null && !validateLockOwnership(sessionId)) {\n            LOGGER.warn(\"Main Loop: Lost lock ownership check for [%s], stepping down\", lockKey);\n            loseLeadership();\n          }\n        } else {\n          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());\n        }\n        errorRetryCount = 0;\n      }","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L254-L290","documentation":"Warning logged when the loop acquired the Consul KV lock but validateLockOwnership() shows the key's session does not match the local session id (or validation failed). The selector refuses to promote to leader and emits a consul/leader/ownership_mismatch metric; it will retry on the next cycle.","triggerScenarios":"Between tryAcquireLock success and validation, the lock's owning session changed — another node acquired the key, the key was deleted and recreated, or the KV read returned a value owned by a stale session. Also fires when the KV read itself errors (validateLockOwnership returns false on exception).","commonSituations":"Split-brain recovery after a Consul outage, two nodes racing for the same lockKey (duplicate lockKey config), manual KV edits under the lock key, or lock-delay quirks right after session invalidation.","solutions":["Verify only one Druid service uses this lockKey; check for duplicate configurations across nodes.","Inspect the KV entry (consul kv get <lockKey>) to see which session owns it and from where.","After a Consul outage, wait out the 5s lock-delay and let the loop retry; sessions get recreated automatically.","If it persists, check ACL read consistency and whether a stale session is pinned to the key; delete the stale session."],"exampleFix":"// before: two services sharing the same election key with conflicting selectors\n// ensure only the intended role's selector uses druid/leader/coordinator,\n// then destroy the stale session:\n//   curl http://consul:8500/v1/session/node/<node>?dc=dc1  (inspect) and destroy stale session","handlingStrategy":"retry","validationCode":"// before assuming leadership externally\nResponse<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);\nboolean ownedByUs = kv.getValue() != null && expectedSessionId.equals(kv.getValue().getSession());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never share a lockKey across unrelated selector instances","Alert on the consul/leader/ownership_mismatch metric","Rely on session Behavior=DELETE so locks release when sessions die"],"tags":["consul","leader-election","race-condition","split-brain","kv-lock"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}