{"record":{"id":"069b71bbbb0eb425","repo":"apache/druid","slug":"ownership-check-failed-for-s","errorCode":null,"errorMessage":"Ownership check failed for [%s]","messagePattern":"Ownership check failed for \\[(.+?)\\]","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":452,"sourceCode":"    catch (Exception e) {\n      LOGGER.error(e, \"Failed to acquire lock on key [%s]\", lockKey);\n      return false;\n    }\n  }\n\n  private void becomeLeader()\n  {\n    String currentSession = this.sessionId;\n\n    if (currentSession == null || stopping || Thread.currentThread().isInterrupted()) {\n      LOGGER.warn(\"Aborting promotion: session=%s, stopping=%s\",\n                  currentSession != null ? currentSession.substring(0, Math.min(8, currentSession.length())) + \"...\" : \"null\",\n                  stopping);\n      return;\n    }\n\n    if (!validateLockOwnership(currentSession)) {\n      LOGGER.warn(\"Ownership check failed for [%s]\", lockKey);\n      emitOwnershipMismatchMetric();\n      return;\n    }\n\n    if (!leader.compareAndSet(false, true)) {\n      LOGGER.info(\"Already leader for [%s]\", lockKey);\n      return;\n    }\n\n    // Re-validate after CAS to handle race conditions\n    if (!validateLockOwnership(currentSession)) {\n      leader.set(false);\n      LOGGER.error(\"Lost ownership during promotion for [%s]\", lockKey);\n      emitOwnershipMismatchMetric();\n      return;\n    }\n\n    int newTerm = term.incrementAndGet();","sourceCodeStart":434,"sourceCodeEnd":470,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L434-L470","documentation":"Warning in becomeLeader() when the pre-promotion validateLockOwnership() check fails: although the lock acquire returned true moments earlier, the KV key is now missing, owned by a different session, or unreadable. The selector aborts promotion (never calls listener.becomeLeader()) and emits an ownership_mismatch metric, leaving the loop to retry.","triggerScenarios":"Race between tryAcquireLock success and the ownership read in becomeLeader — the session expired and Consul released the key, another node acquired the key after lock-delay, or the KV read threw an exception (validateLockOwnership returns false on error).","commonSituations":"Highly contended leader elections with multiple nodes on the same lockKey, Consul sessions near TTL expiry during election, Consul flakiness causing failed KV reads, or stale sessions holding the key from a prior crash.","solutions":["Let the loop retry — promotion is safely aborted; no corrupted state results.","Verify session TTL and renewal health so the session doesn't lapse mid-election.","Check for competing selectors using the same lockKey and remove duplicates.","Inspect the KV key's owning session to identify a stale/competing holder and destroy it."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before treating a node as leader, confirm from Consul\nResponse<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);\nboolean safe = kv != null && kv.getValue() != null\n    && expectedSessionId.equals(kv.getValue().getSession());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use one selector per lockKey cluster-wide","Keep sessions renewed during elections","Alert on consul/leader/ownership_mismatch"],"tags":["consul","leader-election","race-condition","kv-lock"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}