{"record":{"id":"52a556662f191ef4","repo":"apache/druid","slug":"lock-key-s-missing-when-validating-ownership","errorCode":null,"errorMessage":"Lock key [%s] missing when validating ownership","messagePattern":"Lock key \\[(.+?)\\] missing when validating ownership","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":506,"sourceCode":"      try {\n        consulClient.sessionDestroy(sessionId, buildQueryParams(), config.getAuth().getAclToken());\n      }\n      catch (Exception e) {\n        LOGGER.error(e, \"Failed to destroy session [%s]\", sessionId);\n      }\n    }\n  }\n\n  private boolean validateLockOwnership(String expectedSessionId)\n  {\n    try {\n      Response<GetValue> response = consulClient.getKVValue(\n          lockKey,\n          config.getAuth().getAclToken(),\n          buildQueryParams()\n      );\n      if (response == null || response.getValue() == null) {\n        LOGGER.warn(\"Lock key [%s] missing when validating ownership\", lockKey);\n        return false;\n      }\n      String actualSessionId = response.getValue().getSession();\n      if (actualSessionId == null) {\n        LOGGER.warn(\"Lock key [%s] has no session owner\", lockKey);\n        return false;\n      }\n      boolean matches = expectedSessionId.equals(actualSessionId);\n      if (!matches) {\n        LOGGER.warn(\n            \"Lock key [%s] owned by session [%s], expected [%s]\",\n            lockKey,\n            actualSessionId,\n            expectedSessionId\n        );\n      }\n      return matches;\n    }","sourceCodeStart":488,"sourceCodeEnd":524,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L488-L524","documentation":"Warning in validateLockOwnership() when the Consul KV read for the lock key returns no value — the key does not exist at validation time, so ownership cannot be confirmed and the method returns false. Callers (leaderElectionLoop, becomeLeader) then skip/abort promotion or step down.","triggerScenarios":"Reading getKVValue(lockKey) returns a null value because the key was deleted (session expiry with Behavior=DELETE releases the lock entry), the key was never created because tryAcquireLock never succeeded, an operator deleted the KV entry, or the query targeted the wrong datacenter.","commonSituations":"Right after a leader session expires, the lock key disappears and followers see this before re-acquiring; manual consul kv delete operations; druid.discovery.consul.service.datacenter misconfigured so the key isn't visible in the queried DC.","solutions":["Usually transient: the loop re-acquires the lock and recreates the key automatically.","Verify druid.discovery.consul.service.datacenter matches the DC where the key lives.","Ensure no automation/operator deletes the leader lock key.","If persistent, check ACL token read permissions on the key and Consul KV health."],"exampleFix":"// before\n// dc1 holds the key, but query targets another DC\ndruid.discovery.consul.service.datacenter=dc2\n// after\ndruid.discovery.consul.service.datacenter=dc1","handlingStrategy":"retry","validationCode":"// pre-check the key exists before relying on leadership state\nResponse<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);\nif (kv == null || kv.getValue() == null) {\n  // lock key absent: no leader currently, trigger a new election\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Confirm datacenter config points at the DC holding the key","Treat the lock key as ephemeral — never delete it manually","Alert on sustained absence of the lock key"],"tags":["consul","kv-lock","resource-not-found","leader-election"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}