{"record":{"id":"ea0ca03644ce9afb","repo":"apache/druid","slug":"lock-key-s-owned-by-session-s-expected-s","errorCode":null,"errorMessage":"Lock key [%s] owned by session [%s], expected [%s]","messagePattern":"Lock key \\[(.+?)\\] owned by session \\[(.+?)\\], expected \\[(.+?)\\]","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java","lineNumber":516,"sourceCode":"  {\n    try {\n      Response<GetValue> response = consulClient.getKVValue(\n          lockKey,\n          config.getAuth().getAclToken(),\n          buildQueryParams()\n      );\n      if (response == null || response.getValue() == null) {\n        LOGGER.warn(\"Lock key [%s] missing when validating ownership\", lockKey);\n        return false;\n      }\n      String actualSessionId = response.getValue().getSession();\n      if (actualSessionId == null) {\n        LOGGER.warn(\"Lock key [%s] has no session owner\", lockKey);\n        return false;\n      }\n      boolean matches = expectedSessionId.equals(actualSessionId);\n      if (!matches) {\n        LOGGER.warn(\n            \"Lock key [%s] owned by session [%s], expected [%s]\",\n            lockKey,\n            actualSessionId,\n            expectedSessionId\n        );\n      }\n      return matches;\n    }\n    catch (Exception e) {\n      LOGGER.error(e, \"Failed to validate lock ownership for [%s]\", lockKey);\n      return false;\n    }\n  }\n\n  private void emitOwnershipMismatchMetric()\n  {\n    ConsulMetrics.emitCount(\n        emitter,","sourceCodeStart":498,"sourceCodeEnd":534,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java#L498-L534","documentation":"This warning is logged by ConsulLeaderSelector.validateLockOwnership when the Consul lock key exists and has a session, but that session id does not match the local node's expected session id. It means another live session currently owns the leadership lock, so validateLockOwnership returns false and becomeLeader aborts promotion (emitting the ownership-mismatch metric). This is the normal guard that prevents two nodes from simultaneously acting as leader.","triggerScenarios":"leaderElectionLoop or becomeLeader calls validateLockOwnership(currentSession) right after acquiring or renewing a session; getKVValue returns a GetValue whose getSession() differs from the local sessionId — typically because another replica won the lock first, or the local session was recreated with a new id while the old key still references the previous session.","commonSituations":"Two Druid nodes configured with the same lockKey contending for leadership (expected); a node that lost its session (TTL expiry) created a new session but hasn't yet reacquired the key; Consul session data stale after a failover; misconfigured identical node identifiers or shared lock key across unrelated clusters.","solutions":["Confirm which node holds the lock via consul kv get <lockKey> and the session's node — if it's a healthy peer, this is normal contention and no action is needed","If the owning session is stale/dead, delete the key or destroy the session (consul session destroy <id>) so a new election can occur","Check for session churn: GC pauses or network flaps causing the local session to be recreated and lose the key; tune lock-ttl and renew interval","Ensure each cluster uses a unique lockKey prefix so unrelated deployments don't fight over the same key"],"exampleFix":"// before: local session recreated but key still owned by old session\nString actualSessionId = response.getValue().getSession(); // \"abc-123\"\nboolean matches = expectedSessionId.equals(actualSessionId); // expected \"def-456\" -> false\n// after: destroy the stale session then re-run the election loop\n$ consul session destroy abc-123\n$ consul kv delete druid/leader-lock","handlingStrategy":"retry","validationCode":"Response<GetValue> r = consulClient.getKVValue(lockKey, aclToken, qp);\nif (r != null && r.getValue() != null && r.getValue().getSession() != null\n    && !expectedSessionId.equals(r.getValue().getSession())) {\n  // another session owns the lock: skip promotion this cycle\n}","typeGuard":"static boolean ownsLock(GetValue v, String mySession) {\n  return v != null && v.getSession() != null && mySession.equals(v.getSession());\n}","tryCatchPattern":"if (!validateLockOwnership(sessionId)) {\n  leader.set(false);\n  emitOwnershipMismatchMetric();\n  return; // do not promote; the next leaderElectionLoop iteration will retry\n}","preventionTips":["Ensure only one cluster shares a given lockKey prefix","Tune session TTL/renewal to avoid session recreation churn that strands the key under an old session id","Monitor consul/leader/ownership_mismatch for persistent mismatch vs transient contention","Verify peer health before assuming the mismatch is a fault — a healthy leader owning the lock is expected"],"tags":["consul","distributed-lock","leader-election","session-mismatch"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}