apache/druid · warning

Lock key [%s] owned by session [%s], expected [%s]

Error message

Lock key [%s] owned by session [%s], expected [%s]

What it means

This warning is logged by ConsulLeaderSelector.validateLockOwnership when the Consul lock key exists and has a session, but that session id does not match the local node's expected session id. It means another live session currently owns the leadership lock, so validateLockOwnership returns false and becomeLeader aborts promotion (emitting the ownership-mismatch metric). This is the normal guard that prevents two nodes from simultaneously acting as leader.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:516

  {
    try {
      Response<GetValue> response = consulClient.getKVValue(
          lockKey,
          config.getAuth().getAclToken(),
          buildQueryParams()
      );
      if (response == null || response.getValue() == null) {
        LOGGER.warn("Lock key [%s] missing when validating ownership", lockKey);
        return false;
      }
      String actualSessionId = response.getValue().getSession();
      if (actualSessionId == null) {
        LOGGER.warn("Lock key [%s] has no session owner", lockKey);
        return false;
      }
      boolean matches = expectedSessionId.equals(actualSessionId);
      if (!matches) {
        LOGGER.warn(
            "Lock key [%s] owned by session [%s], expected [%s]",
            lockKey,
            actualSessionId,
            expectedSessionId
        );
      }
      return matches;
    }
    catch (Exception e) {
      LOGGER.error(e, "Failed to validate lock ownership for [%s]", lockKey);
      return false;
    }
  }

  private void emitOwnershipMismatchMetric()
  {
    ConsulMetrics.emitCount(
        emitter,

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Confirm which node holds the lock via consul kv get <lockKey> and the session's node — if it's a healthy peer, this is normal contention and no action is needed
2. If the owning session is stale/dead, delete the key or destroy the session (consul session destroy <id>) so a new election can occur
3. Check for session churn: GC pauses or network flaps causing the local session to be recreated and lose the key; tune lock-ttl and renew interval
4. Ensure each cluster uses a unique lockKey prefix so unrelated deployments don't fight over the same key

Example fix

// before: local session recreated but key still owned by old session
String actualSessionId = response.getValue().getSession(); // "abc-123"
boolean matches = expectedSessionId.equals(actualSessionId); // expected "def-456" -> false
// after: destroy the stale session then re-run the election loop
$ consul session destroy abc-123
$ consul kv delete druid/leader-lock

Response<GetValue> r = consulClient.getKVValue(lockKey, aclToken, qp);
if (r != null && r.getValue() != null && r.getValue().getSession() != null
    && !expectedSessionId.equals(r.getValue().getSession())) {
  // another session owns the lock: skip promotion this cycle
}

static boolean ownsLock(GetValue v, String mySession) {
  return v != null && v.getSession() != null && mySession.equals(v.getSession());
}

if (!validateLockOwnership(sessionId)) {
  leader.set(false);
  emitOwnershipMismatchMetric();
  return; // do not promote; the next leaderElectionLoop iteration will retry
}

When it happens

Trigger: leaderElectionLoop or becomeLeader calls validateLockOwnership(currentSession) right after acquiring or renewing a session; getKVValue returns a GetValue whose getSession() differs from the local sessionId — typically because another replica won the lock first, or the local session was recreated with a new id while the old key still references the previous session.

Common situations: Two Druid nodes configured with the same lockKey contending for leadership (expected); a node that lost its session (TTL expiry) created a new session but hasn't yet reacquired the key; Consul session data stale after a failover; misconfigured identical node identifiers or shared lock key across unrelated clusters.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Lock key [%s] has no session owner
• can't start
• can't stop
• watchSeconds (%ds) is much larger than leaderSessionTtl (%ds
• leaderSessionTtl is %s; leader failover may take up to %s

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ea0ca03644ce9afb. Report an issue: GitHub.