apache/druid · warning

Ownership check failed for [%s]

Error message

Ownership check failed for [%s]

What it means

Warning in becomeLeader() when the pre-promotion validateLockOwnership() check fails: although the lock acquire returned true moments earlier, the KV key is now missing, owned by a different session, or unreadable. The selector aborts promotion (never calls listener.becomeLeader()) and emits an ownership_mismatch metric, leaving the loop to retry.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:452

    catch (Exception e) {
      LOGGER.error(e, "Failed to acquire lock on key [%s]", lockKey);
      return false;
    }
  }

  private void becomeLeader()
  {
    String currentSession = this.sessionId;

    if (currentSession == null || stopping || Thread.currentThread().isInterrupted()) {
      LOGGER.warn("Aborting promotion: session=%s, stopping=%s",
                  currentSession != null ? currentSession.substring(0, Math.min(8, currentSession.length())) + "..." : "null",
                  stopping);
      return;
    }

    if (!validateLockOwnership(currentSession)) {
      LOGGER.warn("Ownership check failed for [%s]", lockKey);
      emitOwnershipMismatchMetric();
      return;
    }

    if (!leader.compareAndSet(false, true)) {
      LOGGER.info("Already leader for [%s]", lockKey);
      return;
    }

    // Re-validate after CAS to handle race conditions
    if (!validateLockOwnership(currentSession)) {
      leader.set(false);
      LOGGER.error("Lost ownership during promotion for [%s]", lockKey);
      emitOwnershipMismatchMetric();
      return;
    }

    int newTerm = term.incrementAndGet();

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Let the loop retry — promotion is safely aborted; no corrupted state results.
2. Verify session TTL and renewal health so the session doesn't lapse mid-election.
3. Check for competing selectors using the same lockKey and remove duplicates.
4. Inspect the KV key's owning session to identify a stale/competing holder and destroy it.

// before treating a node as leader, confirm from Consul
Response<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);
boolean safe = kv != null && kv.getValue() != null
    && expectedSessionId.equals(kv.getValue().getSession());

When it happens

Trigger: Race between tryAcquireLock success and the ownership read in becomeLeader — the session expired and Consul released the key, another node acquired the key after lock-delay, or the KV read threw an exception (validateLockOwnership returns false on error).

Common situations: Highly contended leader elections with multiple nodes on the same lockKey, Consul sessions near TTL expiry during election, Consul flakiness causing failed KV reads, or stale sessions holding the key from a prior crash.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Lock ownership validation failed for [%s]; will retry
• Skipping leadership for [%s] because session is null
• Lock key [%s] missing when validating ownership
• can't start
• can't stop

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/069b71bbbb0eb425. Report an issue: GitHub.