apache/druid · warning

Lock ownership validation failed for [%s]; will retry

Error message

Lock ownership validation failed for [%s]; will retry

What it means

Warning logged when the loop acquired the Consul KV lock but validateLockOwnership() shows the key's session does not match the local session id (or validation failed). The selector refuses to promote to leader and emits a consul/leader/ownership_mismatch metric; it will retry on the next cycle.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:272

        if (acquired && !leader.get()) {
          boolean interrupted = Thread.currentThread().isInterrupted();
          if (stopping || interrupted) {
            LOGGER.info(
                "Skipping leadership for [%s] because selector is stopping (interrupted=%s)",
                lockKey,
                interrupted
            );
          } else if (sessionId == null) {
            LOGGER.warn("Skipping leadership for [%s] because session is null", lockKey);
          } else if (validateLockOwnership(sessionId)) {
            long electionStart = System.nanoTime();
            becomeLeader();
            long electionLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - electionStart);
            ConsulMetrics.emitTimer(emitter, "consul/leader/election_latency", electionLatency,
                "lock", lockKey);
          } else {
            LOGGER.warn("Lock ownership validation failed for [%s]; will retry", lockKey);
            emitOwnershipMismatchMetric();
          }
        } else if (!acquired && leader.get()) {
          loseLeadership();
        }

        if (leader.get()) {
          // Session renewal handled by sessionKeeperLoop; here we just verify lock ownership
          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());
          if (sessionId != null && !validateLockOwnership(sessionId)) {
            LOGGER.warn("Main Loop: Lost lock ownership check for [%s], stepping down", lockKey);
            loseLeadership();
          }
        } else {
          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());
        }
        errorRetryCount = 0;
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify only one Druid service uses this lockKey; check for duplicate configurations across nodes.
2. Inspect the KV entry (consul kv get <lockKey>) to see which session owns it and from where.
3. After a Consul outage, wait out the 5s lock-delay and let the loop retry; sessions get recreated automatically.
4. If it persists, check ACL read consistency and whether a stale session is pinned to the key; delete the stale session.

Example fix

// before: two services sharing the same election key with conflicting selectors
// ensure only the intended role's selector uses druid/leader/coordinator,
// then destroy the stale session:
//   curl http://consul:8500/v1/session/node/<node>?dc=dc1  (inspect) and destroy stale session

// before assuming leadership externally
Response<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);
boolean ownedByUs = kv.getValue() != null && expectedSessionId.equals(kv.getValue().getSession());

When it happens

Trigger: Between tryAcquireLock success and validation, the lock's owning session changed — another node acquired the key, the key was deleted and recreated, or the KV read returned a value owned by a stale session. Also fires when the KV read itself errors (validateLockOwnership returns false on exception).

Common situations: Split-brain recovery after a Consul outage, two nodes racing for the same lockKey (duplicate lockKey config), manual KV edits under the lock key, or lock-delay quirks right after session invalidation.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Ownership check failed for [%s]
• Skipping leadership for [%s] because session is null
• Lock key [%s] missing when validating ownership
• can't start
• can't stop

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/7798e43151864e46. Report an issue: GitHub.