What causes "Lock key [%s] missing when validating ownership" in apache/druid?

Usually transient: the loop re-acquires the lock and recreates the key automatically.

Lock key [%s] missing when validating ownership

Error message

Lock key [%s] missing when validating ownership

What it means

Warning in validateLockOwnership() when the Consul KV read for the lock key returns no value — the key does not exist at validation time, so ownership cannot be confirmed and the method returns false. Callers (leaderElectionLoop, becomeLeader) then skip/abort promotion or step down.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:506

      try {
        consulClient.sessionDestroy(sessionId, buildQueryParams(), config.getAuth().getAclToken());
      }
      catch (Exception e) {
        LOGGER.error(e, "Failed to destroy session [%s]", sessionId);
      }
    }
  }

  private boolean validateLockOwnership(String expectedSessionId)
  {
    try {
      Response<GetValue> response = consulClient.getKVValue(
          lockKey,
          config.getAuth().getAclToken(),
          buildQueryParams()
      );
      if (response == null || response.getValue() == null) {
        LOGGER.warn("Lock key [%s] missing when validating ownership", lockKey);
        return false;
      }
      String actualSessionId = response.getValue().getSession();
      if (actualSessionId == null) {
        LOGGER.warn("Lock key [%s] has no session owner", lockKey);
        return false;
      }
      boolean matches = expectedSessionId.equals(actualSessionId);
      if (!matches) {
        LOGGER.warn(
            "Lock key [%s] owned by session [%s], expected [%s]",
            lockKey,
            actualSessionId,
            expectedSessionId
        );
      }
      return matches;
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

Usually transient: the loop re-acquires the lock and recreates the key automatically.
Verify druid.discovery.consul.service.datacenter matches the DC where the key lives.
Ensure no automation/operator deletes the leader lock key.
If persistent, check ACL token read permissions on the key and Consul KV health.

Example fix

// before
// dc1 holds the key, but query targets another DC
druid.discovery.consul.service.datacenter=dc2
// after
druid.discovery.consul.service.datacenter=dc1

Defensive patterns

Strategy: retry

Validate before calling

// pre-check the key exists before relying on leadership state
Response<GetValue> kv = consulClient.getKVValue(lockKey, token, QueryParams.DEFAULT);
if (kv == null || kv.getValue() == null) {
  // lock key absent: no leader currently, trigger a new election
}

Prevention

Confirm datacenter config points at the DC holding the key
Treat the lock key as ephemeral — never delete it manually
Alert on sustained absence of the lock key

When it happens

Trigger: Reading getKVValue(lockKey) returns a null value because the key was deleted (session expiry with Behavior=DELETE releases the lock entry), the key was never created because tryAcquireLock never succeeded, an operator deleted the KV entry, or the query targeted the wrong datacenter.

Common situations: Right after a leader session expires, the lock key disappears and followers see this before re-acquiring; manual consul kv delete operations; druid.discovery.consul.service.datacenter misconfigured so the key isn't visible in the queried DC.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/52a556662f191ef4. Report an issue: GitHub.