apache/druid · warning

Lock key [%s] has no session owner

Error message

Lock key [%s] has no session owner

What it means

This warning is logged by ConsulLeaderSelector.validateLockOwnership when the Consul KV lock key exists but carries no Consul session attached (the 'Session' field of the KV entry is null). It means the distributed leadership lock has been orphaned — the session that created it expired or was destroyed but the key was not deleted — so leadership cannot be validated or assumed. validateLockOwnership returns false, causing leaderElectionLoop/becomeLeader to abort promotion and emit an ownership-mismatch metric.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:511

      }
    }
  }

  private boolean validateLockOwnership(String expectedSessionId)
  {
    try {
      Response<GetValue> response = consulClient.getKVValue(
          lockKey,
          config.getAuth().getAclToken(),
          buildQueryParams()
      );
      if (response == null || response.getValue() == null) {
        LOGGER.warn("Lock key [%s] missing when validating ownership", lockKey);
        return false;
      }
      String actualSessionId = response.getValue().getSession();
      if (actualSessionId == null) {
        LOGGER.warn("Lock key [%s] has no session owner", lockKey);
        return false;
      }
      boolean matches = expectedSessionId.equals(actualSessionId);
      if (!matches) {
        LOGGER.warn(
            "Lock key [%s] owned by session [%s], expected [%s]",
            lockKey,
            actualSessionId,
            expectedSessionId
        );
      }
      return matches;
    }
    catch (Exception e) {
      LOGGER.error(e, "Failed to validate lock ownership for [%s]", lockKey);
      return false;
    }
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Verify the lock key in Consul (consul kv get <lockKey>) — if its Session is empty, delete the orphaned key (consul kv delete <lockKey>) so the next election loop recreates it with a valid session
  2. Check Consul server health and session TTL configuration; ensure lock-ttl is comfortably larger than the election/renew interval so sessions don't lapse
  3. Restart the Druid node running ConsulLeaderSelector so it creates a fresh session and reacquires the lock
  4. Confirm no external process or human is writing to the lock key path; restrict ACL token write access to that KV prefix

Example fix

// before (shell inspection / manual repair)
$ consul kv get -detailed druid/leader-lock   # Session: (empty)
// after
$ consul kv delete druid/leader-lock          # let the election loop recreate with a session
Defensive patterns

Strategy: retry

Validate before calling

// before calling becomeLeader / after TTL expiries, check key ownership in Consul:
Response<GetValue> r = consulClient.getKVValue(lockKey, aclToken, qp);
boolean orphaned = r != null && r.getValue() != null && r.getValue().getSession() == null;
if (orphaned) { consulClient.deleteKVValue(lockKey, aclToken); }

Type guard

static boolean hasSessionOwner(GetValue v) { return v != null && v.getSession() != null; }

Try / catch

// validateLockOwnership already swallows exceptions; guard at the election-loop level:
if (!validateLockOwnership(sessionId)) {
  emitOwnershipMismatchMetric();
  // back off and retry next election cycle instead of promoting
  Thread.sleep(retryIntervalMs);
  continue;
}

Prevention

When it happens

Trigger: leaderElectionLoop or becomeLeader calls validateLockOwnership(currentSession); consulClient.getKVValue(lockKey, ...) returns a GetValue whose getSession() is null — e.g. the session TTL expired while the lock key remained in Consul, or the key was created/written manually without a session.

Common situations: Consul session invalidation after a network partition or long GC pause exceeding the lock-ttl; another operator or script writing the lock key manually (e.g. via consul kv put) without acquiring a session; Consul restart with session persistence issues; clock/TTL misconfiguration causing sessions to expire faster than the election loop renews them.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a8c2fd0c973566a2. Report an issue: GitHub.