Main Loop: Lost lock ownership check for [%s], stepping down

Error message

Main Loop: Lost lock ownership check for [%s], stepping down

What it means

Warning logged by the leader-election loop when a periodic re-check shows the Consul KV lock key is no longer owned by this node's session. The selector steps down (loseLeadership) so the listener's stopBeingLeader() runs and another node can take over. This is the primary fail-safe against silently holding leadership after losing the underlying lock.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulLeaderSelector.java:283

          } else if (validateLockOwnership(sessionId)) {
            long electionStart = System.nanoTime();
            becomeLeader();
            long electionLatency = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - electionStart);
            ConsulMetrics.emitTimer(emitter, "consul/leader/election_latency", electionLatency,
                "lock", lockKey);
          } else {
            LOGGER.warn("Lock ownership validation failed for [%s]; will retry", lockKey);
            emitOwnershipMismatchMetric();
          }
        } else if (!acquired && leader.get()) {
          loseLeadership();
        }

        if (leader.get()) {
          // Session renewal handled by sessionKeeperLoop; here we just verify lock ownership
          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());
          if (sessionId != null && !validateLockOwnership(sessionId)) {
            LOGGER.warn("Main Loop: Lost lock ownership check for [%s], stepping down", lockKey);
            loseLeadership();
          }
        } else {
          Thread.sleep(config.getService().getHealthCheckInterval().getMillis());
        }
        errorRetryCount = 0;
      }
      catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        break;
      }
      catch (Exception e) {
        LOGGER.error(e, "Error in leader election loop");

        if (leader.get()) {
          loseLeadership();
        }
        sessionId = null;

View on GitHub (pinned to 9b90983fd2)

Solutions

Fix session renewal reliability: ensure healthCheckInterval/3 renewals succeed; check 'Failed to renew session' warnings.
Increase leaderSessionTtl (up to 120s) if transient RPC latency causes expiry between renewals.
Verify Consul agent stability and network between the Druid node and Consul.
Confirm no other process writes to the lock key; after step-down, the loop will re-elect automatically.

Example fix

// before: TTL 10s with RPC hiccups expiring it between renewals
druid.discovery.consul.leader.sessionTtl=PT10S
// after
druid.discovery.consul.leader.sessionTtl=PT30S

Defensive patterns

Strategy: retry

Try / catch

// listeners must tolerate stopBeingLeader() at any time
public void stopBeingLeader() {
  if (!currentlyLeading) return; // idempotent step-down
  // release resources
}

Prevention

Make listener.becomeLeader/stopBeingLeader idempotent and fast
Renew session well within TTL (keeper renews at healthCheckInterval/3)
Keep TTL >= 3x expected Consul RPC latency
Monitor consul/leader/stop and renew/fail metrics

When it happens

Trigger: While leader, the scheduled validateLockOwnership() finds the key missing, owned by another session, or the KV read throws — e.g. the session expired (renewal failures), another node force-acquired the key, or the KV entry was deleted.

Common situations: Consul agent restart or network partition causing session expiry, leaderSessionTtl too short for the renewal cadence, an operator deleting the KV key, or a competing node overwriting the key.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4a4d34a2bdc01ce5. Report an issue: GitHub.