databendlabs/databend · error

invalid lease state

Error message

invalid lease state: {state}

What it means

The cluster lease keeper tracks lease state in an atomic. After a renew failure, finish_renew_failure transitions the state: it accepts LEASE_ACQUIRED or an already-failed state, honors shutdown requests, and for any other state treats the internal state machine as broken, panicking via unreachable! with "invalid lease state".

Solutions

1. Upgrade to a version where lease state transitions tolerate concurrent shutdown/delete states.
2. Capture and report the actual state value in logs before restarting the node.
3. Restart the query node to rebuild a clean lease state.
4. Reduce concurrent shutdown/renew races by draining renewals before shutdown.

Example fix

// before
Err(state) => unreachable!("invalid lease state: {state}"),
// after
Err(state) => {
    tracing::warn!("unexpected lease state {} after renew failure", state);
    RenewResult::Failed(error)
}

// Before renewing, check the node is not shutting down and lease is healthy
if keeper.is_shutdown() || keeper.lease_state() == LEASE_DELETED {
    return; // skip renew, rebuild lease
}

fn lease_state_valid(state: isize) -> bool {
    state == LEASE_ACQUIRED || state == LEASE_RENEW_FAILED || state == LEASE_SHUTDOWN_REQUESTED
}

match renew_result {
    Ok(_) => {},
    Err(e) if e.to_string().contains("invalid lease state") => {
        // reinitialize the lease keeper on the node
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: A concurrent shutdown/renew/delete changes the lease state between the compare_exchange and the error branch, leaving a state (e.g. LEASE_DELETED) that finish_renew_failure does not expect.

Common situations: Node shutdown or metadata (meta-service) disconnection racing with lease renewal; rapid connect/disconnect churn in clustered deployments.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• current_page has taken
• expect cluster.
• internal error: entered unreachable code
• rename_database: src (db) should exist
• _ => unreachable!()

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/b91093858ec79773. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/locks/lease_keeper.rs:200

    }

    fn is_retryable(error: &ErrorCode) -> bool {
        matches!(
            error.code(),
            ErrorCode::META_SERVICE_ERROR | ErrorCode::TXN_RETRY_MAX_TIMES
        )
    }

    fn finish_renew_failure(&self, error: ErrorCode) -> RenewResult {
        match self.state.compare_exchange(
            LEASE_RUNNING,
            LEASE_RENEW_FAILED,
            Ordering::AcqRel,
            Ordering::Acquire,
        ) {
            Ok(_) | Err(LEASE_RENEW_FAILED) => RenewResult::Failed(error),
            Err(LEASE_SHUTDOWN_REQUESTED) => RenewResult::Shutdown,
            Err(state) => unreachable!("invalid lease state: {state}"),
        }
    }

    async fn try_delete<O: LeaseOps>(&self, max_retry_elapsed: Duration, ops: &O) -> Result<()> {
        let mut backoff = set_backoff(
            Some(Duration::from_millis(2)),
            None,
            Some(max_retry_elapsed),
        );

        loop {
            match ops.delete().await {
                Ok(()) => return Ok(()),
                Err(error) if !Self::is_retryable(&error) => return Err(error),
                Err(error) => {
                    let Some(delay) = backoff.next_backoff() else {
                        return Err(error);
                    };

View on GitHub (pinned to 288d84d76e)