{"record":{"id":"b91093858ec79773","repo":"databendlabs/databend","slug":"invalid-lease-state-state","errorCode":null,"errorMessage":"invalid lease state: {state}","messagePattern":"invalid lease state: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/query/service/src/locks/lease_keeper.rs","lineNumber":200,"sourceCode":"    }\n\n    fn is_retryable(error: &ErrorCode) -> bool {\n        matches!(\n            error.code(),\n            ErrorCode::META_SERVICE_ERROR | ErrorCode::TXN_RETRY_MAX_TIMES\n        )\n    }\n\n    fn finish_renew_failure(&self, error: ErrorCode) -> RenewResult {\n        match self.state.compare_exchange(\n            LEASE_RUNNING,\n            LEASE_RENEW_FAILED,\n            Ordering::AcqRel,\n            Ordering::Acquire,\n        ) {\n            Ok(_) | Err(LEASE_RENEW_FAILED) => RenewResult::Failed(error),\n            Err(LEASE_SHUTDOWN_REQUESTED) => RenewResult::Shutdown,\n            Err(state) => unreachable!(\"invalid lease state: {state}\"),\n        }\n    }\n\n    async fn try_delete<O: LeaseOps>(&self, max_retry_elapsed: Duration, ops: &O) -> Result<()> {\n        let mut backoff = set_backoff(\n            Some(Duration::from_millis(2)),\n            None,\n            Some(max_retry_elapsed),\n        );\n\n        loop {\n            match ops.delete().await {\n                Ok(()) => return Ok(()),\n                Err(error) if !Self::is_retryable(&error) => return Err(error),\n                Err(error) => {\n                    let Some(delay) = backoff.next_backoff() else {\n                        return Err(error);\n                    };","sourceCodeStart":182,"sourceCodeEnd":218,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/query/service/src/locks/lease_keeper.rs#L182-L218","documentation":"The cluster lease keeper tracks lease state in an atomic. After a renew failure, finish_renew_failure transitions the state: it accepts LEASE_ACQUIRED or an already-failed state, honors shutdown requests, and for any other state treats the internal state machine as broken, panicking via unreachable! with \"invalid lease state\".","triggerScenarios":"A concurrent shutdown/renew/delete changes the lease state between the compare_exchange and the error branch, leaving a state (e.g. LEASE_DELETED) that finish_renew_failure does not expect.","commonSituations":"Node shutdown or metadata (meta-service) disconnection racing with lease renewal; rapid connect/disconnect churn in clustered deployments.","solutions":["Upgrade to a version where lease state transitions tolerate concurrent shutdown/delete states.","Capture and report the actual state value in logs before restarting the node.","Restart the query node to rebuild a clean lease state.","Reduce concurrent shutdown/renew races by draining renewals before shutdown."],"exampleFix":"// before\nErr(state) => unreachable!(\"invalid lease state: {state}\"),\n// after\nErr(state) => {\n    tracing::warn!(\"unexpected lease state {} after renew failure\", state);\n    RenewResult::Failed(error)\n}","handlingStrategy":"try-catch","validationCode":"// Before renewing, check the node is not shutting down and lease is healthy\nif keeper.is_shutdown() || keeper.lease_state() == LEASE_DELETED {\n    return; // skip renew, rebuild lease\n}","typeGuard":"fn lease_state_valid(state: isize) -> bool {\n    state == LEASE_ACQUIRED || state == LEASE_RENEW_FAILED || state == LEASE_SHUTDOWN_REQUESTED\n}","tryCatchPattern":"match renew_result {\n    Ok(_) => {},\n    Err(e) if e.to_string().contains(\"invalid lease state\") => {\n        // reinitialize the lease keeper on the node\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Drain background renewal tasks before node shutdown","Add tolerated states (LEASE_DELETED, LEASE_EXPIRED) to transition handlers","Monitor meta-service connectivity to reduce renew-failure races"],"tags":["rust","lease","concurrency","state-machine"],"backgroundTag":"invalid-state-transition","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}