{"record":{"id":"3f353e26becfcdb5","repo":"FuelLabs/fuel-core","slug":"lost-lock-during-repair-another-leader-took-over","errorCode":null,"errorMessage":"Lost lock during repair — another leader took over","messagePattern":"Lost lock during repair — another leader took over","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":1200,"sourceCode":"        // Start from the pre-existing count (nodes already confirmed to\n        // have this specific block during reconciliation). Only count\n        // newly Written nodes — HeightExists means the node has *some*\n        // block at this height, but it might be a different block from\n        // a competing leader's partial write.\n        let mut total_with_block = pre_existing_count;\n        for result in self.publish_block_on_all_nodes(epoch, block, &block_data) {\n            match result {\n                Ok(WriteBlockResult::Written) => {\n                    total_with_block = total_with_block.saturating_add(1);\n                }\n                Ok(WriteBlockResult::HeightExists) => {\n                    // Node has some block at this height — may or may\n                    // not be ours. Don't count it; the pre_existing_count\n                    // already includes nodes confirmed to have our block.\n                }\n                Ok(WriteBlockResult::FencingRejected) => {\n                    // Lost the lock — repair is invalid, abort\n                    return Err(anyhow!(\n                        \"Lost lock during repair — another leader took over\"\n                    ));\n                }\n                Err(err) => {\n                    tracing::debug!(\"Repair write to node failed: {err}\");\n                }\n            }\n        }\n        let reached_quorum = self.quorum_reached(total_with_block);\n        if reached_quorum {\n            poa_metrics().repair_success_total.inc();\n        } else {\n            poa_metrics().repair_failure_total.inc();\n        }\n        Ok(reached_quorum)\n    }\n}\n","sourceCodeStart":1182,"sourceCodeEnd":1218,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L1182-L1218","documentation":"During repair, at least one Redis node returned WriteBlockResult::FencingRejected from write_block.lua: the epoch stored in the node's lease no longer matches this node's token — another leader acquired the lease with a higher epoch. Repair aborts immediately; this is the fencing mechanism doing its job by preventing two leaders from completing writes at the same height.","triggerScenarios":"This node's lease expired (lease_ttl_millis elapsed plus drift margin) while it was still reconciling its old partial writes, and another node won acquire_lease_if_free in the meantime.","commonSituations":"Long pauses (GC, disk stall, VM freeze) that outlive the lease TTL; TTL/drift configured too small for worst-case reconciliation time; duplicate deployments competing for the same lease key.","solutions":["Accept the abort: stop producing/repairing on this node and let the new lease holder reconcile — the error is correct behavior.","Compare lease_ttl_millis and lease_drift_millis against worst-case reconciliation duration and raise them if needed.","Verify only one deployment competes for the lease key (check lease_key / lease_owner_token config for duplicates).","Re-run reconciliation only after this node re-acquires the lease."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Fencing rejection during repair means leadership is gone — fall back to\n// 'let the new leader reconcile' instead of retrying with the stale epoch.\nif let Err(e) = repair_result {\n    if e.to_string().contains(\"Lost lock during repair\") {\n        tracing::warn!(\"lost leadership during repair; deferring to new leader\");\n        return Ok(false); // not repaired, by design\n    }\n    return Err(e);\n}","preventionTips":["Set lease_ttl_millis (plus drift margin) above worst-case reconciliation duration.","Avoid long stop-the-world pauses on leader nodes (GC tuning, no VM freezing under load).","Ensure exactly one competitor per lease_key in deployments.","Treat FencingRejected as a stop signal everywhere — never retry with the same epoch."],"tags":["redis","fencing","lease","split-brain","reconciliation"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}