FuelLabs/fuel-core · error · anyhow::Error

Lost lock during repair — another leader took over

Error message

Lost lock during repair — another leader took over

What it means

During repair, at least one Redis node returned WriteBlockResult::FencingRejected from write_block.lua: the epoch stored in the node's lease no longer matches this node's token — another leader acquired the lease with a higher epoch. Repair aborts immediately; this is the fencing mechanism doing its job by preventing two leaders from completing writes at the same height.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:1200

        // Start from the pre-existing count (nodes already confirmed to
        // have this specific block during reconciliation). Only count
        // newly Written nodes — HeightExists means the node has *some*
        // block at this height, but it might be a different block from
        // a competing leader's partial write.
        let mut total_with_block = pre_existing_count;
        for result in self.publish_block_on_all_nodes(epoch, block, &block_data) {
            match result {
                Ok(WriteBlockResult::Written) => {
                    total_with_block = total_with_block.saturating_add(1);
                }
                Ok(WriteBlockResult::HeightExists) => {
                    // Node has some block at this height — may or may
                    // not be ours. Don't count it; the pre_existing_count
                    // already includes nodes confirmed to have our block.
                }
                Ok(WriteBlockResult::FencingRejected) => {
                    // Lost the lock — repair is invalid, abort
                    return Err(anyhow!(
                        "Lost lock during repair — another leader took over"
                    ));
                }
                Err(err) => {
                    tracing::debug!("Repair write to node failed: {err}");
                }
            }
        }
        let reached_quorum = self.quorum_reached(total_with_block);
        if reached_quorum {
            poa_metrics().repair_success_total.inc();
        } else {
            poa_metrics().repair_failure_total.inc();
        }
        Ok(reached_quorum)
    }
}

View on GitHub (pinned to b9d4d170da)

Solutions

1. Accept the abort: stop producing/repairing on this node and let the new lease holder reconcile — the error is correct behavior.
2. Compare lease_ttl_millis and lease_drift_millis against worst-case reconciliation duration and raise them if needed.
3. Verify only one deployment competes for the lease key (check lease_key / lease_owner_token config for duplicates).
4. Re-run reconciliation only after this node re-acquires the lease.

// Fencing rejection during repair means leadership is gone — fall back to
// 'let the new leader reconcile' instead of retrying with the stale epoch.
if let Err(e) = repair_result {
    if e.to_string().contains("Lost lock during repair") {
        tracing::warn!("lost leadership during repair; deferring to new leader");
        return Ok(false); // not repaired, by design
    }
    return Err(e);
}

When it happens

Trigger: This node's lease expired (lease_ttl_millis elapsed plus drift margin) while it was still reconciling its old partial writes, and another node won acquire_lease_if_free in the meantime.

Common situations: Long pauses (GC, disk stall, VM freeze) that outlive the lease TTL; TTL/drift configured too small for worst-case reconciliation time; duplicate deployments competing for the same lease key.

Related errors

• Cannot repair block because fencing token is not initialized
• Cannot publish block because fencing token is not initialize
• Cannot reconcile: only {}/{} Redis nodes responded ({} faile
• Timed out reading stream entries from Redis node
• Backlog unresolved at height {current_height}: stream indica

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/3f353e26becfcdb5. Report an issue: GitHub.