FuelLabs/fuel-core · critical · anyhow::Error

cannot access epoch token, poisoned lock

Error message

cannot access epoch token, poisoned lock

What it means

release_if_owner (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:952): when quorum ownership is not held, the adapter clears the shared current_epoch_token (std Mutex<Option<u64>>) so a future leader re-promotes from scratch. lock() returned a PoisonError — a thread panicked earlier while holding this mutex — so the token can be neither read nor written in this process and even the release path fails.

Solutions

1. Restart the fuel-core process — poisoned std Mutexes do not recover in-process
2. Find and fix the original panic earlier in the logs; this error is only its aftershock
3. Run under a supervisor with auto-restart; the lease TTL expiry hands leadership to another node safely in the meantime

fn is_poisoned_lock(e: &anyhow::Error) -> bool {
    e.to_string().contains("poisoned")
}

match leader_lock.release_if_owner().await {
    Err(e) if is_poisoned_lock(&e) => {
        // poison is terminal in-process: exit and let the supervisor restart;
        // lease TTL expiry releases leadership safely even if release failed
        tracing::error!("epoch mutex poisoned on release; exiting for restart");
        std::process::exit(1);
    }
    other => other,
}

When it happens

Trigger: Any earlier panic in code holding current_epoch_token (release, acquire, or epoch adoption); every subsequent lock on it fails, including this one during shutdown or leadership loss. The Redis lease itself still expires via TTL, but in-process epoch state is frozen.

Common situations: Seen after an unrelated panic in the PoA service; the node keeps hitting this on every release attempt until restarted; the drop_release_guard may still fire a best-effort release on drop.

Related errors

• epoch token lock poisoned
• The lock is poisoned
• At least one redis url is required for leader lock
• Not implemented yet
• The override height is zero. The override height should be…

AI-assisted analysis of FuelLabs/fuel-core@add100d30d (2026-09-05). Data as JSON: /api/errors/229307dbb9eeac8d. Report an issue: GitHub.

Appendix: source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:1109

        // the lease key when the stored owner matches our token, so
        // calling it on every node is safe regardless of whether we
        // currently hold quorum. We deliberately do NOT call
        // `has_lease_owner_quorum` here: that would spawn the
        // `spawn_acquire_lease_on_unowned_nodes` extension as a
        // detached task, which can race our own DELs and resurrect
        // the lease on a node we are simultaneously releasing
        // (observed as "Lost lock during repair" / FENCING_ERROR
        // when the next leader's writes hit the resurrected entry).
        let _ = futures::future::join_all(
            self.redis_nodes
                .iter()
                .map(|redis_node| self.release_lease_on_node(redis_node)),
        )
        .await;
        let mut current_epoch_token = self
            .current_epoch_token
            .lock()
            .map_err(|_| anyhow!("cannot access epoch token, poisoned lock"))?;
        *current_epoch_token = None;
        Ok(())
    }

    fn current_epoch_token_value(&self) -> Option<u64> {
        self.current_epoch_token
            .lock()
            .ok()
            .and_then(|epoch| *epoch)
    }

    /// Publish `block` to every Redis node in parallel and return as soon
    /// as `Written` quorum is reached. Each per-node publish runs in its
    /// own `tokio::spawn`ed task so it completes in the background even
    /// after we short-circuit — we trade a bit of background work per
    /// publish for maximum durability: every reachable node gets a real
    /// write attempt, bounded by `node_timeout`, instead of the call
    /// being cancelled mid-RESP when the parent drops a `FuturesUnordered`.

View on GitHub (pinned to add100d30d)