FuelLabs/fuel-core · error · anyhow::Error

Failed to release lease on quorum

Error message

Failed to release lease on quorum

What it means

release_if_owner releases the leader lease on every configured Redis node via release_lease_on_node, counts confirmations, and clears the local epoch token only when released_count >= self.quorum (quorum_reached, poa.rs:369). When fewer than quorum nodes confirm release, it returns this error and deliberately keeps the local token set, so the node still considers itself leader until the lease TTL expires.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:972

            return Ok(());
        }

        let releases = futures::future::join_all(
            self.redis_nodes
                .iter()
                .map(|redis_node| self.release_lease_on_node(redis_node)),
        )
        .await;
        let released_count = releases.into_iter().filter(|released| *released).count();
        if self.quorum_reached(released_count) {
            let mut current_epoch_token = self
                .current_epoch_token
                .lock()
                .map_err(|_| anyhow!("cannot access epoch token, poisoned lock"))?;
            *current_epoch_token = None;
            Ok(())
        } else {
            Err(anyhow!("Failed to release lease on quorum"))
        }
    }

    fn current_epoch_token_value(&self) -> Option<u64> {
        self.current_epoch_token
            .lock()
            .ok()
            .and_then(|epoch| *epoch)
    }

    fn publish_block_on_all_nodes(
        &self,
        epoch: u64,
        block: &SealedBlock,
        block_data: &[u8],
    ) -> Vec<anyhow::Result<WriteBlockResult>> {
        // Detached `std::thread::spawn` (not scoped) lets us return as soon
        // as `Written` quorum is reached without waiting for slow nodes.

View on GitHub (pinned to b9d4d170da)

Solutions

1. Check connectivity and health to every configured Redis endpoint (redis-cli ping) and restore the unreachable ones.
2. Verify the configured quorum is <= the number of configured Redis nodes and matches the intended failure tolerance.
3. If endpoints cannot be restored quickly, rely on lease TTL: the lease self-expires and the next leader acquires it — safe by design.
4. Once connectivity is healthy, retry the step-down/release.

// Pre-flight before triggering step-down: confirm a quorum of Redis
// endpoints answers PING, so release can realistically succeed.
async fn quorum_reachable(redis_nodes: &[RedisNode], quorum: usize) -> bool {
    let oks = futures::future::join_all(
        redis_nodes.iter().map(|n| async { n.client().ping().await.is_ok() }),
    ).await;
    oks.into_iter().filter(|ok| *ok).count() >= quorum
}

// On failure: log a warning and let the lease TTL handle safety — do NOT
// block shutdown on quorum release.
if let Err(e) = release_result {
    if e.to_string().contains("Failed to release lease on quorum") {
        tracing::warn!("lease release fell short of quorum; waiting for TTL expiry");
    } else {
        return Err(e);
    }
}

When it happens

Trigger: More than (redis_nodes.len() - quorum) nodes return false from release_lease_on_node — connection error, Lua release script error (poa.rs:362-365 clears cached connections on Err), or the node no longer holds the key. Example: 5 Redis nodes, quorum 3, 3 nodes unreachable → only 2 releases.

Common situations: Network partition between the fuel node and part of the Redis quorum during shutdown or leader step-down; Redis endpoints restarted or overloaded; a quorum configured larger than the number of reachable nodes.

Related errors

• Timed out while connecting to redis leader-lock node
• At least one redis url is required for leader lock
• Cannot reconcile: only {}/{} Redis nodes responded ({} faile
• Backlog unresolved at height {current_height}: repair failed
• publish abandoned: quorum reached before this node responded

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/adf88162741d2dc6. Report an issue: GitHub.