FuelLabs/fuel-core · error · anyhow::Error

Timed out reading latest stream entry from Redis node

Error message

Timed out reading latest stream entry from Redis node

What it means

read_latest_stream_entry_on_node (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:621) runs the READ_LATEST_STREAM_ENTRY_SCRIPT Lua script against the {lease_key}:block:stream stream under timeout(node_timeout). The timeout elapsing before the script reply produces this error; the cached connection is cleared (bumping connection_reset_total) so the next attempt reconnects fresh.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:621

        });
    }

    async fn read_latest_stream_entry_on_node(
        &self,
        redis_node: &RedisNode,
    ) -> anyhow::Result<Option<(u32, String)>> {
        let mut connection = self.multiplexed_connection(redis_node).await?;
        let latest_entry = timeout(
            self.node_timeout,
            redis::Script::new(READ_LATEST_STREAM_ENTRY_SCRIPT)
                .key(&self.block_stream_key)
                .invoke_async::<Vec<String>>(&mut connection),
        )
        .await;
        match latest_entry {
            Err(_) => {
                self.clear_cached_connection(redis_node).await;
                Err(anyhow!(
                    "Timed out reading latest stream entry from Redis node"
                ))
            }
            Ok(Err(e)) => {
                self.clear_cached_connection(redis_node).await;
                Err(anyhow!(
                    "Failed to read latest stream entry from Redis node: {e}"
                ))
            }
            Ok(Ok(entry)) => {
                if entry.len() != 2 {
                    return Ok(None);
                }
                let height = entry[0]
                    .parse::<u32>()
                    .map_err(|e| anyhow!("Invalid latest stream entry height: {e}"))?;
                Ok(Some((height, entry[1].clone())))
            }

View on GitHub (pinned to b9d4d170da)

Solutions

1. Retry — the adapter already reset the cached connection, so the next round reconnects
2. Check Redis health: SLOWLOG, CPU, memory; remove whatever long-blocking work is on the node
3. Keep the stream trimmed (stream_max_len) so latest-entry reads stay O(1)-ish
4. Raise node_timeout if the node is merely slow, not broken

let ping_under_timeout = timeout(node_timeout, connection.ping()).await.is_ok();
// if false for a node, expect its reads to time out this round; check quorum margin first

// per-node: log and exclude, then rely on quorum:
if let Err(e) = read_latest(node).await {
    tracing::warn!(%e, "node excluded this round");
    failed += 1;
    anyhow::ensure!(failed <= nodes - quorum, "quorum at risk");
}

When it happens

Trigger: A slow or stalled Redis node: CPU saturation, swap thrash, another long-running script blocking Redis's single thread, an oversized stream, or network latency pushing the round trip past node_timeout.

Common situations: Heavy XRANGE load from reconciliation rounds on long streams; Redis under-provisioned for the block-publish rate; transient network blips; node_timeout tuned only for fast-path operations.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out reading stream entries from Redis node
• Timed out while connecting to redis leader-lock node
• Failed to read latest stream entry from Redis node: {e}
• Invalid latest stream entry height: {e}
• Failed to read stream entries from Redis node: {e}

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/270ef309df75cb9c. Report an issue: GitHub.