{"record":{"id":"270ef309df75cb9c","repo":"FuelLabs/fuel-core","slug":"timed-out-reading-latest-stream-entry-from-redis-n","errorCode":null,"errorMessage":"Timed out reading latest stream entry from Redis node","messagePattern":"Timed out reading latest stream entry from Redis node","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":621,"sourceCode":"        });\n    }\n\n    async fn read_latest_stream_entry_on_node(\n        &self,\n        redis_node: &RedisNode,\n    ) -> anyhow::Result<Option<(u32, String)>> {\n        let mut connection = self.multiplexed_connection(redis_node).await?;\n        let latest_entry = timeout(\n            self.node_timeout,\n            redis::Script::new(READ_LATEST_STREAM_ENTRY_SCRIPT)\n                .key(&self.block_stream_key)\n                .invoke_async::<Vec<String>>(&mut connection),\n        )\n        .await;\n        match latest_entry {\n            Err(_) => {\n                self.clear_cached_connection(redis_node).await;\n                Err(anyhow!(\n                    \"Timed out reading latest stream entry from Redis node\"\n                ))\n            }\n            Ok(Err(e)) => {\n                self.clear_cached_connection(redis_node).await;\n                Err(anyhow!(\n                    \"Failed to read latest stream entry from Redis node: {e}\"\n                ))\n            }\n            Ok(Ok(entry)) => {\n                if entry.len() != 2 {\n                    return Ok(None);\n                }\n                let height = entry[0]\n                    .parse::<u32>()\n                    .map_err(|e| anyhow!(\"Invalid latest stream entry height: {e}\"))?;\n                Ok(Some((height, entry[1].clone())))\n            }","sourceCodeStart":603,"sourceCodeEnd":639,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L603-L639","documentation":"read_latest_stream_entry_on_node (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:621) runs the READ_LATEST_STREAM_ENTRY_SCRIPT Lua script against the {lease_key}:block:stream stream under timeout(node_timeout). The timeout elapsing before the script reply produces this error; the cached connection is cleared (bumping connection_reset_total) so the next attempt reconnects fresh.","triggerScenarios":"A slow or stalled Redis node: CPU saturation, swap thrash, another long-running script blocking Redis's single thread, an oversized stream, or network latency pushing the round trip past node_timeout.","commonSituations":"Heavy XRANGE load from reconciliation rounds on long streams; Redis under-provisioned for the block-publish rate; transient network blips; node_timeout tuned only for fast-path operations.","solutions":["Retry — the adapter already reset the cached connection, so the next round reconnects","Check Redis health: SLOWLOG, CPU, memory; remove whatever long-blocking work is on the node","Keep the stream trimmed (stream_max_len) so latest-entry reads stay O(1)-ish","Raise node_timeout if the node is merely slow, not broken"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"let ping_under_timeout = timeout(node_timeout, connection.ping()).await.is_ok();\n// if false for a node, expect its reads to time out this round; check quorum margin first","typeGuard":null,"tryCatchPattern":"// per-node: log and exclude, then rely on quorum:\nif let Err(e) = read_latest(node).await {\n    tracing::warn!(%e, \"node excluded this round\");\n    failed += 1;\n    anyhow::ensure!(failed <= nodes - quorum, \"quorum at risk\");\n}","preventionTips":["Monitor Redis SLOWLOG and CPU; long-blocking scripts stall all subsequent commands","Keep stream_max_len bounded so reads stay fast","Alert on the connection_reset_total metric — it counts every timeout-driven reset"],"tags":["fuel-core","redis","timeout","stream","lua"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}