{"record":{"id":"0226234f1e74cbba","repo":"FuelLabs/fuel-core","slug":"timed-out-reading-stream-entries-from-redis-node","errorCode":null,"errorMessage":"Timed out reading stream entries from Redis node","messagePattern":"Timed out reading stream entries from Redis node","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":711,"sourceCode":"            return Ok(Vec::new());\n        }\n\n        let mut connection = self.multiplexed_connection(redis_node).await?;\n        let count = u32::try_from(max_entries).unwrap_or(u32::MAX);\n        let stream_entries = timeout(\n            self.node_timeout,\n            redis::Script::new(READ_STREAM_ENTRIES_SCRIPT)\n                .key(&self.block_stream_key)\n                .arg(next_height)\n                .arg(count)\n                .invoke_async::<Vec<(u32, u64, Vec<u8>, String)>>(&mut connection),\n        )\n        .await;\n\n        let entries = match stream_entries {\n            Err(_) => {\n                self.clear_cached_connection(redis_node).await;\n                return Err(anyhow!(\"Timed out reading stream entries from Redis node\"));\n            }\n            Ok(Err(e)) => {\n                self.clear_cached_connection(redis_node).await;\n                return Err(anyhow!(\n                    \"Failed to read stream entries from Redis node: {e}\"\n                ));\n            }\n            Ok(Ok(entries)) => entries,\n        };\n\n        let mut blocks = Vec::new();\n        for (height, epoch, bytes, _stream_id) in entries {\n            match postcard::from_bytes::<SealedBlock>(&bytes) {\n                Ok(block) => blocks.push((height, epoch, block)),\n                Err(e) => {\n                    tracing::warn!(\n                        \"Skipping stream entry: failed to deserialize block at height {height}: {e}\"\n                    );","sourceCodeStart":693,"sourceCodeEnd":729,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L693-L729","documentation":"read_stream_entries_on_node (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:711) fetches up to stream_max_len entries starting at next_height via READ_STREAM_ENTRIES_SCRIPT under timeout(node_timeout). The timeout elapsing maps to this error; the connection cache is cleared and the whole node counts as a failed read toward the quorum check.","triggerScenarios":"Large batch reads (stream_max_len big) over slow links or against a busy Redis; Redis blocked by another long Lua script; node_timeout smaller than worst-case XRANGE duration for the batch size.","commonSituations":"Long backlog after downtime so every reconciliation round scans many entries; undersized Redis CPU; network degradation between the node and Redis.","solutions":["Let rounds retry — reads are idempotent and reconciliation resumes where it left off","Lower stream_max_len to bound the per-round batch size","Raise node_timeout and/or give Redis more CPU and memory","Trim already-reconciled history so batches stay small"],"exampleFix":"// before: stream_max_len: 10_000, node_timeout: 500ms\n// after:  stream_max_len: 1_000,  node_timeout: 3s","handlingStrategy":"retry","validationCode":"// size the batch to the timeout budget before calling:\nlet entries_per_sec = measured_read_throughput();\nlet safe_max = (entries_per_sec * node_timeout.as_secs_f64()) as usize;\nlet count = stream_max_len.min(safe_max);","typeGuard":null,"tryCatchPattern":"// treat as node-down: adapter already cleared the cached connection;\n// next round reconnects. Alert only when failed nodes threaten quorum:\nif failed > nodes - quorum { alert!(\"batch-read timeouts threaten quorum\"); }","preventionTips":["Bound stream_max_len so worst-case XRANGE fits comfortably inside node_timeout","Provision Redis CPU for backlog-reconciliation load, not just steady-state publishing","Trim reconciled stream history so batches stay small"],"tags":["fuel-core","redis","timeout","stream","reconciliation"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}