FuelLabs/fuel-core · error · anyhow::Error

Timed out reading stream entries from Redis node

Error message

Timed out reading stream entries from Redis node

What it means

read_stream_entries_on_node (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:711) fetches up to stream_max_len entries starting at next_height via READ_STREAM_ENTRIES_SCRIPT under timeout(node_timeout). The timeout elapsing maps to this error; the connection cache is cleared and the whole node counts as a failed read toward the quorum check.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:711

            return Ok(Vec::new());
        }

        let mut connection = self.multiplexed_connection(redis_node).await?;
        let count = u32::try_from(max_entries).unwrap_or(u32::MAX);
        let stream_entries = timeout(
            self.node_timeout,
            redis::Script::new(READ_STREAM_ENTRIES_SCRIPT)
                .key(&self.block_stream_key)
                .arg(next_height)
                .arg(count)
                .invoke_async::<Vec<(u32, u64, Vec<u8>, String)>>(&mut connection),
        )
        .await;

        let entries = match stream_entries {
            Err(_) => {
                self.clear_cached_connection(redis_node).await;
                return Err(anyhow!("Timed out reading stream entries from Redis node"));
            }
            Ok(Err(e)) => {
                self.clear_cached_connection(redis_node).await;
                return Err(anyhow!(
                    "Failed to read stream entries from Redis node: {e}"
                ));
            }
            Ok(Ok(entries)) => entries,
        };

        let mut blocks = Vec::new();
        for (height, epoch, bytes, _stream_id) in entries {
            match postcard::from_bytes::<SealedBlock>(&bytes) {
                Ok(block) => blocks.push((height, epoch, block)),
                Err(e) => {
                    tracing::warn!(
                        "Skipping stream entry: failed to deserialize block at height {height}: {e}"
                    );

View on GitHub (pinned to b9d4d170da)

Solutions

1. Let rounds retry — reads are idempotent and reconciliation resumes where it left off
2. Lower stream_max_len to bound the per-round batch size
3. Raise node_timeout and/or give Redis more CPU and memory
4. Trim already-reconciled history so batches stay small

Example fix

// before: stream_max_len: 10_000, node_timeout: 500ms
// after:  stream_max_len: 1_000,  node_timeout: 3s

// size the batch to the timeout budget before calling:
let entries_per_sec = measured_read_throughput();
let safe_max = (entries_per_sec * node_timeout.as_secs_f64()) as usize;
let count = stream_max_len.min(safe_max);

// treat as node-down: adapter already cleared the cached connection;
// next round reconnects. Alert only when failed nodes threaten quorum:
if failed > nodes - quorum { alert!("batch-read timeouts threaten quorum"); }

When it happens

Trigger: Large batch reads (stream_max_len big) over slow links or against a busy Redis; Redis blocked by another long Lua script; node_timeout smaller than worst-case XRANGE duration for the batch size.

Common situations: Long backlog after downtime so every reconciliation round scans many entries; undersized Redis CPU; network degradation between the node and Redis.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out reading latest stream entry from Redis node
• Backlog unresolved at height {current_height}: stream indica
• Backlog unresolved at height {current_height}: repair error:
• Timed out while connecting to redis leader-lock node
• Failed to read latest stream entry from Redis node: {e}

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/0226234f1e74cbba. Report an issue: GitHub.