FuelLabs/fuel-core · critical · anyhow::Error

Backlog unresolved at height {current_height}: stream indica

Error message

Backlog unresolved at height {current_height}: stream indicates backlog but no entries found at next height

What it means

Backlog loop in unreconciled_blocks (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:821): at least one node's latest-entry read said the stream reaches at or beyond next_height, yet zero successfully-read nodes hold any entry at current_height, and no earlier height was reconciled in this call. The backlog signal and the stream contents disagree, so the missing range cannot be reconstructed from Redis and block production/import stalls.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:821

            poa_metrics().stream_trim_headroom.set(headroom);
        }

        let mut current_height = u32::from(next_height);

        for _ in 0..max_reconcile_blocks_per_round {
            let nodes_with_height = blocks_by_node
                .iter()
                .filter(|blocks_by_height| blocks_by_height.contains_key(&current_height))
                .count();

            tracing::debug!(
                "unreconciled_blocks: height={current_height} nodes_with_height={nodes_with_height}/{}",
                blocks_by_node.len()
            );

            if nodes_with_height == 0 {
                if reconciled.is_empty() {
                    return Err(anyhow!(
                        "Backlog unresolved at height {current_height}: \
                         stream indicates backlog but no entries found at next height"
                    ));
                }
                break;
            }

            // Group votes by block_id only (not epoch). The same block can
            // be written to different nodes with different epochs during
            // re-promotion storms — but if the block_id matches, it's the
            // same block and all copies count toward quorum. We track the
            // max epoch per block_id as the tiebreaker for fork resolution
            // when block_ids genuinely differ.
            let votes = blocks_by_node
                .iter()
                .filter_map(|blocks_by_height| blocks_by_height.get(&current_height))
                .flat_map(|blocks_by_epoch| blocks_by_epoch.iter())
                .fold(

View on GitHub (pinned to b9d4d170da)

Solutions

Check logs for 'Skipping stream entry' (corrupt payloads) and per-node read failures at that height, and recover those nodes or entries
Recover Redis data from AOF/RDB/replica, or re-publish the missing blocks from a fuel node that still has them locally
Increase stream_max_len so the stream outlasts expected downtimes
As a coordinated last resort, operator-reset the stream/lease keys after the height range is consciously abandoned (consensus decision, involves rollback)

Example fix

# inspect what the stream actually holds around the missing height:
redis-cli XRANGE <lease_key>:block:stream - + COUNT 100
# if trimmed away, re-publish from a node that has the blocks, or after
# team decision: DEL <lease_key>:block:stream and recover the backlog on-chain

Defensive patterns

Strategy: fallback

Validate before calling

// operator check: does the stream still cover next_height?
// redis-cli XRANGE <lease_key>:block:stream - + COUNT 1   (first height)
// redis-cli XREVRANGE <lease_key>:block:stream + - COUNT 1 (last height)
// if first_height > next_height, the range is trimmed away -> error 115 is expected

Try / catch

match unreconciled_blocks(next_height).await {
    Err(e) if e.to_string().contains("stream indicates backlog but no entries") => {
        // data missing from Redis: retrying cannot recreate it.
        // Fall back: re-publish blocks from a full local node, restore Redis from
        // AOF/RDB/replica, or run the coordinated stream-reset procedure.
    }
    other => other,
}

Prevention

Size stream_max_len for the longest expected validator downtime plus margin
Enable Redis persistence (AOF everysec) or run replicas for the leader-lock cluster
Never XTRIM/FLUSH the block:stream key manually without accounting for un-reconciled heights

When it happens

Trigger: Stream entries were trimmed or lost (stream_max_len too small for the downtime, Redis flush or failover without persistence) while the latest-entry hint still references them; or the only nodes holding that height all failed their batch read; or all payloads at that height were skipped as undeserializable.

Common situations: Validator offline longer than the stream retention; Redis restarted without AOF/RDB; manual XTRIM/FLUSH; version-skewed writes into the same stream.

Related errors

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/6241c00159bc2fe4. Report an issue: GitHub.