{"record":{"id":"6241c00159bc2fe4","repo":"FuelLabs/fuel-core","slug":"backlog-unresolved-at-height-current-height-str","errorCode":null,"errorMessage":"Backlog unresolved at height {current_height}: stream indicates backlog but no entries found at next height","messagePattern":"Backlog unresolved at height (.+?): stream indicates backlog but no entries found at next height","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":821,"sourceCode":"            poa_metrics().stream_trim_headroom.set(headroom);\n        }\n\n        let mut current_height = u32::from(next_height);\n\n        for _ in 0..max_reconcile_blocks_per_round {\n            let nodes_with_height = blocks_by_node\n                .iter()\n                .filter(|blocks_by_height| blocks_by_height.contains_key(&current_height))\n                .count();\n\n            tracing::debug!(\n                \"unreconciled_blocks: height={current_height} nodes_with_height={nodes_with_height}/{}\",\n                blocks_by_node.len()\n            );\n\n            if nodes_with_height == 0 {\n                if reconciled.is_empty() {\n                    return Err(anyhow!(\n                        \"Backlog unresolved at height {current_height}: \\\n                         stream indicates backlog but no entries found at next height\"\n                    ));\n                }\n                break;\n            }\n\n            // Group votes by block_id only (not epoch). The same block can\n            // be written to different nodes with different epochs during\n            // re-promotion storms — but if the block_id matches, it's the\n            // same block and all copies count toward quorum. We track the\n            // max epoch per block_id as the tiebreaker for fork resolution\n            // when block_ids genuinely differ.\n            let votes = blocks_by_node\n                .iter()\n                .filter_map(|blocks_by_height| blocks_by_height.get(&current_height))\n                .flat_map(|blocks_by_epoch| blocks_by_epoch.iter())\n                .fold(","sourceCodeStart":803,"sourceCodeEnd":839,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L803-L839","documentation":"Backlog loop in unreconciled_blocks (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:821): at least one node's latest-entry read said the stream reaches at or beyond next_height, yet zero successfully-read nodes hold any entry at current_height, and no earlier height was reconciled in this call. The backlog signal and the stream contents disagree, so the missing range cannot be reconstructed from Redis and block production/import stalls.","triggerScenarios":"Stream entries were trimmed or lost (stream_max_len too small for the downtime, Redis flush or failover without persistence) while the latest-entry hint still references them; or the only nodes holding that height all failed their batch read; or all payloads at that height were skipped as undeserializable.","commonSituations":"Validator offline longer than the stream retention; Redis restarted without AOF/RDB; manual XTRIM/FLUSH; version-skewed writes into the same stream.","solutions":["Check logs for 'Skipping stream entry' (corrupt payloads) and per-node read failures at that height, and recover those nodes or entries","Recover Redis data from AOF/RDB/replica, or re-publish the missing blocks from a fuel node that still has them locally","Increase stream_max_len so the stream outlasts expected downtimes","As a coordinated last resort, operator-reset the stream/lease keys after the height range is consciously abandoned (consensus decision, involves rollback)"],"exampleFix":"# inspect what the stream actually holds around the missing height:\nredis-cli XRANGE <lease_key>:block:stream - + COUNT 100\n# if trimmed away, re-publish from a node that has the blocks, or after\n# team decision: DEL <lease_key>:block:stream and recover the backlog on-chain","handlingStrategy":"fallback","validationCode":"// operator check: does the stream still cover next_height?\n// redis-cli XRANGE <lease_key>:block:stream - + COUNT 1   (first height)\n// redis-cli XREVRANGE <lease_key>:block:stream + - COUNT 1 (last height)\n// if first_height > next_height, the range is trimmed away -> error 115 is expected","typeGuard":null,"tryCatchPattern":"match unreconciled_blocks(next_height).await {\n    Err(e) if e.to_string().contains(\"stream indicates backlog but no entries\") => {\n        // data missing from Redis: retrying cannot recreate it.\n        // Fall back: re-publish blocks from a full local node, restore Redis from\n        // AOF/RDB/replica, or run the coordinated stream-reset procedure.\n    }\n    other => other,\n}","preventionTips":["Size stream_max_len for the longest expected validator downtime plus margin","Enable Redis persistence (AOF everysec) or run replicas for the leader-lock cluster","Never XTRIM/FLUSH the block:stream key manually without accounting for un-reconciled heights"],"tags":["fuel-core","redis","stream","data-loss","reconciliation"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}