{"record":{"id":"84afd441e6eb8ca9","repo":"FuelLabs/fuel-core","slug":"backlog-unresolved-at-height-current-height-no","errorCode":null,"errorMessage":"Backlog unresolved at height {current_height}: no winning block candidate","messagePattern":"Backlog unresolved at height (.+?): no winning block candidate","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":914,"sourceCode":"                            break;\n                        }\n                        Err(e) => {\n                            tracing::warn!(\n                                \"Repair error at height {current_height}: {e}\"\n                            );\n                            if reconciled.is_empty() {\n                                return Err(anyhow!(\n                                    \"Backlog unresolved at height {current_height}: \\\n                                     repair error: {e}\"\n                                ));\n                            }\n                            break;\n                        }\n                    }\n                }\n            } else {\n                if reconciled.is_empty() {\n                    return Err(anyhow!(\n                        \"Backlog unresolved at height {current_height}: \\\n                         no winning block candidate\"\n                    ));\n                }\n                break;\n            }\n\n            let Some(next) = current_height.checked_add(1) else {\n                break;\n            };\n            current_height = next;\n        }\n\n        Ok(reconciled)\n    }\n\n    async fn can_produce_block(&self) -> anyhow::Result<bool> {\n        tracing::debug!(\"Checking Redis leader lock\");","sourceCodeStart":896,"sourceCodeEnd":932,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L896-L932","documentation":"Defensive branch in unreconciled_blocks (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:914): nodes_with_height was > 0, so some node reported the height, yet folding the per-height entries into block_id-keyed votes produced no candidate at all; with an empty reconciled list the round aborts. Since every stored entry contributes a vote, this fires only when the height's entries were effectively emptied upstream — most plausibly all its payloads failed postcard deserialization and were skipped with a warning.","triggerScenarios":"All entries at current_height were dropped during read_stream_entries_on_node deserialization (logged as 'Skipping stream entry: failed to deserialize block at height …'), leaving a height key present on nodes but no votable block; or an internal invariant break in vote grouping.","commonSituations":"Version skew: a differently-serialized SealedBlock format written into the stream by another fuel-core version; corrupted payloads after a Redis restore.","solutions":["Look for 'Skipping stream entry' warnings at the same height — they carry the real deserialization cause","Align all fuel-core nodes sharing the lease_key/stream to one version","Remove or rebuild the corrupt stream entries (operator action via XDEL) so the height regains a votable candidate"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// operator check for the usual root cause (all payloads at a height skipped):\n// grep node logs for 'Skipping stream entry: failed to deserialize block at height H'\n// if present, that height has no votable block on any healthy node","typeGuard":null,"tryCatchPattern":"match unreconciled_blocks(next_height).await {\n    Err(e) if e.to_string().contains(\"no winning block candidate\") => {\n        // almost always downstream of deserialization skips: fix version skew /\n        // corrupt entries (XDEL), then let the next round re-vote the height\n    }\n    other => other,\n}","preventionTips":["Pin every node sharing a lease_key to the same fuel-core version","Alert on 'Skipping stream entry' warnings — they are the early signal of this failure","Validate restored Redis data by deserializing a sample of stream payloads before promoting it"],"tags":["fuel-core","redis","stream","serialization","invariant"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}