{"record":{"id":"7557dddcd46cf05d","repo":"FuelLabs/fuel-core","slug":"backlog-unresolved-at-height-current-height-rep","errorCode":null,"errorMessage":"Backlog unresolved at height {current_height}: repair failed to reach quorum","messagePattern":"Backlog unresolved at height (.+?): repair failed to reach quorum","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":891,"sourceCode":"                        \"Repairing sub-quorum block at height {current_height} \\\n                         (found on {count}/{} nodes)\",\n                        blocks_by_node.len()\n                    );\n                    match self.repair_sub_quorum_block(&block, count) {\n                        Ok(true) => {\n                            tracing::info!(\n                                \"Repair succeeded — block at height {current_height} \\\n                                 now has quorum\"\n                            );\n                            reconciled.push(block);\n                        }\n                        Ok(false) => {\n                            tracing::warn!(\n                                \"Repair failed to reach quorum at height \\\n                                 {current_height} — will retry next round\"\n                            );\n                            if reconciled.is_empty() {\n                                return Err(anyhow!(\n                                    \"Backlog unresolved at height {current_height}: \\\n                                     repair failed to reach quorum\"\n                                ));\n                            }\n                            break;\n                        }\n                        Err(e) => {\n                            tracing::warn!(\n                                \"Repair error at height {current_height}: {e}\"\n                            );\n                            if reconciled.is_empty() {\n                                return Err(anyhow!(\n                                    \"Backlog unresolved at height {current_height}: \\\n                                     repair error: {e}\"\n                                ));\n                            }\n                            break;\n                        }","sourceCodeStart":873,"sourceCodeEnd":909,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L873-L909","documentation":"The winning block at current_height was found on fewer than quorum nodes (a partial write by a failed former leader), so unreconciled_blocks attempted repair_sub_quorum_block to re-propose it everywhere (poa.rs:877-897). The repair returned Ok(false) — the re-write did not reach quorum — and because nothing earlier in this round had been reconciled, the whole reconciliation aborts with this error.","triggerScenarios":"A previous leader wrote the block to only a minority of Redis nodes; at repair time too few nodes are healthy or too many refuse the write (HEIGHT_EXISTS with other data, memory pressure, read-only replicas) for re-propagation to reach quorum.","commonSituations":"Redis nodes flapping during leader transitions; write errors from maxmemory or replica state; re-promotion storms producing epoch conflicts on some nodes.","solutions":["Bring at least quorum Redis nodes to a healthy writable state so the repair write can succeed next round","Use the 'Repair failed to reach quorum at height …' warning plus per-node write errors to find which nodes refuse","Rely on automatic retry — repair is retried on every reconciliation round while conditions improve","If nodes reject because they already hold different data at that height, inspect the stream entries and resolve the conflict deliberately"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before repair, count writable healthy nodes:\nlet writable = probe_writes(&nodes).await; // SET/DEL round-trip on a probe key\nanyhow::ensure!(writable >= quorum, \"only {writable} writable nodes; repair cannot reach quorum\");","typeGuard":null,"tryCatchPattern":"match unreconciled_blocks(next_height).await {\n    Err(e) if e.to_string().contains(\"repair failed to reach quorum\") => {\n        // transient write-availability problem: repair retries next round;\n        // fix the refusing nodes (memory, read-only role, HEIGHT_EXISTS conflicts)\n    }\n    other => other,\n}","preventionTips":["Keep >= quorum Redis nodes writable (not read-only, not at maxmemory) at all times","Monitor per-node write errors during leader transitions","Avoid overlapping leadership changes while a sub-quorum repair backlog exists"],"tags":["fuel-core","redis","quorum","repair","reconciliation"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}